What is the US Open?
The US Open is a major golf championship, the second during a given calendar year, that commences in mid-June. It is dubbed "Golf's Toughest Test," and for great reason. The USGA, the organization that stages the US Open, sets up its golf courses as a rigorous test to challenge a golfer's shot-making ability, course management, and composure under pressure.
The USGA considers the following factors in setting up US Open golf courses:
Length, variation and playing characteristics of individual holes;
Length of overall golf course relative to total par;
Teeing ground locations (i.e., angles of play, variation of distance day to day);
Fairway width and contours;
Fairway firmness and speed;
Green speed relative to percentage slopes and contours of the putting greens;
Putting green firmness;
Rough height, density and stages of severity;
Bunker preparation (i.e., create challenge of recovery);
Green surrounds (e.g., closely mown areas vs. primary rough);
Hole locations (relative difficulty, balance in location of left vs. right, front vs. back of green, anticipated wind, anticipated
length of approach shot);
Risk and reward options;
Anticipated weather conditions;
Pace of Play
An example in relation to the Olympic Club, the USGA added a bunker on the 17th hole to "complicate the 'lay-up' shot and encourage players to go for the green in two."(1)
Wikipedia's description of the US Open:
The U.S. Open is staged at a variety of courses, set up in such a way
that scoring is very difficult with a premium placed on accurate
driving. U.S. Open play is characterized by tight scoring at or around
par by the leaders, with the winner usually emerging at around even
par. A U.S. Open course is seldom beaten severely, and there have been
many over-par wins (in part because par is usually set at 70 except
for the very longest courses). Normally, an Open course is quite long
and will have a high cut of primary rough (termed "Open rough" by the
American press and fans), undulating greens (such as at Pinehurst No.
2 in 2005, which was described by Johnny Miller of NBC as "like trying
to hit a ball on top of a VW Beetle"), and pinched fairways
(especially on what are expected to be less difficult holes).
US Open Results
Historically, winners finish around even par. Only extreme cases (Low scores: Tiger Woods in 2000, Rory McIlroy in 2011...High scores: Geoff Ogilvy in 2006, and Angel Cabrera in 2007...in recent history) show otherwise.
In 2000, Tiger Woods won at -12, 272. Ernie Els and Miguel Angel-Jimenez finished tied for second place....at +3, 287....15 shots behind Woods. This has been portrayed as the most dominating major victory in history(1). Take away Woods's performance, and you will see that the 2000 US Open played difficult among the golfers.
In 2011, Rory McIlroy broke Woods's record score in relation to par and aggregate by winning at -16, 268. Jason Day finished second at -8, 8 shots behind McIlroy. However, 20 golfers finished under par that week, and 38 golfers finished at +3 or better. Take away McIlroy's performance, and you will see that the 2011 US Open still played favorable among the golfers. Reason for the relatively favorable play that week? Rain, and lots of it. It softented up the course and slowed down fairways and greens. US Open fairways and greens are historically very fast.
A brief comparison to the Masters: The Masters historically scores under par. A Masters winner has finished over par only a few times (like Zach Johnson in 2007. Reason? Mid-50s°F weather the whole week. The Masters usually plays in the 80s°F).
The point of the US Open is to see which golfer survives after four (or more) rounds of golf. The Olympic Club is "so difficult" because of the standards in which the USGA sets up US Open golf courses.
Note: Based on your question, I interpreted that your observation that the Olympic Club is difficult was based on first round scores of the US Open. Therefore, I answered with the purpose of the US Open in mind. There is another answer that illustrates the difficulty of the Olympic Club itself.