NOTE: As discovered in the 1896 Olympic games, keeping time by hand introduced a higher margin of (human) error, as one may expect. This introduced a challenge for the IOC. (SOURCE)
In the 1920, 1924, and 1928 Olympic games, the Swiss-based company Heuer (known today as TAG Heuer) was the official timekeeper. Times had been measured to the nearest 1/5th of a second and official times were only provided for winners.
Omega has been the official timekeeper of the Olympics since the 1932 Summer Olympic games. OMEGA's calibre 1130 stopwatches were used, were accurate to the nearest 1/10th of a second, and had the ability to measure split-times. They were the first company to provide "observatory precision-rating certificates" for timekeeping in all events.
Also according to Omega's excerpt, many of the judges‘ decisions were shrouded in controversy at the '32 Olympic Games. The same time was recorded for the winner and runner-up in five different races. Most notably, the 100 meters between Thomas Edward “Eddie“ Tolan and Ralph Metcalfe.
The timekeepers‘ hand-held stopwatches had recorded three times of
10.3 seconds for Metcalfe and two times of 10.3 and one of 10.4 seconds for Tolan. Even so, Tolan was declared the winner.
However, Tolan's victory was confirmed by video, and introduced a rule we're all familiar with, in bold below.
Gustavus T. Kirby, Chairman of the Jury and inventor of the
“Chronocinema“, a camera which filmed the end of races and was used to
record times to the nearest 1/100th of a second, stated categorically
after examining the film: “The seven judges and myself have viewed the
film of the finish several times. We can state that Tolan won by
exactly five hundredths of a second. Both competitors reached the
finishing tape at exactly the same moment, but the rules specify that
the race is finished only when the athlete‘s torso has completely
crossed the finishing line marked on the ground. Tolan crossed before
Metcalfe.“ This rule, which was often interpreted in different ways,
was changed in 1933. Since then, the winner has been the first person
to cross the line with any part of his or her torso. Kirby‘s camera
was used on an experimental basis and was only consulted in
exceptional cases because it took too long to develop the films.
Summary: Stopwatches had a margin of error, but triangulation (comparing three stopwatches for one competitor) and the "experimental" use of video confirmed a competitor's victory/defeat. However, even with today's technology, Phelps's victory over Čavić was too close to call.