I think the biggest reason no one has touched upon, besides the obvious difficulty in judging a pitch to be far out of the strike zone or not is the hit and run play.
If there is a runner on base and this play is called the batter will (and should) swing at any pitch unless it is egregiously hard to hit. (Sometimes they still swing to throw off the catcher and make throwing out the runner a little more difficult).
The whole point of a hit and run is that the man on base will be taking off as soon as the pitcher goes and will prevent a double play if the batter hits a routine ground ball (hopefully). If the batter manages a hit the runner might even be able to score from first or second on a hit which he would normally not. If the batter doesn't make contact then the runner could be thrown out, technically caught stealing.