It's entirely up to the umpire's discretion:
An INFIELD FLY is a fair fly ball (not including a line drive nor an
attempted bunt) which can be caught by an infielder with ordinary
effort, when first and second, or first, second and third bases are
occupied, before two are out. The pitcher, catcher and any outfielder
who stations himself in the infield on the play shall be considered
infielders for the purpose of this rule. When it seems apparent that a
batted ball will be an Infield Fly, the umpire shall immediately
declare "Infield Fly" for the benefit of the runners. If the ball is
near the baselines, the umpire shall declare "Infield Fly, if Fair."
The ball is alive and runners may advance at the risk of the ball
being caught, or retouch and advance after the ball is touched, the
same as on any fly ball. If the hit becomes a foul ball, it is treated
the same as any foul. If a declared Infield Fly is allowed to fall
untouched to the ground, and bounces foul before passing first or
third base, it is a foul ball. If a declared Infield Fly falls
untouched to the ground outside the baseline, and bounces fair before
passing first or third base, it is an Infield Fly. -- MLB
What constitutes ordinary effort? How do you define that as anything but an arbitrary judgement for the umpire?
The point of the infield fly rule is to protect runners from a forced double play if a infielder purposefully drops an easy pop-up. This is why the rule is written so loosely and is generally left up to the umpire to define as to what "can be caught by an infielder with ordinary effort."
Technically, any ball hit high enough could be an infield fly even if it goes as far as the warning track. However, in this case, the umpire simply made a bad call. Balls that fly that deep are rarely called an infield fly.