Having refereed both sports, I'd like to offer another possible  answer. The laws of rugby support the referee as being the sole 'judge of law and fact'. This places the referee in an esteemed role which is backed up (at least most of the time) by the administrators of the game. Football (soccer) does not have such an explicit status and protection for referees - and the administrators treatment of referees suggests they are largely ambivalent about their role. Because of this refereeing football, at least to me, was a chore and rugby a relative delight - despite the rules of the latter being more complex.