I think it has to do with the traditions. I believe originally rugby was played by the higher class, with a certain upbringing. I have a colleague who works as a part time rugby ref and he's puzzled as to why the ref doesn't send off half of the players in a football game. I, on the other hand, am often puzzled as to how the refs command so much respect in rugby.
Another point of view could be that football's expansion to take over the world is based on the fact that one really doesn't need much to play football. Compared to some other sports (e.g. ice-hockey), it's cheaper and easier to play football, all you need is a ball (or something that you can use as a ball) and a good imagination (to define the field and goals). "What does that have to do with anything?", you might ask.. Well, if anyone and everyone can play football, however they like, you might argue that people take their local manners and traditions and incorporate it into the game. I play football on amateur level, and the "quality" of the game especially with regards to manners and temperament has been very closely related to which nationalities were present on the field.
Finally, and perhaps most importantly, football has become such a huge money-maker so that football associations do not dare/want to impose serious penalties for inappropriate behavior. I mean there isn't a single person in the football world that could imagine giving a 10 match ban to Lionel Messi or Cristiano Ronaldo, for diving. Or a seasonal ban to whatever player for shouting at the ref's face. I believe the situation has gotten this bad because bad behavior has been tolerated for just way too long.