In football, the referee and assistant referees are considered to be part of the field, just like the corner flags, crossbars and goalposts. In the case of the ball hitting the referee, there are two passages that support play continuing.
On page 33 of the 2014/15 FIFA Laws of The Game:
The ball is in play at all other times, including when:
- it rebounds off a goalpost, crossbar or corner flagpost and remains in the field of play
- it rebounds off either the referee or an assistant referee when they are on the field of play
Also, on page 106:
If, when the ball is in play, it touches the referee or an assistant referee who is temporarily on the field of play, play continues because the referee and the assistant referees are part of the match.
As for a referee contacting a player, there is nothing explicitly written in the Laws about this. However, the only way a free kick or penalty kick can be awarded to a team in football, is if a player, substitute or substituted player on the other team commits an infringement, so this is out of the question.
As for a potential dropped ball restart, this would only occur if the player and/or referee was injured as a result of the collision. Page 31 covers when a dropped ball is appropriate:
A dropped ball is a method of restarting play when, while the ball is still in play,
the referee is required to stop play temporarily for any reason not mentioned
elsewhere in the Laws of the Game.
As the referee is not required to stop play, it is not stopped and hence play continues.