From the 2013-2014 FIFA Laws of the Game, p. 120
...
A goalkeeper is considered to be in control of the ball:
- while the ball is between his hands or between his hand and any surface
(e.g. ground, own body)
- while holding the ball in his outstretched open hand
- while in the act of bouncing it on the ground or tossing it into the air
When a goalkeeper has gained possession of the ball with his hands, he cannot
be challenged by an opponent.
...
Restart of play
- Indirect free kick from the position where the offence occurred
(see Law 13 – Position of free kick)
Considering your scenario, if the goalkeeper has as much as a finger on the ball and it's also touching the ground, the post, another part of his body, or even the opponent (all of these are surfaces), he cannot be challenged. An indirect free kick should be awarded to the goalkeeper's team.
If it's not clear as to who made contact with the ball first, most referees will let play continue, as any infringement is doubtful.