I remember that according to defined laws/rules by FIFA, referee is the only one who decides which side to choose for the corner under the situation you described.
For the comments about 'who cares which side for the corner', sometimes which side to use for the corner really matters, for example, when it's a overwhelming windy day or if the sun is shining from one side line of the field. According to the widely recognized principle of encouraging offensive side, normally the referee should/would choose a side which benefits the offensive side under this situation when which side to use for corner does matter.