Interesting, this depends on what kind of boat you're in.
I'm going to guess that you're in a sit-on-top or something similar and/or have a much heavier person in the back?
It this case the turning is probably down to the wind acting directly on the boat.
The easiest thing to do would be to adjust your trim (your front/back weight distribution), say by switching who goes in the front and who goes in the back. This should lower the nose and raise the tail, meaning that there's a bit less surface area at the front to be acted upon by the wind and more of the front in the water.
FYI: In a sea-kayak or similar you'd expect the boat to be turning the opposite way. this is because (assuming the boat is normally trimmed) the wind acts equally all along the side of the kayak. However because you're moving forward the water exerts more force on the sides of the front than at the back, meaning that when the wind pushes against the front it has more to push against, causing the back of the boat to swing out with the wind (ie. opposite to what's pctured above. Sea-kayaks/touring boats you will often have a skeg (a small flap that can be dropped down from the stern as needed, but can't have it's direction changed like a rudder), this prevents the wind from pushing the back out. Rudders are also sold for directional control but are unnecessary, don't help control direction in wind better than a skeg and usually inhibit newer paddlers developing.