Prior to 1995, all referees for international matches (men's and women's) came from the same list. In 1994, four female assistant referees made it onto the list for the first time.
In 1995, FIFA decided to keep two separate lists of referees - one for women's football (that only female referees were eligible for) and one for men's referees.
In the 1999 Women's World Cup, all officials at the tournament were selected from the women's list for the first time - previously there weren't enough on the list and the vacant spots had to be filled by referees from the men's list.
I've grabbed this historical information from an article on FIFA's website
From my own background as a referee, I'm aware that most national associations (particularly in Western nations) try to ensure that for national women's leagues, female referees are chosen first. Unfortunately, there are still not enough female referees to cover all of the games, so some of the spots, particularly for assistant referees are filled by developing male referees.
In the lowest levels of the game, there is a complete shortage of referees, so generally women and men referee both women's and men's football. There aren't enough female referees to solely cover the women's games, and the standard of play in the men's football is poor enough that the average fitness/speed disparity between men and women doesn't even really come into play.