As colleges can't sign a new player like an NFL team can, what do they do if two or three of their starting QBs go down? Is there a rule that lets them get new players or do they have to play their position players?
Currently, the NCAA limits teams to 85 scholarships. The teams may recruit any number of scholarship quarterbacks. In addition, teams can also invite players to walk-on, so there are likely to be more than 3 quarterbacks on each team's roster. There is a cap of 105 players on a roster for Division 1 FBS teams:
Limit on Number of Participants—Bowl Subdivision. [FBS] In bowl subdivision football, there shall be a limit of 105 student-athletes who may engage in practice activities prior to the institution’s first day of classes or the institution’s first contest, whichever occurs earlier. Exception—National Service Academies.
However, teams are limited in the number of players that can be suited up for each game - these rules are set up by conferences:
Many Conferences such as the Big 10, ACC and others have set conference travel limits to 72 players While the WAC limits its travel squads to 64. The NCAA Football Playing Rules do not address this issue as it is an NCAA administrative regulation not a playing rule. The NCAA By-laws have changed over the years. At one time it was 48 squad members as the limit for the traveling team. it has changed over the years from unlimited to 100 to 80 and down to 60. I did not find the limit in the 2011 by-laws. I suspect the travel limit is 60 players with up to 72 to a conference championship. The Conferences each have their own rules for travel limits.
In practice, if multiple quarterbacks go down, the teams will still need to field eleven players on offense. The replacement quarterbacks can be existing quarterbacks on the roster (either scholarship or walk-on players) or other position players.
Due to the sheer number of players that NCAA teams may keep on a roster (105, as mentioned in another answer), teams will often carry an excess number of quarterbacks - typically 2-3 (redshirt) freshman and then 2-3 upperclassmen. In my experience, the college team I played on and the other teams we played against always had at least 4 quarterbacks in their full roster.
Even after the quarterback position dries up, you typically have other players on the team who may have played quarterback in high school who might be used or considered an "emergency" QB type of player if, for some freak reason, all dressed quarterbacks for a given game become unavailable to play due to injury or some other circumstance.