Sometimes the referee will flag a team for having an "illegal formation". What are the restrictions on how a team is allowed to line up?
Section 5 Position of Players at the Snap
Article 1: Offensive Team. The offensive team must be in compliance with the following at the snap:
(a) It must have seven or more players on its line (3-18); and
(b) All players who are not on the line, other than the receiver of the snap under center, must be at least one yard behind it.
(c) No player may be out of bounds.
An illegal formation for the offense would be one where there are not 7 men on the line. There are no restrictions on the defense's formation during normal play, however the defense can be penalized during special teams play:
During a punt, a field-goal attempt, or a Try Kick, a defensive player, who is within one yard of the line of scrimmage at the snap, must have his entire body outside the snapper’s shoulder pads.
Sorry I can't comment on
wax eagle's answer, as I don't have 50 rep; I did, however, want to note that in 2013 the NFL added formation requirements to the Defense:
ILLEGAL FORMATION: The defense cannot have more than six players on the line of scrimmage on either side of the snapper during kicking plays. This rule is designed to protect players who are in a vulnerable position. Violations of this rule will result in a five-yard penalty.
Some additional ways to view valid formation for offense at different levels:
Fewer than 7 players line up on the line of scrimmage (NFL/High School); more than four players in the backfield (NCAA only); eligible receivers fail to line up as the leftmost and rightmost players on the line in the NFL; or when five properly numbered ineligible players fail to line up on the line.