From the rulebook:
Section 5 Position of Players at the Snap
Article 1: Offensive Team. The offensive team must be in compliance with the following at the snap:
(a) It must have seven or more players on its line (3-18); and
(b) All players who are not on the line, other than the receiver of the snap under center, must be at least
one yard behind it.
(c) No player may be out of bounds.
An illegal formation for the offense would be one where there are not 7 men on the line. There are no restrictions on the defense's formation during normal play, however the defense can be penalized during special teams play:
During a punt, a field-goal attempt, or a Try Kick, a defensive
player, who is within one yard of the line of scrimmage at the snap,
must have his entire body outside the snapper’s shoulder pads.