As you said, Los Angeles is the second largest city in the US, so it seems ridiculous that they do not have an NFL team. There are lots of opinions as to why this is the case.
In the last few years that the Rams and the Raiders were in LA, both teams were struggling, both on the field and financially. They both played in aging stadiums, and neither were successful in obtaining a new stadium in the area.
The reasons why both teams struggled financially are debatable. One reason could be that there aren't enough football fans to support two teams. (They both left the same year, so this is hard to disprove.) Another reason could be that, with 6 other major league sports teams in the area, the sports market is already saturated.
Any team that would move to LA would need a new stadium; a team would not move there to play at the Coliseum, the Rose Bowl, or any other old stadium in the area. This is a hurdle, as both the Rams and the Raiders were unable to get a stadium built. As time passes, it will be more likely that funding for a new stadium will be able to be obtained.
Today, LA is used as a threat by NFL teams seeking new stadiums. The Buffalo Bills are one of the latest teams to consider LA. Tim Graham, writing for the Buffalo News, wrote an article outlining reasons why he thinks that the time is not right for an NFL team in LA. Among these reasons are an unfavorable business climate in California and a lack of public enthusiasm in funding a new stadium.
It seems to happen every year: a team threatens to go to LA if they don't get a new stadium, and the public funds the stadium to prevent the team from going to LA. At the end of this upcoming season, the Rams, Raiders, and Chargers, all of whom have been in LA in the past, have expiring stadium leases. They will all undoubtedly use LA as a bargaining chip in their stadium negotiations. If not one of these teams, it seems inevitable that at some point in the future, a team will move to LA to give it a try.
Update 1/2016: The Rams, Chargers, and Raiders all applied to move to LA this year. The NFL has approved the move for the Rams, who will start playing in LA this year (2016) at the Coliseum while their new stadium in Inglewood is built, which is scheduled for completion in time for the 2019 season. The stadium is designed to support two teams; the Chargers will be given an opportunity to move, and if they decide against it, the Raiders will be allowed to move. Source: LA Times
Update 1/2017: The Chargers opted to move to LA, and will begin playing at StubHub Center in Carson until the Inglewood stadium, which is now named City of Champions Stadium, is completed in 2019.
