First you have two questions here. The first regarding circumstantial analysis about when to punt and so forth. ESPN has done many simulations on when the computers say you should punt, kick, or go for it. To summarize if I remember correctly it was basically go for it until it is over 4th and 10 past your 30. Inside your 30, punt on 4th and 5 or longer. FGs were a trickier stance because you have to look at chance to make a FG but then % you will get the first down and also score a TD. Basically kicking on 4th and 5 or longer and maybe allowing a few more yards for longer FGs. On ESPN Greg Easterbrook has had a lot of simulations. As you said punting leaves lot of potential points on the table according to the analysis.
However... This is not baseball. Punting is very comparable to bunting by the stat heads. But baseball is a series of events that have little impact on one another. Football is the opposite. This is where this type of historical statistical analysis breaks down.
Example: 1st quarter and the other team just marched down the field and scored a TD on a 6 minute drive. You get the ball on your 30. Three plays later you are sitting on your 35. Numbers say go for it. The numbers don't understand that your offense isn't clicking yet. Numbers don't take into account that if your defense goes right back out, they will be shell-shocked seeing the team start with the field position. Also the other team would be on an emotional high getting the ball there. So if you do get the first down? Well that doesn't guarantee that you will get another yard on the drive.
Really in football you have two factors around this analysis that is almost impossible to control. Fatigue and emotion. These are real things.
My own opinion on this is somewhere in between never punt and the current status quo. I think it is more about match up and situation. Now does it make me cringe 99.9% of the time I see a coach at any level punt the ball on the opponents 45 on 4th and 2. YES! Unless you have no offensive ability it never makes sense to give up the ball and probable scoring chance for 2 yards.
Second question - sabermetrics
Every NFL team has "sabermetrics". Some much more advanced than others. Even keeping stats for teams I coach we grade every player each play. For defensive lineman for example we have a zone rating (player controlling zone), leverage rating (how well does player shed block or control offensive lineman), depth rating (our lineman are getting up field), blow-ups (causing play to jump backwards), and so on.
I get great real numbers I can quantifiably show that player X is outperforming player Y. These stats are great when player Y looks like Hercules and everyone thinks he should play and player X looks average but performs consistently. What I have found is that most coaches play kids based on their maximum potential - and this happens in the NFL too. I can tell you this - less than 2% of players play all out all the time. So these stats are great.
But the flaws... First it is next to impossible to get these stats without being the coach. To rate someone you must understand both the play and what they have been taught as their responsibility. You must also understand if other players interfered with a player (which happens ALL THE TIME in football - and yes we keep that as a stat too). The second is that really this person needs to be the same person doing stats all time and can do so in a fair way.
Let's take offensive lineman as a sabermetrics example. Let's say you have a guy who is firing off the ball and just destroying guys so you want to measure speed off the ball as a positive trait. Well I have coached guys who fly off the ball and guys who are more about technique and impact points. So again we have to figure out if a metric is "positive" then if a player doesn't do a positive metric well then he should be performing below average or you have a bad metric.
Again the sabermetrics in football are all over the place. KC Joyner has some and I laugh when I see some of his stats and some of his stuff is great. Football Outsiders does some advanced metrics too. This isn't baseball though. One guys trash could be another guys treasure because they are in opposite schemes and have different wants. So I think some of the advanced metrics will show up on more sites but really the numbers that matter are how you are grading out in-house.
Addendum: Thinking about this I had an extra thought. First I would like to accept my pro-don't-punt bias since I have coached for years and feel like my play calling forms an advantage (but don't all coaches feel this way?).
The point that I didn't include in my analysis is time. I find that most people compare these types of situations as to being at a blackjack table. If you have a 51% chance of winning when hitting on 16 you should always do it. But if you are sitting at a blackjack table you have unlimited time.
During a football game if you are on the wrong end of a few of these situational outcomes early in the game your team is behind the 8-ball. So if you find yourself down 28-7 at half instead of 14-7 based on two fourth down fails your game plan for the second half is much more limited. The defense knowing that you have to pass more is at an advantage.
So going back to the blackjack example it would be like saying you will always hit on 16 but after 1 hour your odds of winning on 16 is 40% per hand. The time factor is a big deal for teams with more talent. They don't want to shoot themselves early in the game when they know they have a better than 50% chance of winning every drive. A team that lacks talent could definitely use a higher scoring deviation to increase its odds though.
