It seems to me that the likelihood of refs getting a fumble recovery call right does not depend on which team recovers it. Sure there are simple fumbles where there's no scrambling for the ball and most of those are offensive recoveries but those reviews will be real quick.
There are two aspects to consider in a play with a possible fumble: 1. was there a fumble or was the ball carrier down? and 2. if there was a fumble, who recovered it? Now think about how often the on-the-field ruling is correct for each of those aspects--how often are they overturned?
I can't find any data to back it up, but I think that when there is a fumble, the ruling of who recovers it is correct a very high percentage of the time (or at least is rarely overturned). The ruling of whether or not a player is down before losing the ball is more likely to be a close call, unclear, and sometimes overturned.
When a fumble is ruled on the play, either the defense is ruled to recover it (the impact on the game is very high) or the offense is ruled to recover it (low impact on the game compared to if there had been no fumble--maybe a little field position but that's it). Assuming that the officials are usually right about who recovers a fumble, any review is more likely than not going to be focused on whether there really was a fumble in the first place. For the high impact plays where the defense recovers, this is worth looking at. For the low impact plays where the offense recovers, it's maybe not worth the time to look at all of them and coaches are still able to challenge the play if they think the officials got it wrong.
So the important point is that officials are more likely to have been correct in deciding which team recovered a fumble than they are to have been correct in ruling whether a fumble took place. When you take that assumption, these plays are only then worth reviewing when the (assumed) "correct" call of the officials is a recovery by the defense because that play has real significance on the outcome of the game.
All turnovers are reviewed. Your logic linking fumbles and turnovers is flawed.
Why not review every time a DB might have intercepted the ball using the same logic?
The NFL already has guidelines for this. Each team has two challenges and a third if their first two were right. I cannot remember one game since replay started that a coach challenged three times and won and there was another blown call.
Why aren't all fumbles reviewed? Because it would be agonizing having to stop the game every time there was a bad snap or missed hand off.
Automatically reviewed plays are plays that have clear, significant impact on the game. As Coach-D noted, you can't review everything - it would take too long. As such, turnovers and scoring plays are subject to automatic review, because they have significant impact on the game.
What you're sort-of-asking, I think, is that since this is a potential turnover, why isn't it reviewed if it's ruled not a turnover? The same question is possible for almost-scoring plays: if a runner is ruled down at the 1 foot line, there's no review even if he might have scored (unless the coach throws his flag). The answer is simply that they aren't because they're not ruled as turnovers or scoring plays. That's what the coach's flags are for.