The highest rated quarterbacks in football appear to release the ball just seconds after the snap. Sophisticated pass rushing and secondary coverages give the offense few opportunities to advance the ball on longer routes to gain yardage. Also, scrambling is in decline as QBs take a few steps back and throw.
I think there is a push to get rid of the ball quickly. Due to the reasons you stated. However there are successful QB's that have slower release times than the league average. This article talks about the recent stats in which Peyton Manning and Tom Brady release the ball quickly. I think that translate to success, but there are players that hold the ball longer that have success too. I think it partly has to do with running or scrambling ability.
A quarterback that scrambles around for a few seconds and then throws the ball, and does this multiple times a game will increase his average release time. Even though on other plays he might get rid of the ball as quick as some of the quickest guys. Players like Peyton Manning and Tom Brady do not use their legs at all, even to bide time. If they don't get rid of the ball quickly, then they wont be able to extend the play. Players like Aaron Rodgers and Drew Brees are not necessarily on the same level as RGIII, Cam Netwon, Colin Kapernick, etc., but they can use their legs to extend the pocket or extend the play.
Lastly you have to consider the team as a whole. Can the quarterback and his receivers pick up the defense preplay and decide where the ball is going? Once they decide this can the still be on time? There are a lot of factors that go into it. I think along with ability to run and scramble there is experience and the team you have around you.