When I started swimming seriously, about 4 years ago, I struggled with exactly the same problem. It took me a long time to break the 100m barrier for front crawl, doing bi-lateral breathing. Uni-lateral breathing was a little better but I still struggled with a sensation of panic and being out of breath.
Once I had broken the 100m limit, suddenly it became possible to go much farther, as if there is a transition between how I would breath normally on land and how I needed to breath whilst swimming. I still feel a slight pinch during training warm-up at this point, as if my lungs 'change gear'.
To beat the limit I :
- slowed my swimming to a very gentle pace so as to be as relaxed as possible
- repeated 50m swims at first, with a break between each to recover, then 100m swims with a break
- eventually, I felt I could attempt to go past the 100m and try for 150m
Once I was getting to 150m, I very quickly made it to 200, 300 etc. One of the key things that helped me was to beat the barrier was expelling 50% of each breath immediately after inhaling, ie. as my face re-entered the water, I already started to breath out. The remainder of the air I slowly breathed out, through nose and mouth simultaneously, during the time my face remained under water, such that after 3 strokes it was all gone and ready for the next inhale.
As others have indicated here, breathing out is very important; when swimming you should be either breathing in or out but not holding your breath. My coach explained the panicy feeling you get as being the need to expel the C0-2 from your lungs following breathing in, which is not something you'd instinctively think to do.