Generally in Rugby Union when a player knocks the ball forward from their hands without making a clean catch, a scrum is called for a knock on. But when a player kicks the ball and the ball is knocked down by an opponent, no knock on is called and they are able to gather up the ball. Normally a kick like this occurs as a defensive action close to the try line, so a charge down often results in a try.
Law 12 of the Laws of Rugby Union specifically exclude a charge-down that happens immediately following a kick as being called a knock-on or throw forward.
I think this has been a law in Rugby for a long time and is not a recent innovation. Before 1958 it was even possible to rush the the kicker and charge down a penalty kick from immediately the time the ball was placed on the ground. To avoid this, under these older laws another person used to place the ball as the kicker ran up - a bit like how someone places the ball for the kicker taking a field-goal or a point-after in American Football.
So the charge down has been with the game for a long time, but why? I think as Harley Holcombe's answer says, that this is to allow pressure to be put on the kicker. It allows someone rushing the kicker to have an opportunity to gain regain possession, otherwise the only option would be to try to tackle the kicker. This might be to reduce injuries in the kicker, but more likely just adds excitement and unpredictability in taking tactical kicks - re-balancing the risk and reward for taking a tactical kick. Mostly it fails as kickers are wary of the tactic, but it has worked well for England in the recent Six Nations championship.
I believe this rule was brought in (to rugby league at least) to encourage charge downs. This puts more pressure on the kicker and encourages the tackler to attack the ball, rather than hitting the kicker, who may not be ready for it and could get injured.
Another reason to have the chargedown rule is that a successful chargedown can add excitement, check out some examples on YouTube.
The question "Why is a charge down not regarded as a knock on in rugby union?" can be answered as above, by quoting law 12. But my answer to the question relates to the principles of the game as well as the laws.
There are two kinds of infringements that can occur on a rugby field, technical and intentional. Technical infringements are usually sanctioned with a scrum or free kick, intentional infringements are sanctioned with a penalty and possible admonishment of a player and a sending off.
So, is the charge down of a kick an infringement at all? If so, is it a technical infringement or intentional? Or, is a charge down of a kick a skill specific to rugby union? (and other codes such Australian rules, Gaelic football and rugby league).
Has an infringement occurred around the ball, the ball carrier or team mate of the ball carrier? No.
What has happened is an opposition player has, with good timing and skill and bravery, managed to get their body in the way of the flight of the ball and, through no act of foul play or other illegality they may benefit from this action.
A simple rule of thumb when refereeing a game of rugby union. Penalize negative actions, reward positive actions.