are there any rules against taking a catch with the hand without the Mitt on it?
There are no rules against catching a ball in flight with the hand. See Rule 2.00 below.
Will the batter still be given out?
Yes. See Rule 6.05 below.
Rule 2.00 in the 2013 MLB Official Baseball Rules states (emphasis mine):
A CATCH is the act of a fielder in getting secure possession in his
hand or glove of a ball in flight and firmly holding it; providing he
does not use his cap, protector, pocket or any other part of his
uniform in getting possession. It is not a catch, however, if
simultaneously or immediately following his contact with the ball,
he collides with a player, or with a wall, or if he falls down, and as
a result of such collision or falling, drops the ball. It is not a
catch if a fielder touches a fly ball which then hits a member of the
offensive team or an umpire and then is caught by another defensive
player. In establishing the validity of the catch, the fielder shall
hold the ball long enough to prove that he has complete control of the
ball and that his release of the ball is voluntary and intentional. If
the fielder has made the catch and drops the ball while in the act of
making a throw following the catch, the ball shall be adjudged to have
Rule 6.05 states:
A batter is out when -
(a) His fair or foul fly ball (other than a foul tip) is legally caught by (see CATCH in Rule 2.00) a fielder