I have tried different hand formation and tried different positions of hand to hit the ball, but the smash either goes out of court or is very low powered.I just don't get how to hit it. So that its high powered and stays inside the court?
Spiking in volleyball is much more about timing and approach than hand placement. I would first focus on learning when to jump and swing in regard to the set and flight of the ball. If you can't contact the ball at the right time, then it doesn't really matter what you hit it with. Personally, after practicing approach and swing, the "hitting" aspect seemed to just fall into place, but I know that's not really an acceptable answer to your question.
If you check out this site, it gives a step-by-step approach of how to spike a volleyball. For the most part, I agree with everything that it suggests. Step 8 & 9 instruct:
(8) ...hit the ball, contacting the upper half of the ball with the heel of your open hand at the very top of your swing
(9) Snap your wrist downward to impart topspin and direct the ball down into your opponent's court while spreading your fingers
If you feel like you've already got your approach down, try these suggestions, I think they'll work for you.