Its not your fault. According to newtons first law " Objects in motion stay in motion unless acted upon by an outside force."
Force from the swing
Look at this diagram. The force from your swing is pushing the ball forward while the force from gravity is pulling it down but, the force of gravity is greater than the force of your swing.