I don't really know what's going on in your program, but from your description it sounds like your keyboard input is directly controlling the speed of the ball in some way. Is it being influenced by the keyboard repeat rate or something? If so, that's not a good idea in a realtime game.

I suggest you have a single timer which controls all game activity and redraws the screen (it's only Pong, no need to get sophisticated!). Every time the timer fires, you move the paddles and the ball, then redraw the screen. The keyboard events shouldn't move the paddles or update the screen directly - instead, when you receive a keyboard event, make a note of what it was and utilise that information the next time the timer fires.