Let A=b/a, launch the ball so that the slope of its trajectory is k*A, where k is an irrational number. Then this ball won't get back forever.

It's quite easy to prove. Reflect the trajectory with respect to the side when it hits a side, you will get a straight line, as if the ball penetrate through the side. If the ball gets back in the real world, this imaginary straight line will end up at a point where m*a units are travelled horizontally while n*b units vertically, where both m and n are integers. Then the slope of this straight line is n*b/(m*a)=(n/m)*A, that is, a rational multiple of A.