I presume that y is the height the ball bounces to?

y = 5*|e^{-at}*sin(3t)|

We want to find the a value such that y(max) = 3. What then, is the maximum height of the ball for variable a?

Of course you can do this with Calculus, but just look at the function y(t) for a moment. The e^{-at} is going to modify the sine function such that the magnitude of the max and min y values are going to be reduced as time goes on. (ie. The envelope of the sine function decays exponentially, if you understand the terminology.) So the largest maxima value for y will be at the first maxima value for sin(3t), which is at t = (pi)/6 s. So this is the t value for the maximum value of y.

Thus

y(max) = 5*|e^{-a*(pi)/6}*sin(3*(pi)/6)| = 5*e^{-a*(pi)/6}

which we need to set to 3 m for the largest possible a value. So solve:

3 = 5*e^{-a*(pi)/6}

e^{-a*(pi)/6} = 3/5

-a*(pi)/6 = ln(3/5) = ln(3) - ln(5)

a*(pi)/6 = ln(5) - ln(3)

a = 6/(pi) * [ln(5) - ln(3)] = 0.267468 (or so)

Any larger a value will produce a first maximum in y that is less than 3 m. (You can also check this with Calculus, but you should be able to use simple logic to see this.)

-Dan