Let the side that lies on the leg of length 8 be of length x.
Let the side that lies on the leg of length 6 be of length y.
Then A = xy.
Now note that there are two smaller right-triangles on two sides of the rectangle. These triangles are similar to the 6-8-10 right-triangle.
So you have a choice of restriction:
Use one of these restrictions to write A in terms of a single variable. Now apply calculus to find the answer.