Units??

Let the side that lies on the leg of length 8 be of length x.

Let the side that lies on the leg of length 6 be of length y.

Then A = xy.

Now note that there are two smaller right-triangles on two sides of the rectangle. These triangles are similar to the 6-8-10 right-triangle.

So you have a choice of restriction:

or

Use one of these restrictions to write A in terms of a single variable. Now apply calculus to find the answer.