Given: Need to find the maximum area of a rectangle inscribed in a right-triangle with legs with lengths 6 and 8 respectively.
Find an equation for the boundary lines:
Now, find the equation of the area of the circle:
Now find the vertex
Substitute into the equation for area to get your answer (I'll let you do it yourself )