How much progress have you made with this problem?
Let x and y be the sides.
x will be the side of the rectangle along the triangle side of length 5, and y will be the side along the triangle side of length 12.
Identify the right triangle (lower right if side of length 5 is the base) with legs (5-x) and y. This triangle is similar to the big right triangle, so the following ratio is valid:
Solve for y:
The objective function is the area of the rectangle:
Write this area as a single variable function by using substitution (*) above, then apply differential calculus techniques to optimize.
I hope this helps.