Ellen:

How much progress have you made with this problem?

Let x and y be the sides.

x will be the side of the rectangle along the triangle side of length 5, and y will be the side along the triangle side of length 12.

Identify the right triangle (lower right if side of length 5 is the base) with legs (5-x) and y. This triangle is similar to the big right triangle, so the following ratio is valid:

Solve for y:

(*)

The objective function is the area of the rectangle:

Write this area as a single variable function by using substitution (*) above, then apply differential calculus techniques to optimize.

I hope this helps.

Good luck!