The area of the rectangle is: .\(\displaystyle A \;=\;xy\) .[1]

From the similar right triangles, we have: .\(\displaystyle \frac{3-y}{x} \:=\:\frac{y}{4-x}\) . . which simplifies to: .\(\displaystyle y \;=\;3-\tfrac{3}{4}x\) .[2]

Substitute [2] into [1]: .\(\displaystyle A \;=\;x\left(3-\tfrac{3}{4}x\right)\)

And you must maximize: .\(\displaystyle A \;=\;3x - \tfrac{3}{4}x^2\)