Let $\displaystyle BC = x$ and $\displaystyle AB = y$. The area of triangle BCD is $\displaystyle \frac{x^2 \sqrt{3}}{4}$, and the area of ABDE is $\displaystyle xy$. Hence the area of the pentagon P is
$\displaystyle [P] = \frac{x^2 \sqrt{3}}{4} + xy = \frac{x^2 \sqrt{3} + 4xy}{4}$
However you know that the perimeter is 10, so $\displaystyle 3x + 2y = 10 \Rightarrow y = \frac{10-3x}{2}$. Substitute into the area equation to obtain
$\displaystyle [P] = \frac{x^2 \sqrt{3} + 4x(\frac{10-3x}{2}))}{4}$
Simplify, and differentiate both sides with respect to x and find critical points.
Suppose that the vertical sides of the rectangle each have length $\displaystyle a,$ and suppose that the other sides have length $\displaystyle b.$
The area of the pentagon is
$\displaystyle A = ab + \frac12b\left(\frac{\sqrt3}2b \right)$
$\displaystyle \Rightarrow A = ab + \frac{b^2\sqrt3}4,$
and the perimeter is
$\displaystyle P = 2a + 3b = 10$
$\displaystyle \Rightarrow b = \frac13\left(10 - 2a\right).$
Substituting this into the area equation above produces
$\displaystyle A = a\left[\frac13(10-2a)\right] + \left[\frac13(10-2a)\right]^2\frac{\sqrt3}4$
$\displaystyle = \frac19\left[\left(\sqrt3-6\right)a^2 + \left(30 - 10\sqrt3\right)a + 25\sqrt3\right]$
Differentiating,
$\displaystyle \frac{dA}{da} = \frac19\left[\left(2\sqrt3 - 12\right)a + 30 - 10\sqrt3\right]$
We locate the critical value:
$\displaystyle \frac{dA}{da} = 0$
$\displaystyle \Rightarrow\frac19\left[\left(2\sqrt3 - 12\right)a + 30 - 10\sqrt3\right]=0$
$\displaystyle \Rightarrow\left(2\sqrt3 - 12\right)a = 10\sqrt3 - 30$
$\displaystyle \Rightarrow a = \frac{10\sqrt3 - 30}{2\sqrt3 - 12} = \frac{5\sqrt3 - 15}{\sqrt3 - 6}$
$\displaystyle \Rightarrow a = \frac{15 - 5\sqrt3}{6 - \sqrt3}$
Now you can find $\displaystyle b.$