I'm still working on this one . . . so I'll just babble . . .
I having trouble controlling the variable.
You see, the rectangle can be "tipped" at a variety of angles.
* * * P2
*| o o|*
| o |
P8o | |o*
* | | *
*o| | oP4
| o |
*|o o |*
P6 * * *
The area of the octagon would be:
Edit: You're right, RubyRed . . . I overlooked the word "maximum".
Yes, the top and bottom triangles are congruent,
. . as are the left and right triangles.
If my diagram is hard to read . . .
Points are the vertices of the square,
. . which is clearly outlined.
Points are the vertices of the rectangle,
. . outlined with o's (harder to see).