The best thing to do here is DEFINITELY draw a picture.
Thats the extent of my paint skills, but you can see that if we have an equilateral triangle, we can imagine one half of that triangle, with the base positioned on the X-axis. What we want are the dimensions of that rectangle that will give us the largest area. We know that to be the optimization of the formula A=(length)(width), but what the heck function are we using to define these? For our length it is clear that it is going to be some value between 0 and 5 non-inclusive (because then our area is just 0). That means our length should be:
But what should our Y be? Well look at one of the sides of that triangle. Doesn't that look like a line? A line that has points and . From this we can form an equation that defines that line, which will allow you to define the height of your rectangle in terms of X. From there, you will be able to get your equation for the area of a rectangle, and you should then be able to optimize it.
Do you think you can take it from there?