I used no calculus in my answer!

Assume that the rectangle inscribed has the largest area. Now, let us try to show that (where I let be the area of the scalene triangle)

If you drop down a vertical line from the top of the triangle to its base, wrt the rectangle, you will create similar triangles all around it.

Consider the two triangles on the right side of the vertical line. Each of the smaller triangles have a height of (which is what I called the height of the rectangle) Since there are two of these triangles, let us assume that the vertical line has a length of .

Now, let us consider the whole scalene triangle and the similar scalene produced above the rectangle. The base of the smaller triangle is (what I called the length of the rectangle). By similar triangles (where b is the base of the original scalene triangle.)

Thus, the area of the original scalene triangle is . Since we're told that the area is constant, we can then say that

Now, since I let be the length of the rectangle and be the width of the rectangle. Thus, its area is . But we saw that . Therefore, the area of the rectangle is

-------------------------------------------------------------------------

After I finished typing this, I realized how to use calculus...but be prepared, for it will get messy... ><

Now, using calculus:

Like we did in the geometry approach, drop down a vertical line from the apex to the base. Let the base and the vertical line be perpendicular, and let the intersection of the base and vertical line represent the origin of the Cartesian system. See the figure below.

Now, let the vertical line have a height of . Since we have created two triangles, let the triangle to the right of the vertical line have a hypotenuse of length , and let the triangle to the left of the vertical line have a hypotenuse of length . Let the base of the rectangle to the right of the vertical line have a length of and the base of the rectangle to the left of the vertical line have a length of . Now, to find the bases of these triangles, apply the Pythagorean theorem. Thus, the base of the triangle to the right of the vertical line will have a length of , and the base of the triangle to the left of the vertical line will have a base of length .

Now, let us create equations of line segments to represent the height of the rectangle ( ) as it varies along and .

Focusing on the segment with length :

Let us assume that the y-intercept of the line is . Note that when , the segment intersects the base ( ). So we can conclude that the slope of the line segment is . Thus the equation of the line that represents as varies with respect to is .

Focusing on the segment with length :

Let us assume that the y-intercept of the line is . Note that when , the segment intersects the base ( ). [Also keep in mind that is in the region of in the cartesian system. Since I want to make , when I find the equation of the line, although the slope is positive, maintains the negativity.] So we can conclude that the slope of the line segment is . Thus the equation of the line that represents as varies with respect to is .

Now that we have all this information, we can set up the area equations.

Let the area of the rectangle to the right of the vertical line be represented by . Thus, . Now, we maximize the Area. We get . To maximize, we must have

Since this is the maximum value of the base, then the maximum height must be

Similarly, let the area of the rectangle to the left of the vertical line be represented by . Thus, . Now, we maximize the Area. We get . To maximize, we must have .

Since this is the maximum value of the base, then the maximum height must be

Therefore, the total area of the rectangle is .

But, the height of the triangle is and the length of the base is . Since the area is constant, we see that .

Therefore, the maximum area of our rectangle

I hope you can follow this. If you have any additional questions regarding this, please let us know.

Again, I have given you two different approaches. Hopefully, you can make sense of this.