Call x the side length of the square. Since the two pens have to share a common side, that means one side length of the rectangle will also be x. Call the other length y.
Since you know that you have 60m of fencing, that means 5x + 2y = 60. From here we can get y = -(5/2)x + 30.
Now get an expression for the total area and use calculus to find the maximum (i.e. derivative = 0, second derivative negative at that point).