1. I think you need to integrate the MC function to determine the total cost (TC) function. Then your answer should be TC(b) - TC(a). The marginal costs are per-unit costs; you seemed like you wanted the total cost change.
2. If the amount increases by 500%, it means you're going to end up with 600% of what you started with. So I think the formula you want is:
(1+r/12)^180 = 6, which, if I did the math right, works out to about 12.0% for r. (You want to divide r by 12, not 180, because it's an annual interest rate compounded monthly.)
- Steve J