I'm going to assume your derivative calculation and simplification are correct. Then yes you are right the next step is to look where this expression equals 0 or is undefined. So,
The cosine function is periodic and will hit 0 an infinite amount of time. Generally , where k is any odd integer.
The other non-trivial time f'=0 is when This means that either a=0 or one of the terms has to be negative. Since both cases are not possible, this solution doesn't hold with the constraints.
The derivative is undefined when (the squared doesn't matter here). So we're looking for when And since b>a always, that means b/a is greater than one and this has no solutions.