May I assume your whole integrand is cos(at)*cos(bt) *dt?

Because that last part, the "dt", is very important.

INT.[cos(at)cos(bt)]dt ---------------(1)

As it is, the integrand is the product of two functions. To integrate that, we can do it by parts. INT[u]dv = uv -INT.[v]du. But that won't do here because there is no sin(at) or sin(bt) involved.

So we have to separate/decompose the product into addition or subtraction.

There is this trig identity

cosAcosB = (1/2)[cos(A-B) +cos(A+B)] ---------(i)

that we can use to decompose the original integrand.

cos(at)cos(bt)

= (1/2)[cos(at -bt) +cos(at +bt)]

= (1/2)[cos(t(a-b)) +cos(t(a+b))]

So, (1) becomes

= (1/2)INT.[cos((a-b)t) +cos((a+b)t)]dt -----------(2)

It is easy now.

= (1/2)INT.[cos((a-b)t)]dt +1/2INT.[cos((a+b)t)]dt

= (1/2)INT.[cos((a-b)t)]*[(a-b)/(a-b)]dt +(1/2)INT.[cos((a+bt))]*[(a+b)/(a+b)]dt

= [(1/2)(1/(a-b))]INT.[cos((a-b)t)]*(a-b)dt +[(1/2)(1/(a+b))]INT.[cos((a+b)t)]*(a+b)dt

= [1 / (2(a-b))]sin((a-b)t) +[1 / (2(a+b))]sin((a+b)t) +C

That is it.