What specific type of answer is your teacher looking for? I'm assuming that you are looking for numerical answers. Until I know for sure, I can give you a little bit of guidance, that, hopefully, is not too obvious.
I would break down the problem into steps. First, we have your given problem:
cos[tan^-1 (5/12) - tan^-1 (13/4)]
tan^-1 (5/12) = arctan (5/12) which implies:
tan a = 5/12 Recalling tan a = x/y.
tan^-1 (13/4) = arctan (13/4) which implies:
tan b = 13/4 Recalling tab b=x/y.
The subtraction inside the brackets is the subtraction of two angles. I'm assuming that you are looking for numbers, since, off the top of my head, these trig ratios in your problem do not represent any fundamental angles or identities. I'm also assuming you're working in degrees, but if you're working in radians, you'll still get the same answer, because you are looking for a final answer as a ratio, not as an angle measure.
arctan (5/12) ~ 22.61986495 deg ~ 0.394791119 radians
arctan (13/4) ~ 72.89727103 deg ~ 1.272297395 radians
Subtract these two, and you get an angle, which we will call z:
z ~ -50.27740608 degrees ~ -0.877506275 radians
We now need cos z ~ cos (-50.27740608 degrees) ~ 0.639071171. If you obtain cos z where z ~ 1.272297395 rads, you will get the same cos value.
Hope this helps.