Your equation just one simple error. I see you have made it plus [xn + f(x)/-f'(x)], in substitution for the denominator having reversed signs. But you also need to reverse the sign for the '1' at the end as well.
Find the smallest positive value which satisfies x = 2.600cos(2.400x)
I used the formula
xn+1 = xn + [2.600cos(2.400xn)-xn]/[6.240sin(2.400xn)-1]
I ended up with x = 0.4900 after a few runs through but my online homework tells me this is incorrect. My calculator seems to get close to this when I use the zoom function to check though.