First, you need to express and in terms of . ( is for positive integer n always an nth degree polynomial of ; they're called the Chebyshev_polynomials)
For you should have the double angle identity .
For , use the cosine sum formula with 3x=2x+x to get:
(from the double angle identities)
(simplifying, and subtracting one from both sides)
(dividing by two)
This is an easily factorable polynomial, and should give you three equations of the form for some a, which you can then solve.