It would help if you told us what you tried and where you got stuck.
(Substituting y for tan(x) in (2) was a good idea. Now multiply both sides of the equation by y.)
1.) I want to use a substitution but I can't because of 2 sinx. I need cosx there. cosx² I can write like 1-sinx² but there is only cosx. I must chang and I must change cosx to sinx but I don't know how.
2.) y² - 1 - 2/√3y = 0
y² - 2/√3y - 1 = 0
D= b²- 4ac
D= (-2/√3)²-4 * 1 * (-1)
√D = 4 √3/3
x1,2 = -b ± √D/2a
x1,2 = 2/ √3 ± 4 √3/3:2
x1 = √3....................tg is π/3
x2= - √3/3..............tg is 5/6π and 11/6π. Is that solition right?