Does X^H mean Hermitian conjugation? By "5x5 symmetric and complex" do you mean self-adjoint, i.e. X=X^H, A=A^H, B=B^H? Are you looking for something working for specific A, B, or for general?

I thought about decomposing A and B into spectral decomposition (or "diagonalization", Spectral theorem - Wikipedia, the free encyclopedia) : A=U D1 U^H and B=V D2 V^H, where D1 and D2 are diagonal, and U and V are unitary. Now the first question is whether D1 equals D2. That depends on your A and B of course. If D1=D2, then one of the possible solutions for X is of course X= U V^H. The only problem is that X is unitary rather than self-adjoint. And I'm kind of skeptical about whether a self-adjoint solution can be (easily) found. If D1 is not D2, then look for X in the form X=U D3 V^H, where D3 is a certain diagonal matrix. In this case such an X is neither unitary nor self-adjoint (but e.g. normal, which isn't too bad either)

of course I'm guessing this equation doesn't have auniquesolution, so I'm not claiming that there can't be a self-adjoint X