I would multiply out the LHS thus:
(a+bi)(a+bi)= a^2 + 2abi - b^2, and equate real and imaginary components to the RHS: a-bi. Two equations and two unknowns.
Re your first post: in general . Also are "z" and "Z" the same? It is bad practice to use capital and non-capital letters interchangeably.
I have no idea where you got "Z= a^2+ 2abi". You seem to have just dropped the "-b^2" from the previous line. And then just dropped "a^2" to get that last line. You can't do that!
If z= a+ bi, then z^2= a^2- b^2+ 2abi. conj(z)= a- bi so your equation, z^2= conj(z), is a^2- b^2+ 2abi= a- bi. Equating real and imaginary parts gives a^2- b^2= a and 2ab= -b.
I get four complex number solutions.