the impedance of a circuit can be calculated by the expression

Z = Z1 X Z2 / Z1 + Z2

If Z1 = 4 + j10 & Z2 = 12 - j3

how do I calculate the impedance Z in rectangular and polar forms?

First, we need to know what the formula really is. Is it
$Z= (Z1)\left(\frac{Z2}{Z1}\right)+ Z2$
which is what you wrote, or is it
$Z= \frac{Z1Z2}{Z_1+ Z_2}$?

With Z1= 4+ j10, Z2= 12- j3, Z1+ Z2= 16+ j7. Now, write the three numbers, 4+j10, 12- j3, and 16+ j7 in polar form: for 4+ j10, $r= \sqrt{4^2+ 10^2}= \sqrt{116}$ and [itex]\theta= arctan(10/4)= arctan(5/2)= 1.19.

Multiplying $r_1 cis(\theta_1)= r_1e^{i\theta_1}$ and $r_2 cis(\theta_2)= r_2e^{i\theta_2}$ gives $(r_1r_2) cis(\theta_1+ \theta_2)= (r_1r_2)e^{i(\theta_1+ \theta_2)}$

sorry its

Z = (Z1 x Z2) / (Z1 + Z2)