Tx = T(λ1x1 + λ2x2) = λ1T(x1) + λ2T(x2)
= λ1x1 - λ2x2 = λ1x1 + λ2x2 - 2λ2x2
= x - 2λ2x2 (this is just simple algebra, to this point).
now (x,x2) = (λ1x1 + λ2x2,x2) = λ1(x1,x2) + λ2(x2,x2)
and since x1,x2 are orthogonal, (x1,x2) = 0, therefore:
(x,x2) = λ2(x2,x2) = λ2|x2|2.
but x2 lies on the unit circle, therefore it has length (and thus length squared) of 1.
so (x,x2) = λ2, and we have:
Tx = x - 2λ2x2 = x - 2(x,x2)x2.
the quantity (x,x2)x2 is a little misleading (because we are dealing with UNIT vectors x1,x2).
normally, this is written as: [(x,x2)/(x2,x2)]x2 which is known as:
the projection of the vector x in the direction of the vector x2. geometrically this is: "the part of x that lies on the line generated by x2".
normally, we project a vector onto the unit vectors e1 = (1,0) and e2 = (0,1).
that is: if x = (x1,x2) = x1e1 + x2e2, then:
the projection of x in the direction of e1 is [(x.e1)/(e1.e1)]e1
= [(x1*1 + x2*0)/(1*1 + 0*0)]e1 = x1e1 = (x1,0).