You're almost there, though your notation perhaps makes things a little messier:

is unitary, so:

(i)

(ii)

(iii) The inverse of an upper triangular matrix is also upper triangular with diagonal elements

Now put together (i) and (iii): , and taking into account (ii) this gives us that all the elements outside of the main diagonal must be zero. Q.E.D.

Or simpler, using your reasoning and (iii): is upper triangular, but , and is LOWER triangular is both upper and lower triangular and then it is diagonal, and thus also is.

Tonio