You understand, do you not, that a matrix A is invertible if and only if det(A) is non-zero? And that det(transpose(A))= det(A) and det(AB)= det(A)det(B) so that det(transpose(A)*A)= det(A*transpose(A))= (det(A))^2= 0 if and only if det(A)= 0? Rather than looking at det(transpose(A)*A) or det(A*transpose(A)), just look at det(A).