"Caculate the determinant of A^2 for any matrix A of order 3X3" does not seem to have anything to do with "A= B^T*B".

Did you mean "Calculate the determinant of A^2 for any matrix A,of the form A= B^T*B for some matrix B, of order 3X3"?

It will help to know that , so that and perhaps that .