As for the rotation matrix approach: you can make it work, if you don't overthink it. The inverse of an elementary rotation matrix (i.e., about one of the three cartesian axes) is the same rotation matrix with a negative angle substituted in. (Don't think inverse trig functions here.) That makes sense, right? If I rotate about the x axis through

radians, I can reverse that operation by going back

radians, or in other words, if I rotate through

radians, I get back to where I started.

If you apply multiple rotations, you have to "undo" them in the reverse order in which you applied them (like a stack in computer science: last in, first out).