Warning: this might be a somewhat boneheaded approach. Given the equation
isn't it true that and are all known? If that's the case, can't you come up with a general description of the matrix in terms of, say, Euler angles, and combine that matrix with
then solve the resulting system, and finally solve the smaller problem? It's kind of like this: Solve ABx = b by first setting y = Bx, then solve Ay = b, and then solve y = Bx. The only difference is that in your second problem, you're solving for the matrix instead of the vector.
Any reason why this wouldn't work?