Here is an example of what I mean. Your original vector is (1,1,0). Lets call that y, and your axis of rotation is orthogonal to this. Lets call that z. To make things easy let's assume it is (0,0,1). Now y cross z which I make to be (1,-1,0), but I suppose could be (-1,1,0) depending on your sign convention for the cross product is a third vector orthogonal to both the other two which we can call x. (In my programs I use an LHS coordinate system, and picked the sign of the cross product so that (1,0,0) cross (0,1,0) is (0,0,1). So I think what I'm doing would work just as well in an ordinary RHS system)

So if we normalise the vectors to length 1 by dividing the first and third by sqrt(2) we have a new orthonormal coordinate system,(A rotation by theta about z looks like

, so it is very easy to express any rotation. Let us call this matrix M

Now the matrix that tells us how to get from our new coordinates to our original coordinates is simply [x,y,z] i.e.

. Call that matrix N

So the rotation matrix for a rotation of theta in our original coordinate system is N * M * N^-1.

Obviously you can do this for any rotation about any axis, though this example will be particularly easy to compute. In fact here I'd be tempted not to bother dividing by

at all and to use the adjugate matrix instead of the inverse, and then finally just halve all the entries in the final matrix, but you have to be careful with tricks like that as the two basis vectors in the plane being rotated must be the same length.

Once you've computed this matrix you can then apply my method from the last post to compute the x,y, and z rotations required. It might be worth checking what I have done is doing things in the order you want by computing the product

That's obviously the most general matrix you can get by multiplying three rotations about the axes. Check that my code usually recovers xi,eta and zeta. (There will certainly be times when it won't).

Conjugation isn't the only way to compute 3D rotation matrices. Some people use quaternions, but I like conjugation as it is easy to work out how to do it from scratch when you have forgotten what order the matrices should go in, and it works for other kinds of transformation as well.