Have you factored in your physics the inertia matrix for your object?
Once you factor this in with the angular momentum you should be able to handle rotating objects.
I'm doing 2D/3D ball collision simulation. All the physics are done but I can't figure out the rotation. Now I got it rotating around x axis, but its not pretty.
I have all that I need, I hope:Both ball position on impact, their outgoing vector (represented by x,y,z), ball diameter, their speed (which is slowed down periodically), balls current rotation.
I tried to accomplish that with rotation matrix, but I failed.
Hope you can help me. Thank you!
Basically the idea is that you have a rotational axis that corresponds to the geometry and mass distribution of the object and this is used in calculations for angular momentum and thus, for rotational velocity/force.
This is not a trivial topic: having a physics engine that is stable is a difficult thing. Essentially you need to have collision detection and force integration schemes that are stable.
Once you have the moment of inertia, you basically apply a force with respect to the axis (think of a rotation) and this involves some matrix and vector mathematics.
You will need routines to figure out the moment of inertia for a geometric object (which is a matrix) and then you will use this matrix to deal with the angular momentum and force.
There is a book that I have personally owned by O'Reilly publishing that deals with these issues:
Physics for Game Developers-O'Reilly Media
If you can get an open source physics engine, this would also help you in your quest.
Again, the topic is not trivial so forgive me if you wanted to get a one post solution.