Each linear operator (A) on a Hilbert Space has a corresponding adjoint operator (A*). This linear operator is self-adjoint if A =A*.
Vibration analysis will probably be concerned with linear differential operators acting on some space of functions. These functions will probably be defined on some interval [a,b], and they will have an inner product , defined by . A linear differential operator L acting on these functions will have an adjoint L*, defined by for all f,g. Unsurprisingly, L is called selfadjoint if L*=L.
That is the mathematical setting for selfadjoint differential operators. If you want something more specific to vibration analysis, you'll have to give us a more concrete example of the sort of differential operators you're meeting in this theory.