In vibration analysis I found the terms: "self-adjoint and non-self-adjoint systems".
Would someone help me understand what does this mean, knowing that I am not aquanted with topology, functional analysis or operator theory.
Vibration analysis will probably be concerned with linear differential operators acting on some space of functions. These functions will probably be defined on some interval [a,b], and they will have an inner product $\displaystyle \langle f,g\rangle$, defined by $\displaystyle \langle f,g\rangle = \int_a^bf(t)g(t)\,dt$. A linear differential operator L acting on these functions will have an adjoint L*, defined by $\displaystyle \langle L^*f,g\rangle = \langle f,Lg\rangle$ for all f,g. Unsurprisingly, L is called selfadjoint if L*=L.
That is the mathematical setting for selfadjoint differential operators. If you want something more specific to vibration analysis, you'll have to give us a more concrete example of the sort of differential operators you're meeting in this theory.