Say you have a non-linear ODE or a system of ODEs. How do you know that a solutions in elementary functions doesn't exist? In other words how can you tell if the equation cannot be solved explicitly?
* First order separable
* First order homogeneous
* Second order linear
... and so on. The subject is too vast to go into in this medium.
See, for example:
Picard's Existence Theorem - ProofWiki
which gives the conditions under which a 1st-order ODE even has a solution, but that doesn't help you actually find such a solution, or whether it exists in elementary functions.
Also check out the Method of Frobenius (GIYF) for a particular approach to 2nd order equations for when you aren't expecting the solution to have anything to do with elementary functions.
And if your ODE is really difficult to solve, you can always use numerical methods.
But all the above can be found in the introduction of most decent texts on the subject.