Because it is particularly easy to write any vector in terms of an orthogonal basis (even easier if it's orthonormal). Here's an application: thinking of the vector space of functions, you can solve differential equations formed by means of, say, an Hermitian operator, by writing down the solution as a sum of eigenvectors of the operator. Physicists do this kind of thing all the time to solve ODE's or even PDE's.
Another application is diagonalization. Suppose you have the first-order system of DE's: . A solution is . But how do you compute Well, if is diagonalizable (you can find an orthogonal matrix such that for some diagonal matrix ), then which is much easier to compute. And the process of diagonalization, or finding the orthogonal matrix , is a process of finding an orthonormal (which is automatically orthogonal) basis.