I guess you mean compact for a topology which is given by a norm. Since M is a finite dimensional vector space, you have to show that O is closed and bounded.
The set of matrices can by identified in the obvious way with , and can be given the standard metric topology. Heine-Borel applies in this setting, too. Bounded is easy here, and for closed, here's a hint: is the map continuous?