The geometric meaning can be associated in terms of an inner product with the Cauchy-Schwartz inequality:
Cauchy?Schwarz inequality - Wikipedia, the free encyclopedia
If you look at a lot of the results regarding variance operators and positive definite-ness, you will see a relationship between metrics/norms and the variance operator.
This correlation aspect has a relationship with an inner product (normalized) just like Cauchy-Schwrdinartz except we are talking about variation and relationship between random variables regarding the relationship of variation between variables and not about a geometric relationship between the orientation of two vectors (which is what a normalized inner product looks at).
So yes in short, think about the correlation in terms of an angle.