Thread: Principal components analysis

For the covariance matrix S it is known that all its p p elements are greater than zero 0. Prove that:
a) Coefficients of the first principal component are all of the same sign,
b) Coefficients of each other principal component cannot be all of the same sign.

Any help greatly appreciated.

Thanks

Hey kungalo.

You have your covariance matrix which is all positive, and for the first principal component you are going to solve an eigen-decomposition problem on your covariance matrix (i.e. find the eigen-vectors and eigen-values and retain the one with the highest eigen-value since this corresponds to the variance of each component). You don't have orthogonality conditions on the first component which means you are only interested in getting the one with the highest variance.

So what this boils down to is proving that all elements in the eigen-vector corresponding to the highest eigen-value are all positive (since the covariance matrix is positive definite, it will always have positive eigenvalues).

As for the next one, this lies on the argument dealing with orthogonality since all later components will always be orthogonal to every other principal compnent.

The easiest thing to do is that if the first component is <a,b,c,d,....> then if something is normal you know that X = PC1 Y = PCN then <X,Y> = 0 and the only way for PCN to have this relation is for at least one of the components of PCN to be negative (we also assume PCN is not the zero vector and has positive magnitude).

I actually worked this one out. It relies on expressing the variance of the principal component as a sum of the elements of the covariance matrix and the elements of the coefficient vector. From there it's pretty easy to see that all the coefficients need to have the same sign in order to maximize the variance of the first principal component, and that there need to be positive and negative coefficients in the second principal component in order for it to be uncorrelated with the first.