Thread: Finding The Gradient Of A Matrix Equation

f(x) = x' * E * x

x' = x transpose
E = a covariance matrix

The problem is: prove that f(x) is convex. The solution has the equations:

gradient of f = 2 * E * x <- How is the gradient calculated here from f(x)?
gradient^2 of f = 2 * E

E > 0 since it is a covariance matrix, therefore f is convex

I want to learn how to calculate the gradient from the function f, the answer is given but I don't know how it was determined.

Originally Posted by jjpioli

gradient of f = 2 * E * x <- How is the gradient calculated here from f(x)?

As is quadratic. Multiply it out to convince yourself.

By quadratic, do you mean if we have a 2x2 matrix:

top left represents x^2
top right represents x*y
bottom left represents y*x
bottom right represents y^2

So then we can build a quadratic equation from that with x and y variables, then we simply find the gradient of that function?

I still don't understand how you can simply go from x' * E * x to grad(f) = 2*E*x

There is a vector tranpose with your variables, say x1 and x2. Then E the 2x2 matrix is eij constants for covariance where , where . Finally x1 and x2 again. Expand them out!

I expanded it out, here is my work:

http://i.imgur.com/iLcaS.jpg

I have one resulting quadratic equation, do I find the gradient of that?

Very close, discard the C's altogether, now

E = [ e11 e12
e21 e21 ]

Try again, remember e12 = e21

You should get



Differentiate this with respect to and and you should be able to see the system

http://i.imgur.com/69Lgj.jpg

After expanding x'Ex, to get the gradient, I would derive with respect to x1, and then derive with respect to x2, resulting in 2*(e12), but my solution says that the gradient is 2*E*x, which is not equal to 2*(e12).

Is my understanding incorrect?

I'm not following your work very well and where are your actual derivatives?

I get



and



2 is clearly a factor for the whole system, now just write back to matrix form.

I see:

http://i.imgur.com/7nQ6z.jpg

Now I have a few additional questions. The gradient is a vector, that represents the direction of greatest slope increase, so I have two elements in my vector, do the elements represent the (x1 or x2) coordinates or the (i or j) coordinates?

Also, from the solution to this problem, it appears that the problem takes the following steps:

f= x'Ex
grad f = 2Ex
grad^2 f = 2E

Is there a rule with matrices where I can simply deduce the gradient without expanding everything out? I feel, now that I understand the underlying concept (through expansion) I should be able to use shortcuts to quicken my problem solving skills. THanks

Not only the greatest slope increase but the gradient at any and

As for the generalisation, you have proven that this process maps back to







Well done!

y = ax^2
y' = 2ax
y'' = 2a

If x is a 2x2 matrix, and E is our same covariance matrix, x'Ex does not appear to be equal to Ex^2. I'm still confused

Originally Posted by jjpioli

If x is a 2x2 matrix, and E is our same covariance matrix, x'Ex does not appear to be equal to Ex^2. I'm still confused

Because X is not a 2x2, E is a 2x2 matrix.

Here is your expression with their corresponding orders X' E X (1x2) (2x2) (2x1) which expands out to a 1x1 as shown in post #7. The matrix expression is a quadratic form. That's all you need to know.