1. ## Finding The Gradient Of A Matrix Equation

f(x) = x' * E * x

x' = x transpose
E = a covariance matrix

The problem is: prove that f(x) is convex. The solution has the equations:

gradient of f = 2 * E * x <- How is the gradient calculated here from f(x)?
gradient^2 of f = 2 * E

E > 0 since it is a covariance matrix, therefore f is convex

I want to learn how to calculate the gradient from the function f, the answer is given but I don't know how it was determined.

2. Originally Posted by jjpioli

gradient of f = 2 * E * x <- How is the gradient calculated here from f(x)?
As $\displaystyle \displastyle X'EX$ is quadratic. Multiply it out to convince yourself.

3. By quadratic, do you mean if we have a 2x2 matrix:

top left represents x^2
top right represents x*y
bottom left represents y*x
bottom right represents y^2

So then we can build a quadratic equation from that with x and y variables, then we simply find the gradient of that function?

I still don't understand how you can simply go from x' * E * x to grad(f) = 2*E*x

4. There is a vector tranpose with your variables, say x1 and x2. Then E the 2x2 matrix is eij constants for covariance where $\displaystyle e_{ij} = e_{ji}$, where $\displaystyle i \neq j$. Finally x1 and x2 again. Expand them out!

5. I expanded it out, here is my work:

http://i.imgur.com/iLcaS.jpg

I have one resulting quadratic equation, do I find the gradient of that?

6. Very close, discard the C's altogether, now

E = [ e11 e12
e21 e21 ]

Try again, remember e12 = e21

7. You should get

$\displaystyle \displaystyle e_{11}x_1^2+2e_{12}x_1x_2+e_{22}x_2^2$

Differentiate this with respect to $\displaystyle x_1$ and $\displaystyle x_2$ and you should be able to see the system $\displaystyle 2EX$

8. http://i.imgur.com/69Lgj.jpg

After expanding x'Ex, to get the gradient, I would derive with respect to x1, and then derive with respect to x2, resulting in 2*(e12), but my solution says that the gradient is 2*E*x, which is not equal to 2*(e12).

Is my understanding incorrect?

9. I'm not following your work very well and where are your actual derivatives?

I get

$\displaystyle \displaystyle f_{x_1} = 2e_{11}x_1+ 2e_{12}x_2$

and

$\displaystyle \displaystyle f_{x_2} = 2e_{21}x_1+ 2e_{22}x_2$

2 is clearly a factor for the whole system, now just write back to matrix form.

10. I see:

http://i.imgur.com/7nQ6z.jpg

Now I have a few additional questions. The gradient is a vector, that represents the direction of greatest slope increase, so I have two elements in my vector, do the elements represent the (x1 or x2) coordinates or the (i or j) coordinates?

Also, from the solution to this problem, it appears that the problem takes the following steps:

f= x'Ex

Is there a rule with matrices where I can simply deduce the gradient without expanding everything out? I feel, now that I understand the underlying concept (through expansion) I should be able to use shortcuts to quicken my problem solving skills. THanks

11. Not only the greatest slope increase but the gradient at any $\displaystyle x_1$ and $\displaystyle x_2$

As for the generalisation, you have proven that this process maps back to

$\displaystyle \displaystyle y = ax^2$

$\displaystyle \displaystyle y' = 2ax$

$\displaystyle \displaystyle y'' = 2a$

Well done!

12. y = ax^2
y' = 2ax
y'' = 2a

If x is a 2x2 matrix, and E is our same covariance matrix, x'Ex does not appear to be equal to Ex^2. I'm still confused

13. Originally Posted by jjpioli

If x is a 2x2 matrix, and E is our same covariance matrix, x'Ex does not appear to be equal to Ex^2. I'm still confused
Because X is not a 2x2, E is a 2x2 matrix.

Here is your expression with their corresponding orders X' E X (1x2) (2x2) (2x1) which expands out to a 1x1 as shown in post #7. The matrix expression is a quadratic form. That's all you need to know.

,

,

,