gradient of a function and matrices

**noway** · September 26th 2012, 11:58 PM

Hi,

I need help computing the gradient of a function.

$f(b) = (y-Xb)^{2}+b^{2}$

Where y and b are vectors, and X is a matrix.

So

$f(b) = (y-Xb)^{T}(y-Xb)+b^{T}b)$

$f(b) = (y^{T}y -y^{T}Xb - b^{T}X^{T}y + b^{T}X^{T}Xb +b^{T}b)$

I need help to find $\triangledown _{b} (y^{T}y -y^{T}Xb - b^{T}X^{T}y + b^{T}X^{T}Xb+b^{T}b)$

Thanks!

**chiro** · September 27th 2012, 12:03 AM

Hey noway.

Usually you have identities for these kinds of things, but if you are trying to prove it then that's different?

Are you trying to simply calculate it or to prove what the result is?

**noway** · September 27th 2012, 12:06 AM

I've calculated it, but I'm doing it incorrectly I guess. It is part of another problem, so I need to get it right before moving on.

**chiro** · September 27th 2012, 12:13 AM

Is this just a standard directional derivative with respect to some vector?

If so it's going to have some dot product and hence some matrix expansion in terms of b and the normal derivative of that matrix expression.

I can't recall the identities off the top of my head, but if you do a search they should be out there.

**johnsomeone** · September 27th 2012, 01:34 AM

There are severals ways to find $\triangledown f(b_0)$ , because there are several ways to interpret it. I'll show you a kinda weird way, but one that's often very useful:

Think of a $C^1$ curve $g: (-1, 1) \rightarrow \mathbb{R}^n$ such that $g(0) = b_0$ and $g'(0) = v$ .

Then $\left. \frac{d}{dt}\right|_{t=0} f(g(t)) = < \triangledown f (b_0), g'(0) > = < \triangledown f (b_0), v >$ .

Now, although there are infinitely many, $g(t) = b_0 + vt$ is the natural choice for g. However, we won't even need that.

$< \triangledown f (b_0), v > = \left. \frac{d}{dt}\right|_{t=0} f(g(t)) = \left.\frac{d}{dt}\right|_{t=0} \left( \lVert y - X g(t) \rVert^2 + \lVert g(t) \rVert^2 \right)$

$= \left.\frac{d}{dt}\right|_{t=0} \left( <y - X g(t), y - X g(t)> + <g(t), g(t)> \right)$

$= \left.\frac{d}{dt}\right|_{t=0} \left( <y, y> - 2<X g(t), y> + <X g(t), X g(t)> + <g(t), g(t)> \right)$

$= \left.\frac{d}{dt}\right|_{t=0} \left( <y, y> - 2<g(t), X^ty> + <X g(t), X g(t)> + <g(t), g(t)> \right)$

$= \left.\frac{d}{dt}\right|_{t=0}( <y, y>) - 2\left.\frac{d}{dt}\right|_{t=0}(< g(t), X^ty>) +.....$

$........ + \left.\frac{d}{dt}\right|_{t=0}<X g(t), X g(t)> + \left.\frac{d}{dt}\right|_{t=0}(<g(t), g(t)>)$

$= 0 - 2< g'(0), X^ty> + (<Xg'(0), Xg(0)> + <Xg(0), Xg'(0)>)+ ....$

$........ + (<g'(0), g(0)> + <g(0), g'(0)>)$

$= - 2 < g'(0), X^ty> + 2 <Xg'(0), Xg(0)> + 2<g'(0), g(0)>$

$= - 2 < v, X^ty> + 2 <Xv, Xb_0> + 2<v, b_0>$

$= - 2 < v, X^ty> + 2 <v, X^tXb_0> + 2<v, b_0>$

$= < v, \{- 2 X^ty + 2 X^tXb_0 + 2 b_0 \}>$

Therefore $< \triangledown f (b_0), v > = < v, \{- 2 X^ty + 2 X^tXb_0 + 2 b_0 \}>$ .

So $< \[ \ \triangledown f (b_0) - \{- 2 X^ty + 2 X^tXb_0 + 2 b_0 \} \ \], v > = 0$ , and since that holds for EVERY $v$ , have:

$\triangledown f (b_0) = - 2 X^ty + 2 X^tXb_0 + 2 b_0 = 2 \{ b_0 - X^t (y - X b_0) \}$ .

Thread: gradient of a function and matrices

LinkBack

Thread Tools

Display

gradient of a function and matrices

Re: gradient of a function and matrices

Re: gradient of a function and matrices

Re: gradient of a function and matrices

Re: gradient of a function and matrices

Similar Math Help Forum Discussions

Gradient for the max function

[SOLVED] Gradient of a function

Gradient of function

Need some help (gradient function?)

gradient function

Search Tags