Results 1 to 5 of 5

Math Help - gradient of a function and matrices

  1. #1
    Newbie
    Joined
    Sep 2012
    From
    United States
    Posts
    2

    gradient of a function and matrices

    Hi,

    I need help computing the gradient of a function.

    f(b) = (y-Xb)^{2}+b^{2}

    Where y and b are vectors, and X is a matrix.

    So

    f(b) = (y-Xb)^{T}(y-Xb)+b^{T}b)

    f(b) = (y^{T}y -y^{T}Xb - b^{T}X^{T}y + b^{T}X^{T}Xb +b^{T}b)

    I need help to find \triangledown _{b} (y^{T}y -y^{T}Xb - b^{T}X^{T}y + b^{T}X^{T}Xb+b^{T}b)

    Thanks!
    Last edited by noway; September 27th 2012 at 12:02 AM.
    Follow Math Help Forum on Facebook and Google+

  2. #2
    MHF Contributor
    Joined
    Sep 2012
    From
    Australia
    Posts
    3,651
    Thanks
    602

    Re: gradient of a function and matrices

    Hey noway.

    Usually you have identities for these kinds of things, but if you are trying to prove it then that's different?

    Are you trying to simply calculate it or to prove what the result is?
    Follow Math Help Forum on Facebook and Google+

  3. #3
    Newbie
    Joined
    Sep 2012
    From
    United States
    Posts
    2

    Re: gradient of a function and matrices

    I've calculated it, but I'm doing it incorrectly I guess. It is part of another problem, so I need to get it right before moving on.
    Follow Math Help Forum on Facebook and Google+

  4. #4
    MHF Contributor
    Joined
    Sep 2012
    From
    Australia
    Posts
    3,651
    Thanks
    602

    Re: gradient of a function and matrices

    Is this just a standard directional derivative with respect to some vector?

    If so it's going to have some dot product and hence some matrix expansion in terms of b and the normal derivative of that matrix expression.

    I can't recall the identities off the top of my head, but if you do a search they should be out there.
    Follow Math Help Forum on Facebook and Google+

  5. #5
    Super Member
    Joined
    Sep 2012
    From
    Washington DC USA
    Posts
    525
    Thanks
    146

    Re: gradient of a function and matrices

    There are severals ways to find \triangledown f(b_0), because there are several ways to interpret it. I'll show you a kinda weird way, but one that's often very useful:

    Think of a C^1 curve g: (-1, 1) \rightarrow \mathbb{R}^n such that g(0) = b_0 and g'(0) = v.

    Then \left. \frac{d}{dt}\right|_{t=0} f(g(t)) = < \triangledown f (b_0), g'(0) > = < \triangledown f (b_0), v >.

    Now, although there are infinitely many, g(t) = b_0 + vt is the natural choice for g. However, we won't even need that.

    < \triangledown f (b_0), v > = \left. \frac{d}{dt}\right|_{t=0} f(g(t)) = \left.\frac{d}{dt}\right|_{t=0} \left( \lVert y - X g(t) \rVert^2 + \lVert g(t) \rVert^2 \right)

    = \left.\frac{d}{dt}\right|_{t=0} \left( <y - X g(t), y - X g(t)> + <g(t), g(t)> \right)

    = \left.\frac{d}{dt}\right|_{t=0} \left( <y, y> - 2<X g(t), y> + <X g(t), X g(t)> + <g(t), g(t)> \right)

    = \left.\frac{d}{dt}\right|_{t=0} \left( <y, y> - 2<g(t), X^ty> + <X g(t), X g(t)> + <g(t), g(t)> \right)

    = \left.\frac{d}{dt}\right|_{t=0}( <y, y>) - 2\left.\frac{d}{dt}\right|_{t=0}(< g(t), X^ty>) +.....

    ........ + \left.\frac{d}{dt}\right|_{t=0}<X g(t), X g(t)> + \left.\frac{d}{dt}\right|_{t=0}(<g(t), g(t)>)

    = 0 - 2< g'(0), X^ty> + (<Xg'(0), Xg(0)> + <Xg(0), Xg'(0)>)+ ....

    ........ + (<g'(0), g(0)> + <g(0), g'(0)>)

    = - 2 < g'(0), X^ty> + 2 <Xg'(0), Xg(0)> + 2<g'(0), g(0)>

    = - 2 < v, X^ty> + 2 <Xv, Xb_0> + 2<v, b_0>

    = - 2 < v, X^ty> + 2 <v, X^tXb_0> + 2<v, b_0>

    = < v, \{- 2 X^ty + 2 X^tXb_0 + 2 b_0 \}>

    Therefore < \triangledown f (b_0), v > = < v, \{- 2 X^ty + 2 X^tXb_0 + 2 b_0 \}>.

    So < \[ \ \triangledown f (b_0) - \{- 2 X^ty + 2 X^tXb_0 + 2 b_0 \} \ \], v > = 0, and since that holds for EVERY v, have:

    \triangledown f (b_0)  = - 2 X^ty + 2 X^tXb_0 + 2 b_0 = 2 \{ b_0 - X^t (y - X b_0) \} .
    Last edited by johnsomeone; September 27th 2012 at 02:14 AM.
    Follow Math Help Forum on Facebook and Google+

Similar Math Help Forum Discussions

  1. Gradient for the max function
    Posted in the Calculus Forum
    Replies: 3
    Last Post: October 9th 2009, 06:21 AM
  2. [SOLVED] Gradient of a function
    Posted in the Calculus Forum
    Replies: 9
    Last Post: August 26th 2008, 01:10 PM
  3. Gradient of function
    Posted in the Algebra Forum
    Replies: 1
    Last Post: April 28th 2008, 11:44 PM
  4. Need some help (gradient function?)
    Posted in the Trigonometry Forum
    Replies: 2
    Last Post: August 28th 2007, 10:24 PM
  5. gradient function
    Posted in the Calculus Forum
    Replies: 1
    Last Post: October 8th 2006, 12:14 PM

Search Tags


/mathhelpforum @mathhelpforum