How to get the Hessian Matrix please?

**ggyyree** · September 28th 2008, 01:56 PM

I have got the following function constructed by matrices and vectors:

$ f(x, \alpha) = \frac{1}{2}[(Ax-y_1)^2+(AP(\alpha)x-y_2)^2] $

in which, $x, y_1, y_2$ are vectors. $A, P(\alpha)$ are matrices.
The gradients with respect to $x, \alpha$ are:

$ \frac{\partial{f(x, \alpha)}}{\partial{x}} = A^T A x - A^T y_1 + P(\alpha)^T A^T A P(\alpha) x - P(\alpha)^T A^T y_2 $

$ \frac{\partial{f(x, \alpha)}}{\partial{\alpha}} = x^T P(\alpha)^T A^T A P(\alpha) x - x^T P'(\alpha)^T A^T y_2 $

in which, $A^T$ is the transpose of $A$ and $P'(\alpha)$ is the first order derivative of $P$ .
Anyone knows how to get the Hessian of the $f(x, \alpha)$ ? Thanks a lot!

**choovuck** · September 28th 2008, 09:02 PM

just take second derivatives the way you took first ones, and write the Hessian as the 2x2 matrix each entry of which is also a matrix (I guess it can be viewed as a tensor if needed)

**ggyyree** · September 29th 2008, 12:26 AM

Originally Posted by choovuck

just take second derivatives the way you took first ones, and write the Hessian as the 2x2 matrix each entry of which is also a matrix (I guess it can be viewed as a tensor if needed)

Thanks a lot! But could you please tell me more details? I guess it should be:

$ HessianMatrix= \begin{bmatrix} \frac{\partial{f(x, \alpha)}}{\partial{x}\partial{x}} &\frac{\partial{f(x, \alpha)}}{\partial{\alpha}\partial{\alpha}} \\ \frac{\partial{f(x, \alpha)}}{\partial{\alpha}\partial{\alpha}} &0 \end{bmatrix} $

because the Hessian should be symmetric, am I right?

Moreover, due to the equation:

$ \frac{\partial{f(x, \alpha)}}{\partial{\alpha}} = x^T P(\alpha)^T A^T A P(\alpha) x - x^T P'(\alpha)^T A^T y_2 $

then the second order partial derivative:

$ \frac{\partial{f(x, \alpha)}}{\partial{\alpha}\partial{\alpha}} $

will involve $P''(\alpha)$ . I do not know if I am correct. Please show me the analytic form of the Hessian. Thank you very much!

**choovuck** · September 29th 2008, 02:59 PM

Originally Posted by ggyyree

I guess it should be:

$ HessianMatrix= \begin{bmatrix} \frac{\partial{f(x, \alpha)}}{\partial{x}\partial{x}} &\frac{\partial{f(x, \alpha)}}{\partial{\alpha}\partial{\alpha}} \\ \frac{\partial{f(x, \alpha)}}{\partial{\alpha}\partial{\alpha}} &0 \end{bmatrix} $

because the Hessian should be symmetric, am I right?

no, the Hessian matrix has as its (i,j)-th element the derivative $\frac{\partial^2 f}{\partial x_i \partial x_j}$ , so in our case
$ HessianMatrix= \begin{bmatrix} \frac{\partial^2{f(x, \alpha)}}{\partial{x}^2} &\frac{\partial^2{f(x, \alpha)}}{\partial{x}\partial{\alpha}} \\ \frac{\partial^2{f(x, \alpha)}}{\partial{\alpha}\partial{x}} &\frac{\partial^2{f(x, \alpha)}}{\partial{\alpha}\partial{\alpha}} \end{bmatrix} $

It is symmetric since $\frac{\partial^2{f(x, \alpha)}}{\partial{\alpha}\partial{x}}=\frac{\part ial^2{f(x, \alpha)}}{\partial{x}\partial{\alpha}}$

I'm not very comfortable with those matrix gradients (I can't even convince myself why your $\frac{\partial f}{\partial \alpha}$ is correct...), so I won't help you with the actual computations, sorry

**choovuck** · September 29th 2008, 03:03 PM

P.S. When you wrote $f(x, \alpha) = \frac{1}{2}[(Ax-y_1)^2+(AP(\alpha)x-y_2)^2]$ , I'm assuming you meant $f(x, \alpha) = \frac{1}{2}[(Ax-y_1)^T (Ax-y_1)+(AP(\alpha)x-y_2)^T (AP(\alpha)x-y_2)]$

**ggyyree** · September 30th 2008, 12:40 AM

Originally Posted by choovuck

no, the Hessian matrix has as its (i,j)-th element the derivative $\frac{\partial^2 f}{\partial x_i \partial x_j}$ , so in our case
$ HessianMatrix= \begin{bmatrix} \frac{\partial^2{f(x, \alpha)}}{\partial{x}^2} &\frac{\partial^2{f(x, \alpha)}}{\partial{x}\partial{\alpha}} \\ \frac{\partial^2{f(x, \alpha)}}{\partial{\alpha}\partial{x}} &\frac{\partial^2{f(x, \alpha)}}{\partial{\alpha}\partial{\alpha}} \end{bmatrix} $

It is symmetric since $\frac{\partial^2{f(x, \alpha)}}{\partial{\alpha}\partial{x}}=\frac{\part ial^2{f(x, \alpha)}}{\partial{x}\partial{\alpha}}$

I'm not very comfortable with those matrix gradients (I can't even convince myself why your $\frac{\partial f}{\partial \alpha}$ is correct...), so I won't help you with the actual computations, sorry

Yes, you are quite right! Thanks a lot!

**ggyyree** · September 30th 2008, 12:41 AM

Originally Posted by choovuck

P.S. When you wrote $f(x, \alpha) = \frac{1}{2}[(Ax-y_1)^2+(AP(\alpha)x-y_2)^2]$ , I'm assuming you meant $f(x, \alpha) = \frac{1}{2}[(Ax-y_1)^T (Ax-y_1)+(AP(\alpha)x-y_2)^T (AP(\alpha)x-y_2)]$

Yes, you are right!

**ggyyree** · September 30th 2008, 12:41 AM

According to your suggestion I have derived the formula. Thank you very much!

**ggyyree** · September 30th 2008, 12:46 AM

By the way, may I ask you another related question please?

I want to know the relations between 'Gradient', 'Jacobian' and 'Hessian'. What I think they are:

'Function': $F$
'Gradient': $G$
'Jacobian' : $J$
'Hessian': $H$

$G= -J^T \times F$
$H= J^T \times J$

Am I right? Are there any definition for these relations please? Thanks very much!

**choovuck** · September 30th 2008, 08:42 PM

no, those relations are just wrong

Gradient - Wikipedia, the free encyclopedia
Hessian matrix - Wikipedia, the free encyclopedia
Jacobian - Wikipedia, the free encyclopedia

Thread: How to get the Hessian Matrix please?

LinkBack

Thread Tools

Display

How to get the Hessian Matrix please?

Similar Math Help Forum Discussions

Hessian Matrix help

help with hessian matrix

doubt hessian matrix

Hessian matrix

Hessian matrix for 2d function

Search Tags