# How to get the Hessian Matrix please?

• Sep 28th 2008, 01:56 PM
ggyyree
How to get the Hessian Matrix please?
I have got the following function constructed by matrices and vectors:

$\displaystyle f(x, \alpha) = \frac{1}{2}[(Ax-y_1)^2+(AP(\alpha)x-y_2)^2]$

in which, $\displaystyle x, y_1, y_2$ are vectors. $\displaystyle A, P(\alpha)$ are matrices.
The gradients with respect to $\displaystyle x, \alpha$ are:

$\displaystyle \frac{\partial{f(x, \alpha)}}{\partial{x}} = A^T A x - A^T y_1 + P(\alpha)^T A^T A P(\alpha) x - P(\alpha)^T A^T y_2$

$\displaystyle \frac{\partial{f(x, \alpha)}}{\partial{\alpha}} = x^T P(\alpha)^T A^T A P(\alpha) x - x^T P'(\alpha)^T A^T y_2$

in which, $\displaystyle A^T$ is the transpose of $\displaystyle A$ and $\displaystyle P'(\alpha)$ is the first order derivative of $\displaystyle P$.
Anyone knows how to get the Hessian of the $\displaystyle f(x, \alpha)$? Thanks a lot! (Happy)
• Sep 28th 2008, 09:02 PM
choovuck
just take second derivatives the way you took first ones, and write the Hessian as the 2x2 matrix each entry of which is also a matrix (I guess it can be viewed as a tensor if needed)
• Sep 29th 2008, 12:26 AM
ggyyree
Quote:

Originally Posted by choovuck
just take second derivatives the way you took first ones, and write the Hessian as the 2x2 matrix each entry of which is also a matrix (I guess it can be viewed as a tensor if needed)

Thanks a lot! But could you please tell me more details? I guess it should be:

$\displaystyle HessianMatrix= \begin{bmatrix} \frac{\partial{f(x, \alpha)}}{\partial{x}\partial{x}} &\frac{\partial{f(x, \alpha)}}{\partial{\alpha}\partial{\alpha}} \\ \frac{\partial{f(x, \alpha)}}{\partial{\alpha}\partial{\alpha}} &0 \end{bmatrix}$

because the Hessian should be symmetric, am I right?

Moreover, due to the equation:

$\displaystyle \frac{\partial{f(x, \alpha)}}{\partial{\alpha}} = x^T P(\alpha)^T A^T A P(\alpha) x - x^T P'(\alpha)^T A^T y_2$

then the second order partial derivative:

$\displaystyle \frac{\partial{f(x, \alpha)}}{\partial{\alpha}\partial{\alpha}}$

will involve $\displaystyle P''(\alpha)$. I do not know if I am correct. Please show me the analytic form of the Hessian. Thank you very much!
• Sep 29th 2008, 02:59 PM
choovuck
Quote:

Originally Posted by ggyyree
I guess it should be:

$\displaystyle HessianMatrix= \begin{bmatrix} \frac{\partial{f(x, \alpha)}}{\partial{x}\partial{x}} &\frac{\partial{f(x, \alpha)}}{\partial{\alpha}\partial{\alpha}} \\ \frac{\partial{f(x, \alpha)}}{\partial{\alpha}\partial{\alpha}} &0 \end{bmatrix}$

because the Hessian should be symmetric, am I right?

no, the Hessian matrix has as its (i,j)-th element the derivative $\displaystyle \frac{\partial^2 f}{\partial x_i \partial x_j}$, so in our case
$\displaystyle HessianMatrix= \begin{bmatrix} \frac{\partial^2{f(x, \alpha)}}{\partial{x}^2} &\frac{\partial^2{f(x, \alpha)}}{\partial{x}\partial{\alpha}} \\ \frac{\partial^2{f(x, \alpha)}}{\partial{\alpha}\partial{x}} &\frac{\partial^2{f(x, \alpha)}}{\partial{\alpha}\partial{\alpha}} \end{bmatrix}$

It is symmetric since $\displaystyle \frac{\partial^2{f(x, \alpha)}}{\partial{\alpha}\partial{x}}=\frac{\part ial^2{f(x, \alpha)}}{\partial{x}\partial{\alpha}}$

I'm not very comfortable with those matrix gradients (I can't even convince myself why your $\displaystyle \frac{\partial f}{\partial \alpha}$ is correct...), so I won't help you with the actual computations, sorry
• Sep 29th 2008, 03:03 PM
choovuck
P.S. When you wrote $\displaystyle f(x, \alpha) = \frac{1}{2}[(Ax-y_1)^2+(AP(\alpha)x-y_2)^2]$, I'm assuming you meant $\displaystyle f(x, \alpha) = \frac{1}{2}[(Ax-y_1)^T (Ax-y_1)+(AP(\alpha)x-y_2)^T (AP(\alpha)x-y_2)]$
• Sep 30th 2008, 12:40 AM
ggyyree
Quote:

Originally Posted by choovuck
no, the Hessian matrix has as its (i,j)-th element the derivative $\displaystyle \frac{\partial^2 f}{\partial x_i \partial x_j}$, so in our case
$\displaystyle HessianMatrix= \begin{bmatrix} \frac{\partial^2{f(x, \alpha)}}{\partial{x}^2} &\frac{\partial^2{f(x, \alpha)}}{\partial{x}\partial{\alpha}} \\ \frac{\partial^2{f(x, \alpha)}}{\partial{\alpha}\partial{x}} &\frac{\partial^2{f(x, \alpha)}}{\partial{\alpha}\partial{\alpha}} \end{bmatrix}$

It is symmetric since $\displaystyle \frac{\partial^2{f(x, \alpha)}}{\partial{\alpha}\partial{x}}=\frac{\part ial^2{f(x, \alpha)}}{\partial{x}\partial{\alpha}}$

I'm not very comfortable with those matrix gradients (I can't even convince myself why your $\displaystyle \frac{\partial f}{\partial \alpha}$ is correct...), so I won't help you with the actual computations, sorry

Yes, you are quite right! Thanks a lot!(Clapping)
• Sep 30th 2008, 12:41 AM
ggyyree
Quote:

Originally Posted by choovuck
P.S. When you wrote $\displaystyle f(x, \alpha) = \frac{1}{2}[(Ax-y_1)^2+(AP(\alpha)x-y_2)^2]$, I'm assuming you meant $\displaystyle f(x, \alpha) = \frac{1}{2}[(Ax-y_1)^T (Ax-y_1)+(AP(\alpha)x-y_2)^T (AP(\alpha)x-y_2)]$

Yes, you are right!
• Sep 30th 2008, 12:41 AM
ggyyree
According to your suggestion I have derived the formula. Thank you very much!(Rofl)
• Sep 30th 2008, 12:46 AM
ggyyree

I want to know the relations between 'Gradient', 'Jacobian' and 'Hessian'. What I think they are:

'Function': $\displaystyle F$
'Gradient': $\displaystyle G$
'Jacobian' : $\displaystyle J$
'Hessian': $\displaystyle H$

$\displaystyle G= -J^T \times F$
$\displaystyle H= J^T \times J$

Am I right? Are there any definition for these relations please? Thanks very much!(Wondering)
• Sep 30th 2008, 08:42 PM
choovuck