# How to get the Hessian Matrix please?

• Sep 28th 2008, 01:56 PM
ggyyree
How to get the Hessian Matrix please?
I have got the following function constructed by matrices and vectors:

$
f(x, \alpha) = \frac{1}{2}[(Ax-y_1)^2+(AP(\alpha)x-y_2)^2]
$

in which, $x, y_1, y_2$ are vectors. $A, P(\alpha)$ are matrices.
The gradients with respect to $x, \alpha$ are:

$
\frac{\partial{f(x, \alpha)}}{\partial{x}} = A^T A x - A^T y_1 + P(\alpha)^T A^T A P(\alpha) x - P(\alpha)^T A^T y_2
$

$
\frac{\partial{f(x, \alpha)}}{\partial{\alpha}} = x^T P(\alpha)^T A^T A P(\alpha) x - x^T P'(\alpha)^T A^T y_2
$

in which, $A^T$ is the transpose of $A$ and $P'(\alpha)$ is the first order derivative of $P$.
Anyone knows how to get the Hessian of the $f(x, \alpha)$? Thanks a lot! (Happy)
• Sep 28th 2008, 09:02 PM
choovuck
just take second derivatives the way you took first ones, and write the Hessian as the 2x2 matrix each entry of which is also a matrix (I guess it can be viewed as a tensor if needed)
• Sep 29th 2008, 12:26 AM
ggyyree
Quote:

Originally Posted by choovuck
just take second derivatives the way you took first ones, and write the Hessian as the 2x2 matrix each entry of which is also a matrix (I guess it can be viewed as a tensor if needed)

Thanks a lot! But could you please tell me more details? I guess it should be:

$
HessianMatrix=
\begin{bmatrix}
\frac{\partial{f(x, \alpha)}}{\partial{x}\partial{x}}
&\frac{\partial{f(x, \alpha)}}{\partial{\alpha}\partial{\alpha}} \\
\frac{\partial{f(x, \alpha)}}{\partial{\alpha}\partial{\alpha}}
&0
\end{bmatrix}
$

because the Hessian should be symmetric, am I right?

Moreover, due to the equation:

$
\frac{\partial{f(x, \alpha)}}{\partial{\alpha}} = x^T P(\alpha)^T A^T A P(\alpha) x - x^T P'(\alpha)^T A^T y_2
$

then the second order partial derivative:

$
\frac{\partial{f(x, \alpha)}}{\partial{\alpha}\partial{\alpha}}
$

will involve $P''(\alpha)$. I do not know if I am correct. Please show me the analytic form of the Hessian. Thank you very much!
• Sep 29th 2008, 02:59 PM
choovuck
Quote:

Originally Posted by ggyyree
I guess it should be:

$
HessianMatrix=
\begin{bmatrix}
\frac{\partial{f(x, \alpha)}}{\partial{x}\partial{x}}
&\frac{\partial{f(x, \alpha)}}{\partial{\alpha}\partial{\alpha}} \\
\frac{\partial{f(x, \alpha)}}{\partial{\alpha}\partial{\alpha}}
&0
\end{bmatrix}
$

because the Hessian should be symmetric, am I right?

no, the Hessian matrix has as its (i,j)-th element the derivative $\frac{\partial^2 f}{\partial x_i \partial x_j}$, so in our case
$
HessianMatrix=
\begin{bmatrix}
\frac{\partial^2{f(x, \alpha)}}{\partial{x}^2}
&\frac{\partial^2{f(x, \alpha)}}{\partial{x}\partial{\alpha}} \\
\frac{\partial^2{f(x, \alpha)}}{\partial{\alpha}\partial{x}}
&\frac{\partial^2{f(x, \alpha)}}{\partial{\alpha}\partial{\alpha}}
\end{bmatrix}
$

It is symmetric since $\frac{\partial^2{f(x, \alpha)}}{\partial{\alpha}\partial{x}}=\frac{\part ial^2{f(x, \alpha)}}{\partial{x}\partial{\alpha}}$

I'm not very comfortable with those matrix gradients (I can't even convince myself why your $\frac{\partial f}{\partial \alpha}$ is correct...), so I won't help you with the actual computations, sorry
• Sep 29th 2008, 03:03 PM
choovuck
P.S. When you wrote $f(x, \alpha) = \frac{1}{2}[(Ax-y_1)^2+(AP(\alpha)x-y_2)^2]$, I'm assuming you meant $f(x, \alpha) = \frac{1}{2}[(Ax-y_1)^T (Ax-y_1)+(AP(\alpha)x-y_2)^T (AP(\alpha)x-y_2)]$
• Sep 30th 2008, 12:40 AM
ggyyree
Quote:

Originally Posted by choovuck
no, the Hessian matrix has as its (i,j)-th element the derivative $\frac{\partial^2 f}{\partial x_i \partial x_j}$, so in our case
$
HessianMatrix=
\begin{bmatrix}
\frac{\partial^2{f(x, \alpha)}}{\partial{x}^2}
&\frac{\partial^2{f(x, \alpha)}}{\partial{x}\partial{\alpha}} \\
\frac{\partial^2{f(x, \alpha)}}{\partial{\alpha}\partial{x}}
&\frac{\partial^2{f(x, \alpha)}}{\partial{\alpha}\partial{\alpha}}
\end{bmatrix}
$

It is symmetric since $\frac{\partial^2{f(x, \alpha)}}{\partial{\alpha}\partial{x}}=\frac{\part ial^2{f(x, \alpha)}}{\partial{x}\partial{\alpha}}$

I'm not very comfortable with those matrix gradients (I can't even convince myself why your $\frac{\partial f}{\partial \alpha}$ is correct...), so I won't help you with the actual computations, sorry

Yes, you are quite right! Thanks a lot!(Clapping)
• Sep 30th 2008, 12:41 AM
ggyyree
Quote:

Originally Posted by choovuck
P.S. When you wrote $f(x, \alpha) = \frac{1}{2}[(Ax-y_1)^2+(AP(\alpha)x-y_2)^2]$, I'm assuming you meant $f(x, \alpha) = \frac{1}{2}[(Ax-y_1)^T (Ax-y_1)+(AP(\alpha)x-y_2)^T (AP(\alpha)x-y_2)]$

Yes, you are right!
• Sep 30th 2008, 12:41 AM
ggyyree
According to your suggestion I have derived the formula. Thank you very much!(Rofl)
• Sep 30th 2008, 12:46 AM
ggyyree

I want to know the relations between 'Gradient', 'Jacobian' and 'Hessian'. What I think they are:

'Function': $F$
'Gradient': $G$
'Jacobian' : $J$
'Hessian': $H$

$G= -J^T \times F$
$H= J^T \times J$

Am I right? Are there any definition for these relations please? Thanks very much!(Wondering)
• Sep 30th 2008, 08:42 PM
choovuck