1. ## How to get the Hessian Matrix please?

I have got the following function constructed by matrices and vectors:

$
f(x, \alpha) = \frac{1}{2}[(Ax-y_1)^2+(AP(\alpha)x-y_2)^2]
$

in which, $x, y_1, y_2$ are vectors. $A, P(\alpha)$ are matrices.
The gradients with respect to $x, \alpha$ are:

$
\frac{\partial{f(x, \alpha)}}{\partial{x}} = A^T A x - A^T y_1 + P(\alpha)^T A^T A P(\alpha) x - P(\alpha)^T A^T y_2
$

$
\frac{\partial{f(x, \alpha)}}{\partial{\alpha}} = x^T P(\alpha)^T A^T A P(\alpha) x - x^T P'(\alpha)^T A^T y_2
$

in which, $A^T$ is the transpose of $A$ and $P'(\alpha)$ is the first order derivative of $P$.
Anyone knows how to get the Hessian of the $f(x, \alpha)$? Thanks a lot!

2. just take second derivatives the way you took first ones, and write the Hessian as the 2x2 matrix each entry of which is also a matrix (I guess it can be viewed as a tensor if needed)

3. Originally Posted by choovuck
just take second derivatives the way you took first ones, and write the Hessian as the 2x2 matrix each entry of which is also a matrix (I guess it can be viewed as a tensor if needed)
Thanks a lot! But could you please tell me more details? I guess it should be:

$
HessianMatrix=
\begin{bmatrix}
\frac{\partial{f(x, \alpha)}}{\partial{x}\partial{x}}
&\frac{\partial{f(x, \alpha)}}{\partial{\alpha}\partial{\alpha}} \\
\frac{\partial{f(x, \alpha)}}{\partial{\alpha}\partial{\alpha}}
&0
\end{bmatrix}
$

because the Hessian should be symmetric, am I right?

Moreover, due to the equation:

$
\frac{\partial{f(x, \alpha)}}{\partial{\alpha}} = x^T P(\alpha)^T A^T A P(\alpha) x - x^T P'(\alpha)^T A^T y_2
$

then the second order partial derivative:

$
\frac{\partial{f(x, \alpha)}}{\partial{\alpha}\partial{\alpha}}
$

will involve $P''(\alpha)$. I do not know if I am correct. Please show me the analytic form of the Hessian. Thank you very much!

4. Originally Posted by ggyyree
I guess it should be:

$
HessianMatrix=
\begin{bmatrix}
\frac{\partial{f(x, \alpha)}}{\partial{x}\partial{x}}
&\frac{\partial{f(x, \alpha)}}{\partial{\alpha}\partial{\alpha}} \\
\frac{\partial{f(x, \alpha)}}{\partial{\alpha}\partial{\alpha}}
&0
\end{bmatrix}
$

because the Hessian should be symmetric, am I right?
no, the Hessian matrix has as its (i,j)-th element the derivative $\frac{\partial^2 f}{\partial x_i \partial x_j}$, so in our case
$
HessianMatrix=
\begin{bmatrix}
\frac{\partial^2{f(x, \alpha)}}{\partial{x}^2}
&\frac{\partial^2{f(x, \alpha)}}{\partial{x}\partial{\alpha}} \\
\frac{\partial^2{f(x, \alpha)}}{\partial{\alpha}\partial{x}}
&\frac{\partial^2{f(x, \alpha)}}{\partial{\alpha}\partial{\alpha}}
\end{bmatrix}
$

It is symmetric since $\frac{\partial^2{f(x, \alpha)}}{\partial{\alpha}\partial{x}}=\frac{\part ial^2{f(x, \alpha)}}{\partial{x}\partial{\alpha}}$

I'm not very comfortable with those matrix gradients (I can't even convince myself why your $\frac{\partial f}{\partial \alpha}$ is correct...), so I won't help you with the actual computations, sorry

5. P.S. When you wrote $f(x, \alpha) = \frac{1}{2}[(Ax-y_1)^2+(AP(\alpha)x-y_2)^2]$, I'm assuming you meant $f(x, \alpha) = \frac{1}{2}[(Ax-y_1)^T (Ax-y_1)+(AP(\alpha)x-y_2)^T (AP(\alpha)x-y_2)]$

6. Originally Posted by choovuck
no, the Hessian matrix has as its (i,j)-th element the derivative $\frac{\partial^2 f}{\partial x_i \partial x_j}$, so in our case
$
HessianMatrix=
\begin{bmatrix}
\frac{\partial^2{f(x, \alpha)}}{\partial{x}^2}
&\frac{\partial^2{f(x, \alpha)}}{\partial{x}\partial{\alpha}} \\
\frac{\partial^2{f(x, \alpha)}}{\partial{\alpha}\partial{x}}
&\frac{\partial^2{f(x, \alpha)}}{\partial{\alpha}\partial{\alpha}}
\end{bmatrix}
$

It is symmetric since $\frac{\partial^2{f(x, \alpha)}}{\partial{\alpha}\partial{x}}=\frac{\part ial^2{f(x, \alpha)}}{\partial{x}\partial{\alpha}}$

I'm not very comfortable with those matrix gradients (I can't even convince myself why your $\frac{\partial f}{\partial \alpha}$ is correct...), so I won't help you with the actual computations, sorry

Yes, you are quite right! Thanks a lot!

7. Originally Posted by choovuck
P.S. When you wrote $f(x, \alpha) = \frac{1}{2}[(Ax-y_1)^2+(AP(\alpha)x-y_2)^2]$, I'm assuming you meant $f(x, \alpha) = \frac{1}{2}[(Ax-y_1)^T (Ax-y_1)+(AP(\alpha)x-y_2)^T (AP(\alpha)x-y_2)]$
Yes, you are right!

8. According to your suggestion I have derived the formula. Thank you very much!

9. By the way, may I ask you another related question please?

I want to know the relations between 'Gradient', 'Jacobian' and 'Hessian'. What I think they are:

'Function': $F$
'Gradient': $G$
'Jacobian' : $J$
'Hessian': $H$

$G= -J^T \times F$
$H= J^T \times J$

Am I right? Are there any definition for these relations please? Thanks very much!