# Derivative of relative entropy

• Mar 15th 2012, 03:32 AM
raphw
(Solved) Derivative of relative entropy
Hello,
I am currently confused over relative entropy as it is defined here: Relative Entropy -- from Wolfram MathWorld
Rewritten in a more tidy fashion, I have:
$\displaystyle \mathcal{H}(\Pr, {\Pr}_{0}) = \sum_{\omega \in \Omega} \Pr(\omega) \cdot \log\frac{\Pr(\omega)}{{\Pr}_{0}(\omega)}}$

I just computed the first and second derivative, because I want to demonstrate how a marginal alteration of probability $\displaystyle \Pr(\hat\omega)$ will cause a growth of relative entropy that becomes the bigger, the more $\displaystyle \Pr(\hat\omega)$ differs from $\displaystyle {\Pr}_0(\hat\omega)$. At least this is what my textbook says on the intuition of relative entropy. Now I eneded up with this:

$\displaystyle \frac{\partial \mathcal{H}(\Pr,{\Pr}_{0})}{\partial \Pr(\hat\omega)} & = & \Big(\ln\frac{\Pr(\hat\omega)}{{\Pr}_{0}(\hat\omeg a)} + 1\Big) \cdot \frac{1}{\ln 2}$

and

$\displaystyle \frac{\partial^2 \mathcal{H}(\Pr,{\Pr}_{0})}{\partial \big(\Pr(\hat\omega)\big)^2} & = & \frac{1}{{\Pr}(\hat\omega)} \cdot \frac{1}{\ln 2}$

I am very much confused about the "+1" in the upper formula. Why is it there? Should setting $\displaystyle \Pr(\hat\omega) = {\Pr}_0(\hat\omega)$ not cause a partial minimum? According to what I know, the derivative should then be zero, not $\displaystyle \frac{1}{\ln 2}$.

Help is very appreciated!
Best, Rafael
• Mar 15th 2012, 03:52 AM
raphw
Re: Derivative of relative entropy
I think, writing the problem down just answered my question. Of course, the partial derivative does not help me, since i the minimization must take the interdependency of all $\displaystyle \omega \neq \hat\omega$ into account. By just deriving to $\displaystyle \Pr(\omega) = 0$, I would already find a minimum which is why my derivative indicates differently than my intention. Thanks for reading!