Hello,

I am currently confused over relative entropy as it is defined here: Relative Entropy -- from Wolfram MathWorld

Rewritten in a more tidy fashion, I have:

$\displaystyle \mathcal{H}(\Pr, {\Pr}_{0}) = \sum_{\omega \in \Omega} \Pr(\omega) \cdot \log\frac{\Pr(\omega)}{{\Pr}_{0}(\omega)}}$

I just computed the first and second derivative, because I want to demonstrate how a marginal alteration of probability $\displaystyle \Pr(\hat\omega)$ will cause a growth of relative entropy that becomes the bigger, the more $\displaystyle \Pr(\hat\omega)$ differs from $\displaystyle {\Pr}_0(\hat\omega)$. At least this is what my textbook says on the intuition of relative entropy. Now I eneded up with this:

$\displaystyle \frac{\partial \mathcal{H}(\Pr,{\Pr}_{0})}{\partial \Pr(\hat\omega)} & = & \Big(\ln\frac{\Pr(\hat\omega)}{{\Pr}_{0}(\hat\omeg a)} + 1\Big) \cdot \frac{1}{\ln 2}$

and

$\displaystyle \frac{\partial^2 \mathcal{H}(\Pr,{\Pr}_{0})}{\partial \big(\Pr(\hat\omega)\big)^2} & = & \frac{1}{{\Pr}(\hat\omega)} \cdot \frac{1}{\ln 2}$

I am very much confused about the "+1" in the upper formula. Why is it there? Should setting $\displaystyle \Pr(\hat\omega) = {\Pr}_0(\hat\omega)$ not cause a partial minimum? According to what I know, the derivative should then be zero, not $\displaystyle \frac{1}{\ln 2}$.

Help is very appreciated!

Best, Rafael