I am currently confused over relative entropy as it is defined here: Relative Entropy -- from Wolfram MathWorld
Rewritten in a more tidy fashion, I have:
I just computed the first and second derivative, because I want to demonstrate how a marginal alteration of probability will cause a growth of relative entropy that becomes the bigger, the more differs from . At least this is what my textbook says on the intuition of relative entropy. Now I eneded up with this:
I am very much confused about the "+1" in the upper formula. Why is it there? Should setting not cause a partial minimum? According to what I know, the derivative should then be zero, not .
Help is very appreciated!