Results 1 to 2 of 2

Thread: Derivative of relative entropy

  1. #1
    Sep 2010

    (Solved) Derivative of relative entropy

    I am currently confused over relative entropy as it is defined here: Relative Entropy -- from Wolfram MathWorld
    Rewritten in a more tidy fashion, I have:
    $\displaystyle \mathcal{H}(\Pr, {\Pr}_{0}) = \sum_{\omega \in \Omega} \Pr(\omega) \cdot \log\frac{\Pr(\omega)}{{\Pr}_{0}(\omega)}}$

    I just computed the first and second derivative, because I want to demonstrate how a marginal alteration of probability $\displaystyle \Pr(\hat\omega)$ will cause a growth of relative entropy that becomes the bigger, the more $\displaystyle \Pr(\hat\omega)$ differs from $\displaystyle {\Pr}_0(\hat\omega)$. At least this is what my textbook says on the intuition of relative entropy. Now I eneded up with this:

    $\displaystyle \frac{\partial \mathcal{H}(\Pr,{\Pr}_{0})}{\partial \Pr(\hat\omega)} & = & \Big(\ln\frac{\Pr(\hat\omega)}{{\Pr}_{0}(\hat\omeg a)} + 1\Big) \cdot \frac{1}{\ln 2}$


    $\displaystyle \frac{\partial^2 \mathcal{H}(\Pr,{\Pr}_{0})}{\partial \big(\Pr(\hat\omega)\big)^2} & = & \frac{1}{{\Pr}(\hat\omega)} \cdot \frac{1}{\ln 2}$

    I am very much confused about the "+1" in the upper formula. Why is it there? Should setting $\displaystyle \Pr(\hat\omega) = {\Pr}_0(\hat\omega)$ not cause a partial minimum? According to what I know, the derivative should then be zero, not $\displaystyle \frac{1}{\ln 2}$.

    Help is very appreciated!
    Best, Rafael
    Last edited by raphw; Mar 15th 2012 at 03:55 AM.
    Follow Math Help Forum on Facebook and Google+

  2. #2
    Sep 2010

    Re: Derivative of relative entropy

    I think, writing the problem down just answered my question. Of course, the partial derivative does not help me, since i the minimization must take the interdependency of all $\displaystyle \omega \neq \hat\omega$ into account. By just deriving to $\displaystyle \Pr(\omega) = 0$, I would already find a minimum which is why my derivative indicates differently than my intention. Thanks for reading!
    Follow Math Help Forum on Facebook and Google+

Similar Math Help Forum Discussions

  1. Replies: 0
    Last Post: Aug 3rd 2010, 07:02 PM
  2. Calculations give me a negative relative entropy
    Posted in the Advanced Statistics Forum
    Replies: 3
    Last Post: Nov 11th 2009, 08:37 PM
  3. Replies: 4
    Last Post: Apr 27th 2009, 02:02 PM
  4. First derivative, relative extrema questions
    Posted in the Calculus Forum
    Replies: 4
    Last Post: Jul 16th 2008, 11:07 PM
  5. Entropy
    Posted in the Advanced Applied Math Forum
    Replies: 4
    Last Post: Dec 27th 2007, 04:58 AM

Search Tags

/mathhelpforum @mathhelpforum