Results 1 to 2 of 2

Thread: Kullback-Lieber divergence

  1. #1
    Oct 2012

    Kullback-Lieber divergence

    The Kullback-Lieber divergence between two distributions with pdfs f(x) and g(x) is defined
    $KL(F;G) = \int_{-\infty}^{\infty} ln \left(\frac{f(x)}{g(x)}\right)f(x)dx$

    Compute the Kullback-Lieber divergence when F is the standard normal distribution and G
    is the normal distribution with mean  and variance 1. For what value of  is the divergence

    I was never instructed on this kind of divergence so I am a bit lost on how to solve this kind of integral. I get that I can simplify my two normal equations in the natural log but my guess is that I should wait until after I take the integral. Any help is appreciated.
    Follow Math Help Forum on Facebook and Google+

  2. #2
    GJA is offline
    Jul 2012

    Re: Kullback-Lieber divergence

    Hi WUrunner,

    If we simplify what's inside the logarithm we should get

    $\displaystyle KL(F;G)=\int_{-\infty}^{\infty}\left(\frac{1}{2}-x\right)\left(\frac{1}{\sqrt{2\pi}}e^{-\frac{x^{2}}{2}}\right)dx.$

    Now multiply this out to get

    $\displaystyle KL(F;G)=\frac{1}{2}\frac{1}{\sqrt{2\pi}}\int_{-\infty}^{\infty} }e^{-\frac{x^{2}}{2}}\right)dx + \frac{1}{\sqrt{2\pi}}\int_{-\infty}^{\infty}(-x)}e^{-\frac{x^{2}}{2}}\right)dx. $

    Now we know that the first integral is $\displaystyle \sqrt{2\pi}$ (see Gaussian integral - Wikipedia, the free encyclopedia). The second integral can be computed using a u-substitution or by noting that we're integrating an odd function about a symmetric interval. When we put all the pieces together (if I've done the computations correctly) we should get

    $\displaystyle KL(F;G)=\frac{1}{2}.$

    Does this straighten things out? Let me know if anything is unclear. Good luck!
    Last edited by GJA; Dec 17th 2012 at 11:51 AM.
    Follow Math Help Forum on Facebook and Google+

Similar Math Help Forum Discussions

  1. calculation of distance using Kullback–Leibler divergence
    Posted in the Advanced Statistics Forum
    Replies: 0
    Last Post: Mar 3rd 2011, 02:08 AM
  2. Kullback-Leibler distance for multivariate normal distribution
    Posted in the Advanced Statistics Forum
    Replies: 3
    Last Post: May 18th 2010, 06:01 AM
  3. divergence
    Posted in the Differential Geometry Forum
    Replies: 2
    Last Post: May 4th 2010, 05:09 PM
  4. Help! Simplification of Kullback Liebler Divergence
    Posted in the Advanced Statistics Forum
    Replies: 0
    Last Post: Sep 12th 2009, 10:59 AM
  5. divergence
    Posted in the Calculus Forum
    Replies: 2
    Last Post: Aug 4th 2008, 06:12 AM

Search Tags

/mathhelpforum @mathhelpforum