1. ## Kullback-Lieber divergence

The Kullback-Lieber divergence between two distributions with pdfs f(x) and g(x) is defined
by
$KL(F;G) = \int_{-\infty}^{\infty} ln \left(\frac{f(x)}{g(x)}\right)f(x)dx$

Compute the Kullback-Lieber divergence when F is the standard normal distribution and G
is the normal distribution with mean  and variance 1. For what value of  is the divergence
minimized?

I was never instructed on this kind of divergence so I am a bit lost on how to solve this kind of integral. I get that I can simplify my two normal equations in the natural log but my guess is that I should wait until after I take the integral. Any help is appreciated.

2. ## Re: Kullback-Lieber divergence

Hi WUrunner,

If we simplify what's inside the logarithm we should get

$KL(F;G)=\int_{-\infty}^{\infty}\left(\frac{1}{2}-x\right)\left(\frac{1}{\sqrt{2\pi}}e^{-\frac{x^{2}}{2}}\right)dx.$

Now multiply this out to get

$KL(F;G)=\frac{1}{2}\frac{1}{\sqrt{2\pi}}\int_{-\infty}^{\infty} }e^{-\frac{x^{2}}{2}}\right)dx + \frac{1}{\sqrt{2\pi}}\int_{-\infty}^{\infty}(-x)}e^{-\frac{x^{2}}{2}}\right)dx.$

Now we know that the first integral is $\sqrt{2\pi}$ (see Gaussian integral - Wikipedia, the free encyclopedia). The second integral can be computed using a u-substitution or by noting that we're integrating an odd function about a symmetric interval. When we put all the pieces together (if I've done the computations correctly) we should get

$KL(F;G)=\frac{1}{2}.$

Does this straighten things out? Let me know if anything is unclear. Good luck!