Kullback-Lieber divergence

The Kullback-Lieber divergence between two distributions with pdfs f(x) and g(x) is defined

by

$KL(F;G) = \int_{-\infty}^{\infty} ln \left(\frac{f(x)}{g(x)}\right)f(x)dx$

Compute the Kullback-Lieber divergence when F is the standard normal distribution and G

is the normal distribution with mean and variance 1. For what value of is the divergence

minimized?

I was never instructed on this kind of divergence so I am a bit lost on how to solve this kind of integral. I get that I can simplify my two normal equations in the natural log but my guess is that I should wait until after I take the integral. Any help is appreciated.

Re: Kullback-Lieber divergence

Hi WUrunner,

If we simplify what's inside the logarithm we should get

Now multiply this out to get

Now we know that the first integral is (see Gaussian integral - Wikipedia, the free encyclopedia). The second integral can be computed using a u-substitution or by noting that we're integrating an odd function about a symmetric interval. When we put all the pieces together (if I've done the computations correctly) we should get

Does this straighten things out? Let me know if anything is unclear. Good luck!