# Maximum entropy given mean and variance

Printable View

• Oct 22nd 2009, 10:58 PM
r2d2bol
Maximum entropy given mean and variance
Hello, I am having trouble setting up the equations for maximum entropy given mean and variance.

The entropy is defined as $f(x) = -\int p(x) \log p(x) dx$. Problem statement is "use lagrangian optimization to find the pdf of maximal entropy if the variance is $\mu$ and variance is $\sigma^2$.

I started with

(a) function to maximize $f(x) = -\int p(x) \log p(x) dx$

(b) first constraint (probabilities must add up to one) $h_1(x) = \int p(x)dx = 1$

(c) second constraint (expectation constraint) $h_2(x) = \int xp(x) dx = \mu$

(d) third constraint (variance constraint) $h_3(x) = \int (x-\mu)^2p(x)dx = \sigma^2$

So the lagrangian looks something like:

$L(x,\lambda_1, \lambda_x, \lambda_3) = -\int p(x) \log p(x) dx + \lambda_1(h(x) - 1) + \lambda_2( h_2(x) - \mu) + \lambda_3( h_3(x) - \sigma^2 )$

(sorry for using h_1,h_2,h_3 in the lagrangian, i get a "latex error, image too big" if I substitute the actual values in the lagrangian formula).

Now I know that for the discrete case, its fairly easy to do and you end up with the normal/gaussian distribution. I'm assuming the answer will be the same in the continuous case, but it appears that I need to use functional derivatives, because I am taking the derivative of a function p(x) , not a variable (at least that's what my math major friend seems to think), but I don't know how to do that. Is there an easier way of doing this?
• Oct 23rd 2009, 12:20 AM
r2d2bol
see Elements of information theory - Google Books , this information theory book uses "functionals" and derivatives over "functions". I'm just a little surprised because this material wasn't covered in class, but perhaps we are required to know this already. At any rate, if anyone knows an alternate solution, feel free to chime in.