Results 1 to 2 of 2

Math Help - Maximum entropy given mean and variance

  1. #1
    Newbie
    Joined
    May 2009
    Posts
    6

    Maximum entropy given mean and variance

    Hello, I am having trouble setting up the equations for maximum entropy given mean and variance.

    The entropy is defined as f(x) = -\int p(x) \log p(x) dx. Problem statement is "use lagrangian optimization to find the pdf of maximal entropy if the variance is \mu and variance is \sigma^2.

    I started with

    (a) function to maximize f(x) = -\int p(x) \log p(x) dx

    (b) first constraint (probabilities must add up to one) h_1(x) = \int p(x)dx = 1

    (c) second constraint (expectation constraint)  h_2(x) = \int xp(x) dx = \mu

    (d) third constraint (variance constraint)  h_3(x) = \int (x-\mu)^2p(x)dx = \sigma^2

    So the lagrangian looks something like:

    L(x,\lambda_1, \lambda_x, \lambda_3) = -\int p(x) \log p(x) dx + \lambda_1(h(x) - 1) + \lambda_2( h_2(x) - \mu) + \lambda_3( h_3(x) - \sigma^2 )

    (sorry for using h_1,h_2,h_3 in the lagrangian, i get a "latex error, image too big" if I substitute the actual values in the lagrangian formula).

    Now I know that for the discrete case, its fairly easy to do and you end up with the normal/gaussian distribution. I'm assuming the answer will be the same in the continuous case, but it appears that I need to use functional derivatives, because I am taking the derivative of a function p(x) , not a variable (at least that's what my math major friend seems to think), but I don't know how to do that. Is there an easier way of doing this?
    Follow Math Help Forum on Facebook and Google+

  2. #2
    Newbie
    Joined
    May 2009
    Posts
    6
    see Elements of information theory - Google Books , this information theory book uses "functionals" and derivatives over "functions". I'm just a little surprised because this material wasn't covered in class, but perhaps we are required to know this already. At any rate, if anyone knows an alternate solution, feel free to chime in.
    Follow Math Help Forum on Facebook and Google+

Similar Math Help Forum Discussions

  1. Entropy questions
    Posted in the Advanced Statistics Forum
    Replies: 1
    Last Post: February 24th 2011, 11:53 AM
  2. Replies: 0
    Last Post: August 3rd 2010, 07:02 PM
  3. large deviations/maximum entropy principle
    Posted in the Advanced Statistics Forum
    Replies: 0
    Last Post: September 27th 2009, 12:38 PM
  4. Entropy
    Posted in the Discrete Math Forum
    Replies: 2
    Last Post: October 6th 2008, 10:48 AM
  5. Entropy
    Posted in the Advanced Applied Math Forum
    Replies: 4
    Last Post: December 27th 2007, 04:58 AM

Search Tags


/mathhelpforum @mathhelpforum