Results 1 to 2 of 2

Thread: joint entropy

  1. #1
    Super Member
    Aug 2009

    joint entropy

    given that x={0,1}, y={0,1}, and that p(0,0)=p(1,1)=a/2 and p(0,1)=p(1,0)=(1-a)/2,why is it that the joint entropy can never be zero?
    Follow Math Help Forum on Facebook and Google+

  2. #2
    Junior Member
    Oct 2011

    Re: joint entropy

    The joint entropy is:

    -\sum_i p_i \log(p_i) = -2 \frac{a}{2} \log\left(\frac{a}{2}\right) - 2 \frac{1-a}{2} \log\left(\frac{1-a}{2}\right) = 2 H\left(\frac{a}{2}\right)

    where H(x) = -x\log(x)-(1-x)\log(1-x) is the so-called binary entropy function, which goes to zero only for x=0 or x=1. So your entropy expression can only go to zero for a=0 or a=2.
    Follow Math Help Forum on Facebook and Google+

Similar Math Help Forum Discussions

  1. entropy
    Posted in the Advanced Statistics Forum
    Replies: 0
    Last Post: May 3rd 2012, 02:19 PM
  2. Replies: 0
    Last Post: Aug 3rd 2010, 08:02 PM
  3. Replies: 5
    Last Post: Dec 6th 2009, 12:30 AM
  4. Entropy
    Posted in the Discrete Math Forum
    Replies: 2
    Last Post: Oct 6th 2008, 11:48 AM
  5. Entropy
    Posted in the Advanced Applied Math Forum
    Replies: 4
    Last Post: Dec 27th 2007, 05:58 AM

Search Tags

/mathhelpforum @mathhelpforum