Results 1 to 2 of 2

Thread: joint entropy

  1. #1
    Super Member
    Aug 2009

    joint entropy

    given that x={0,1}, y={0,1}, and that p(0,0)=p(1,1)=a/2 and p(0,1)=p(1,0)=(1-a)/2,why is it that the joint entropy can never be zero?
    Follow Math Help Forum on Facebook and Google+

  2. #2
    Junior Member
    Oct 2011

    Re: joint entropy

    The joint entropy is:

    $\displaystyle -\sum_i p_i \log(p_i) = -2 \frac{a}{2} \log\left(\frac{a}{2}\right) - 2 \frac{1-a}{2} \log\left(\frac{1-a}{2}\right) = 2 H\left(\frac{a}{2}\right)$

    where $\displaystyle H(x) = -x\log(x)-(1-x)\log(1-x)$ is the so-called binary entropy function, which goes to zero only for $\displaystyle x=0$ or $\displaystyle x=1$. So your entropy expression can only go to zero for $\displaystyle a=0$ or $\displaystyle a=2$.
    Follow Math Help Forum on Facebook and Google+

Similar Math Help Forum Discussions

  1. entropy
    Posted in the Advanced Statistics Forum
    Replies: 0
    Last Post: May 3rd 2012, 01:19 PM
  2. Replies: 0
    Last Post: Aug 3rd 2010, 07:02 PM
  3. Replies: 5
    Last Post: Dec 5th 2009, 11:30 PM
  4. Entropy
    Posted in the Discrete Math Forum
    Replies: 2
    Last Post: Oct 6th 2008, 10:48 AM
  5. Entropy
    Posted in the Advanced Applied Math Forum
    Replies: 4
    Last Post: Dec 27th 2007, 04:58 AM

Search Tags

/mathhelpforum @mathhelpforum