given that x={0,1}, y={0,1}, and that p(0,0)=p(1,1)=a/2 and p(0,1)=p(1,0)=(1-a)/2,why is it that the joint entropy can never be zero?

Printable View

- May 3rd 2012, 02:25 PMalexandrabel90joint entropy
given that x={0,1}, y={0,1}, and that p(0,0)=p(1,1)=a/2 and p(0,1)=p(1,0)=(1-a)/2,why is it that the joint entropy can never be zero?

- May 3rd 2012, 03:17 PMjensRe: joint entropy
The joint entropy is:

$\displaystyle -\sum_i p_i \log(p_i) = -2 \frac{a}{2} \log\left(\frac{a}{2}\right) - 2 \frac{1-a}{2} \log\left(\frac{1-a}{2}\right) = 2 H\left(\frac{a}{2}\right)$

where $\displaystyle H(x) = -x\log(x)-(1-x)\log(1-x)$ is the so-called binary entropy function, which goes to zero only for $\displaystyle x=0$ or $\displaystyle x=1$. So your entropy expression can only go to zero for $\displaystyle a=0$ or $\displaystyle a=2$.