1 Attachment(s)

Information Entropy Problem

can anyone show me the proof how the logarithmic series on the left becomes the expression on the right hand side . kindly see the attached file..Thanks in advance for any help.

regards

[IMG]file:///C:/Users/sazid/AppData/Local/Temp/msohtmlclip1/01/clip_image002.gif[/IMG]

Information Entropy Problem

Quote:

Originally Posted by

**mathstudentcpm** not exactly...ok i have attached the (mutu_inf_best.pdf)file (its a research paper). Kindly see the second page of the paper and i have denoted with a double headed arrow("hanging on forum"). Thats the thing i have to proof.

There is no variable of summation, it cannot be $\displaystyle n$ or $\displaystyle s$ as they appear on the right as well as under the summation sign. You need to look at the meanings of the symbols not the formula to see why this is true.

You are interested in messages of length $\displaystyle n$ in an alphabet with $\displaystyle s$ symbols, there are $\displaystyle s^n$ such messages.

Then as by assumption each message is equally likely:

$\displaystyle p_i=\frac{1}{s^n},\ i=1...s^n$

So the entropy becomes:

$\displaystyle H=-\sum_{i=1}^{s^n} \frac{1}{s^n} \log\left(\frac{1}{s^n}\right)=\sum_{i=1}^{s^n} \frac{1}{s^n} \log\left({s^n}\right)=s_n\left(\frac{1}{s^n} \log({s^n} )\right)$

.... $\displaystyle = \log(s^n)$

CB

Re: Information Entropy Problem

Quote:

Originally Posted by

**CaptainBlack** There is no variable of summation, it cannot be $\displaystyle n$ or $\displaystyle s$ as they appear on the right as well as under the summation sign. You need to look at the meanings of the symbols not the formula to see why this is true.

You are interested in messages of length $\displaystyle n$ in an alphabet with $\displaystyle s$ symbols, there are $\displaystyle s^n$ such messages.

Then as by assumption each message is equally likely:

$\displaystyle p_i=\frac{1}{s^n},\ i=1...s^n$

So the entropy becomes:

$\displaystyle H=-\sum_{i=1}^{s^n} \frac{1}{s^n} \log\left(\frac{1}{s^n}\right)=\sum_{i=1}^{s^n} \frac{1}{s^n} \log\left({s^n}\right)=s_n\left(\frac{1}{s^n} \log({s^n} )\right)$

.... $\displaystyle = \log(s^n)$

CB

Thanks Guru!