Information Entropy Problem

**mathstudentcpm** · September 4th 2011, 11:59 AM

can anyone show me the proof how the logarithmic series on the left becomes the expression on the right hand side . kindly see the attached file..Thanks in advance for any help.

regards

[IMG]file:///C:/Users/sazid/AppData/Local/Temp/msohtmlclip1/01/clip_image002.gif[/IMG]

**Siron** · September 4th 2011, 12:17 PM

Are there any summation bounds given?

**mathstudentcpm** · September 5th 2011, 01:53 AM

Originally Posted by Siron

Are there any summation bounds given?

s is not the base in the attached file. s^n is a positive real number and the series bound is 1 to a finite positive term.

**Siron** · September 5th 2011, 02:08 AM

Originally Posted by mathstudentcpm

s is not the base in the attached file. s^n is a positive real number and the series bound is 1 to a finite positive term.

$\sum_{n=1}^{k} \frac{1}{s^2}\log_s(n)=\log_s(n)$

Is this what you have to prove?

**mathstudentcpm** · September 5th 2011, 03:54 AM

Originally Posted by Siron

$\sum_{n=1}^{k} \frac{1}{s^2}\log_s(n)=\log_s(n)$

Is this what you have to prove?

not exactly...ok i have attached the (mutu_inf_best.pdf)file (its a research paper). Kindly see the second page of the paper and i have denoted with a double headed arrow("hanging on forum"). Thats the thing i have to proof.

**CaptainBlack** · September 5th 2011, 05:16 AM

Originally Posted by mathstudentcpm

not exactly...ok i have attached the (mutu_inf_best.pdf)file (its a research paper). Kindly see the second page of the paper and i have denoted with a double headed arrow("hanging on forum"). Thats the thing i have to proof.

There is no variable of summation, it cannot be $n$ or $s$ as they appear on the right as well as under the summation sign. You need to look at the meanings of the symbols not the formula to see why this is true.

You are interested in messages of length $n$ in an alphabet with $s$ symbols, there are $s^n$ such messages.

Then as by assumption each message is equally likely:

$p_i=\frac{1}{s^n},\ i=1...s^n$

So the entropy becomes:

$H=-\sum_{i=1}^{s^n} \frac{1}{s^n} \log\left(\frac{1}{s^n}\right)=\sum_{i=1}^{s^n} \frac{1}{s^n} \log\left({s^n}\right)=s_n\left(\frac{1}{s^n} \log({s^n} )\right)$

.... $= \log(s^n)$

CB

**mathstudentcpm** · September 5th 2011, 10:01 AM

Originally Posted by CaptainBlack

There is no variable of summation, it cannot be $n$ or $s$ as they appear on the right as well as under the summation sign. You need to look at the meanings of the symbols not the formula to see why this is true.

You are interested in messages of length $n$ in an alphabet with $s$ symbols, there are $s^n$ such messages.

Then as by assumption each message is equally likely:

$p_i=\frac{1}{s^n},\ i=1...s^n$

So the entropy becomes:

$H=-\sum_{i=1}^{s^n} \frac{1}{s^n} \log\left(\frac{1}{s^n}\right)=\sum_{i=1}^{s^n} \frac{1}{s^n} \log\left({s^n}\right)=s_n\left(\frac{1}{s^n} \log({s^n} )\right)$

.... $= \log(s^n)$

CB

Thanks Guru!

Thread: Information Entropy Problem

LinkBack

Thread Tools

Display

Information Entropy Problem

Re: Logarithmic series

Re: Logarithmic series

Re: Logarithmic series

Re: Logarithmic series

Information Entropy Problem

Re: Information Entropy Problem

Similar Math Help Forum Discussions

Entropy questions

Relative increase in entropy is always less than relative increase in maximum entropy

Entropy

Entropy

Query about Information Entropy formula

Search Tags