Results 1 to 4 of 4

Math Help - "Information function" help? Thanks

  1. #1
    Newbie
    Joined
    Oct 2010
    Posts
    2

    "Information function" help? Thanks

    Here's the question
    ---------------------------------

    Let I be the unit interval, i.e. I = {x | 0 x 1}, and let I N be the N-fold cross product of I with itself, i.e., the unit N-box. Let P be that subset of I N consisting of points with positive coordinates summing to 1, i.e.

    P=\{(p_{1},p_{2},...,p_{N})1^N|\sum_p{i}=1, p{i}>0\}

    Let R be the real numbers and define the information function H : P R by

    H(p_{1},p_{2},...,p_{N})=-\sum_{p_{i}=1}^{N} p{i}\log{p{i}}

    H gives the amount of information in a communication system with N alternative messages where the ith message is transmitted with probability pi. (Interesting fact: If the base of the logarithm is chosen to be 2, then the unit of information is the bit, and corresponds to the amount of information in one yes-no question.) Prove that if all the messages are transmitted with equal probability, then the amount of information is equal to log N.

    --------------

    Here's what I did

    <br />
Let p{i}=\frac{1}{N}

    -\sum_{p_{i}=1}^{N} p{i}\log{p{i}}=[(-\frac{1}{2}\log\frac{1}{2})+(-\frac{1}{3}\log\frac{1}{3})+...+(-\frac{1}{N}\log\frac{1}{N})]

    =[(\log2^\frac{1}{2})+(\log3^\frac{1}{3})+...+(\log{  N}^\frac{1}{N})]<br />

    Then I stuck here.
    Did I head to wrong direction?
    How should I solve this?
    Thank you.
    Follow Math Help Forum on Facebook and Google+

  2. #2
    MHF Contributor

    Joined
    Apr 2005
    Posts
    14,973
    Thanks
    1121
    At this point, you might use the fact that log(a)+ log(b)= log(ab).
    Follow Math Help Forum on Facebook and Google+

  3. #3
    MHF Contributor chisigma's Avatar
    Joined
    Mar 2009
    From
    near Piacenza (Italy)
    Posts
    2,162
    Thanks
    5
    For p_{i}= \frac{1}{N} is \ln p_{i} = - \ln N so that is...

    \displaystyle H=  - \sum_{i=1}^{N} p_{i}\ \ln p_{i} = \ln N\ \sum_{i=1}^{N} \frac{1}{N} = \ln N (1)

    Kind regards

    \chi \sigma
    Follow Math Help Forum on Facebook and Google+

  4. #4
    MHF Contributor
    Opalg's Avatar
    Joined
    Aug 2007
    From
    Leeds, UK
    Posts
    4,041
    Thanks
    7
    Quote Originally Posted by mrsi View Post
    Here's the question
    ---------------------------------

    Let I be the unit interval, i.e. I = \{x\ |\ 0 \leqslant  x \leqslant 1\}, and let I^ N be the N-fold cross product of I with itself, i.e., the unit N-box. Let P be that subset of I^ N consisting of points with positive coordinates summing to 1, i.e.

    P=\{(p_{1},p_{2},...,p_{N})\ |\ \sum\limits_{i=1}^Np_i=1,\ p_{i}>0\}

    Let R be the real numbers and define the information function H : P \to R by

    H(p_{1},p_{2},...,p_{N})=-\sum\limits_{i=1}^{N} p_{i}\log{p_{i}}

    [I have modified that to correct several typos.]

    H gives the amount of information in a communication system with N alternative messages where the ith message is transmitted with probability pi. (Interesting fact: If the base of the logarithm is chosen to be 2, then the unit of information is the bit, and corresponds to the amount of information in one yes-no question.) Prove that if all the messages are transmitted with equal probability, then the amount of information is equal to log N.

    --------------

    Here's what I did

    Let p_{i}=\frac{1}{N}

    -\sum\limits_{i=1}^{N} p_{i}\log{p_{i}}=[(-\frac{1}{2}\log\frac{1}{2})+(-\frac{1}{3}\log\frac{1}{3})+...+(-\frac{1}{N}\log\frac{1}{N})]
    That last line is wrong [edit: as chisigma has just pointed out]. If each p_i is equal to 1/N then then sum consists of N terms, each of which is equal to -\frac1N\log\frac1N. (In other words, instead of terms with 2,3,...,N, each term should be the same as the last term.) Then the sum is equal to N\bigl(-\frac1N\log\frac1N\bigr) = -\log \frac1N = \log N.
    Follow Math Help Forum on Facebook and Google+

Similar Math Help Forum Discussions

  1. Replies: 3
    Last Post: October 17th 2011, 02:50 PM
  2. Replies: 1
    Last Post: September 16th 2011, 01:08 AM
  3. Replies: 2
    Last Post: June 4th 2011, 12:11 PM
  4. Replies: 2
    Last Post: April 24th 2011, 07:01 AM
  5. Replies: 1
    Last Post: October 25th 2010, 04:45 AM

Search Tags


/mathhelpforum @mathhelpforum