Results 1 to 4 of 4

Thread: "Information function" help? Thanks

  1. #1
    Newbie
    Joined
    Oct 2010
    Posts
    2

    "Information function" help? Thanks

    Here's the question
    ---------------------------------

    Let I be the unit interval, i.e. I = {x | 0 x 1}, and let I N be the N-fold cross product of I with itself, i.e., the unit N-box. Let P be that subset of I N consisting of points with positive coordinates summing to 1, i.e.

    $\displaystyle P=\{(p_{1},p_{2},...,p_{N})1^N|\sum_p{i}=1, p{i}>0\}$

    Let R be the real numbers and define the information function H : P R by

    $\displaystyle H(p_{1},p_{2},...,p_{N})=-\sum_{p_{i}=1}^{N} p{i}\log{p{i}}$

    H gives the amount of information in a communication system with N alternative messages where the ith message is transmitted with probability pi. (Interesting fact: If the base of the logarithm is chosen to be 2, then the unit of information is the bit, and corresponds to the amount of information in one yes-no question.) Prove that if all the messages are transmitted with equal probability, then the amount of information is equal to log N.

    --------------

    Here's what I did

    $\displaystyle
    Let p{i}=\frac{1}{N}$

    $\displaystyle -\sum_{p_{i}=1}^{N} p{i}\log{p{i}}=[(-\frac{1}{2}\log\frac{1}{2})+(-\frac{1}{3}\log\frac{1}{3})+...+(-\frac{1}{N}\log\frac{1}{N})]$

    $\displaystyle =[(\log2^\frac{1}{2})+(\log3^\frac{1}{3})+...+(\log{ N}^\frac{1}{N})]
    $

    Then I stuck here.
    Did I head to wrong direction?
    How should I solve this?
    Thank you.
    Follow Math Help Forum on Facebook and Google+

  2. #2
    MHF Contributor

    Joined
    Apr 2005
    Posts
    19,779
    Thanks
    3028
    At this point, you might use the fact that log(a)+ log(b)= log(ab).
    Follow Math Help Forum on Facebook and Google+

  3. #3
    MHF Contributor chisigma's Avatar
    Joined
    Mar 2009
    From
    near Piacenza (Italy)
    Posts
    2,162
    Thanks
    6
    For $\displaystyle p_{i}= \frac{1}{N}$ is $\displaystyle \ln p_{i} = - \ln N$ so that is...

    $\displaystyle \displaystyle H= - \sum_{i=1}^{N} p_{i}\ \ln p_{i} = \ln N\ \sum_{i=1}^{N} \frac{1}{N} = \ln N$ (1)

    Kind regards

    $\displaystyle \chi$ $\displaystyle \sigma$
    Follow Math Help Forum on Facebook and Google+

  4. #4
    MHF Contributor
    Opalg's Avatar
    Joined
    Aug 2007
    From
    Leeds, UK
    Posts
    4,041
    Thanks
    10
    Quote Originally Posted by mrsi View Post
    Here's the question
    ---------------------------------

    Let I be the unit interval, i.e. $\displaystyle I = \{x\ |\ 0 \leqslant x \leqslant 1\}$, and let $\displaystyle I^ N$ be the N-fold cross product of I with itself, i.e., the unit N-box. Let P be that subset of $\displaystyle I^ N$ consisting of points with positive coordinates summing to 1, i.e.

    $\displaystyle P=\{(p_{1},p_{2},...,p_{N})\ |\ \sum\limits_{i=1}^Np_i=1,\ p_{i}>0\}$

    Let R be the real numbers and define the information function $\displaystyle H : P \to R$ by

    $\displaystyle H(p_{1},p_{2},...,p_{N})=-\sum\limits_{i=1}^{N} p_{i}\log{p_{i}}$

    [I have modified that to correct several typos.]

    H gives the amount of information in a communication system with N alternative messages where the ith message is transmitted with probability pi. (Interesting fact: If the base of the logarithm is chosen to be 2, then the unit of information is the bit, and corresponds to the amount of information in one yes-no question.) Prove that if all the messages are transmitted with equal probability, then the amount of information is equal to log N.

    --------------

    Here's what I did

    Let $\displaystyle p_{i}=\frac{1}{N}$

    $\displaystyle -\sum\limits_{i=1}^{N} p_{i}\log{p_{i}}=[(-\frac{1}{2}\log\frac{1}{2})+(-\frac{1}{3}\log\frac{1}{3})+...+(-\frac{1}{N}\log\frac{1}{N})]$
    That last line is wrong [edit: as chisigma has just pointed out]. If each $\displaystyle p_i$ is equal to 1/N then then sum consists of N terms, each of which is equal to $\displaystyle -\frac1N\log\frac1N$. (In other words, instead of terms with 2,3,...,N, each term should be the same as the last term.) Then the sum is equal to $\displaystyle N\bigl(-\frac1N\log\frac1N\bigr) = -\log \frac1N = \log N$.
    Follow Math Help Forum on Facebook and Google+

Similar Math Help Forum Discussions

  1. Replies: 3
    Last Post: Oct 17th 2011, 02:50 PM
  2. Replies: 1
    Last Post: Sep 16th 2011, 01:08 AM
  3. Replies: 2
    Last Post: Jun 4th 2011, 12:11 PM
  4. Replies: 2
    Last Post: Apr 24th 2011, 07:01 AM
  5. Replies: 1
    Last Post: Oct 25th 2010, 04:45 AM

Search Tags


/mathhelpforum @mathhelpforum