or equivalentI 0g <+ βlogsββ On the other hand, from()we have A(S) A()<A(S+I) aA(s)βA(t)<(+)A(S) or aA(t)<01 βAS)ββ Thus A(S logS/Y A(t) log t When b is large enough, we have A(t=k log t
a b log t log S < b a + 1 b or equivalently On the other hand, from (*) we have A(S ) A(t ) < A(S ) a a b a+1 or A(s) bA(t) < (a+1)A(S) (**) b a A(t) A(S) < a b + 1 b (***) Thus A(t) A(S) - log t log S < 1 b When b is large enough, we have A(t) = k log t
(b)In the case of unequal and rational probabilities Let where n, - positive integer for all i then the unequal distribution entropy Hs(P1,.,PN) becomes the case of equal probabilities: n H(
(b) In the case of unequal and rational probabilities Let ni ni i=1 N i then the unequal distribution entropy H (p , …, p ) becomes the case of equal probabilities: i i = n , n1 H ( … , …, ... , … , ... ) n n i N p = where n -- positive integer for all i. S 1 N
On one hand we have H5(…)=A(1/)=klog(1/ On the other hand )=H1(p,…wp)+;H/n,…1) Hsp,…,只)+1pan) HPp1,…,R)+k=P;g Hence Hsq,…=kuog(1 i-l i p log n I -k, p log n log p (c)If p are irrational, the equation also valid, see(1)
On one hand we have H ( … ) = A(1/ ) = k log (1/ ) On the other hand H ( … ) = H (p , …, p ) + p H (1/n , …, 1/n ) = H (p , …, p ) + A(n ) = H (p , …, p ) + k log n 1 N i i i N i=1 S S S S S 1 N i=1 N p i i=1 N p i S 1 N Hence H (p , …, p ) = k[log(1/ ) - p log n ] = - k p log n = - k p log p i i i=1 N i i i=1 N i=1 N i i i i 1 N (c) If p are irrational, the equation also valid, see (1). S
In the case of ideal observation p1…,R)=B(1,…p-(1,0,…,0) (0log0=0) k pi log pi Let n=2, p=p, the base of logarithm takes the value 2 and H(1n2, 1/2)=I bit, then kl. Therefore we have I(p,,…,R)=H(p, log p(bits
I(p , …, p ) = H (p , …, p ) - H (1, 0, …, 0) = H (p , …, p ) = - k S S S 1 N 1 N 1 N i=1 N p log p i i Let N=2, p = p , the base of logarithm takes the value 2 and H (1/2, 1/2) = 1 bit, then k=1. Therefore we have In the case of ideal observation I(p , …, p ) = H (p , …, p ) = - p log p i i i=1 N 1 N S 1 N (bits) (0 log 0 = 0)
Conclusion For a system with n possible states and their associated probabilities p,,..., p, the average a priori uncertainty about the state of the system is Hs(p,,R The average amount of information a bout the system obtained after observation in ideal case is numerically equal to the uncertainty it removed
Conclusion For a system with N possible states and their associated probabilities p , …, p , the average a priori uncertainty about the state of the system is H (p , …, p ). 1 N S 1 N The average amount of information about the system obtained after observation in ideal case is numerically equal to the uncertainty it removed