or equivalently ogSββ On the other hand, from()we have A(S) A(2)<A(s+1 aA(s) BA(t)<(a+1A(S) or 段A+ AG罗 Thus A(S) 10gS/ A(t) log t When B is large enough, we have A(t)=k log t
a b log t log S < b a + 1 b or equivalently On the other hand, from (*) we have A(S ) A(t ) < A(S ) a a b a+1 or A(s) bA(t) < (a+1)A(S) (**) b a A(t) A(S) < a b + 1 b (***) Thus A(t) A(S) - log t log S < 1 b When b is large enough, we have A(t) = k log t
(b) In the case of unequal and rational probabilities Let where n i- positive integer for all i then the unequal distribution entropy Hs(p,.,Pv) becomes the case of equal probabilities: H(
(b) In the case of unequal and rational probabilities Let ni ni i=1 N i then the unequal distribution entropy H (p , …, p ) becomes the case of equal probabilities: i i = n , n1 H ( … , …, ... , … , ... ) n n i N p = where n -- positive integer for all i. S 1 N
On one hand we have )=A(1/)=klog(1/) On the other hand HSO )sH1p,…P)+;Hm1/n,…1m) n i=1 Hs(P1,…,队+k-1p;log Hence H(P,…,R)=kg(/)1;P吗 n 1P0g早 0 (c)If p are irrational, the equation also valid, see (1)
On one hand we have H ( … ) = A(1/ ) = k log (1/ ) On the other hand H ( … ) = H (p , …, p ) + p H (1/n , …, 1/n ) = H (p , …, p ) + A(n ) = H (p , …, p ) + k log n 1 N i i i N i=1 S S S S S 1 N i=1 N p i i=1 N p i S 1 N Hence H (p , …, p ) = k[log(1/ ) - p log n ] = - k p log n = - k p log p i i i=1 N i i i=1 N i=1 N i i i i 1 N (c) If p are irrational, the equation also valid, see (1). S
In the case of ideal observation H(1,0,…,0) (0log0=0) k Let n=2, p=p, the base of logarithm takes the value 2 and H(1/2, 1/2)=I bit, then kl. Therefore we have P1,…,R)=Hsp1,…,)=- p log p(bit
I(p , …, p ) = H (p , …, p ) - H (1, 0, …, 0) = H (p , …, p ) = - k S S S 1 N 1 N 1 N i=1 N p log p i i Let N=2, p = p , the base of logarithm takes the value 2 and H (1/2, 1/2) = 1 bit, then k=1. Therefore we have In the case of ideal observation I(p , …, p ) = H (p , …, p ) = - p log p i i i=1 N 1 N S 1 N (bits) (0 log 0 = 0)
onclusion For a system with n possible states and their associated probabilities p,,..., P, the average a priori uncertainty about the state of the system is Hs(p,,R The average amount of information about the system obtained after observation in ideal case is numerically equal to the uncertainty it removed
Conclusion For a system with N possible states and their associated probabilities p , …, p , the average a priori uncertainty about the state of the system is H (p , …, p ). 1 N S 1 N The average amount of information about the system obtained after observation in ideal case is numerically equal to the uncertainty it removed