Information Theory Developed by Shannon in the 40s Maximizing the amount of information that can be transmitted over an imperfect communication channel Data compression(entropy) Transmission rate(channel capacity) CCF-ADL at Zhengzhou University, 17 June25-27,2010
CCF-ADL at Zhengzhou University, June 25-27, 2010 17 Information Theory • Developed by Shannon in the 40s • Maximizing the amount of information that can be transmitted over an imperfect communication channel • Data compression (entropy) • Transmission rate (channel capacity)
Basic Concepts in Information Theory Entropy:Measuring uncertainty of a random variable Kullback-Leibler divergence:comparing two distributions Mutual Information:measuring the correlation of two random variables CCF-ADL at Zhengzhou University, 18 June25-27,2010
CCF-ADL at Zhengzhou University, June 25-27, 2010 18 Basic Concepts in Information Theory • Entropy: Measuring uncertainty of a random variable • Kullback-Leibler divergence: comparing two distributions • Mutual Information: measuring the correlation of two random variables