CATALOG DESCRIPTION:  Information measures and their properties: entropy, divergence, mutual information, channel capacity. Shannon's fundamental theorems for data compression and coding for noisy channels. Applications in communications, statistical inference, probability, physics.

Prerequisites by course: EECS 302 (Probabilities Systems and Random Signals).
Prerequisites by topic: Good understanding of basic probability.  (A review of probability theory will be given in Week 1.)

  • This course fulfills the Theory Depth requirement.

REQUIRED TEXT: Cover & Thomas, Elements of Information Theory, 2nd ed., Wiley, 2006.


  • R. G. Gallager, Information Theory and Reliable Communication, Wiley, 1968.
  • D. J. MacKay, Information Theory, Inference and Learning Algorithms, Cambridge, 2004

COURSE DIRECTOR: Prof. Dongning Guo

Problem Sets: There will be weekly problem sets. Problem sets are required to be handed in by the end of the class (usually on Friday) in which they are due. Each student is allowed to have one exception to turn in his/her homework within 72 hours past due time during the quarter. Late problem sets will not be accepted otherwise. Each of you is encouraged to work on the problem sets on your own and by consulting the textbook. Working together in small groups on the problem sets is encouraged whenever it helps to better learn the material. However, each person must write up his/her own solution to hand in.


PREREQUISITES BY COURSES: EECS 302 (Probabilities Systems and Random Signals)


  1. Overview of information theory and its applications.
  2. Review of probability theory.
  3. Information measures (entropy, divergence, mutual information) and basic properties.
  4. Typical sets and the Asymptotic Equipartition Property.
  5. Data compression/lossless source coding.
  6. Entropy rates for stochastic processes, Markov chains.
  7. Huffman coding, Lempel-Ziv compression.
  8. Channel coding, channel capacity.
  9. The channel coding theorem for discrete memoryless channels.
  10. Converse to the channel coding theorem, joint source channel coding.
  11. Discrete and continuous-time Gaussian channels, band-limited channels.
  12. Source-channel separation.
  13. Rate distortion


  • Problem sets: 30%
  • Midterm exam: 25%
  • Final Exam: 45%