Alison's New App is now available on iOS and Android! Download Now

Diploma in the Foundations of Information Theory

Learn about the basic principles of quantifying the information using information theory with this free online course.

Publisher: NPTEL
Have you ever wondered about the various ways of quantifying information? This course provides a statistical foundation that focuses on a better understanding of the process of analyzing uncertainty and variability in risk assessments. Take the opportunity by registering now and learn about the mathematical ways of modelling the amount of information in events, random variables, and distributions using various probability techniques.
Diploma in the Foundations of Information Theory
  • Duration

    10-15 Hours
  • Students

  • Accreditation






View course modules


The notion and perception of information have gained significant attention in the contemporary world dominated by the information era. This course aims to present the scientific insights of quantification, storage, and communication of digital information. It begins by explaining the process of modelling uncertainty in information using various probability techniques. You will be taught about the procedure for measuring the amount of information in random variables and distributions. In addition, you will learn about ascertaining the ‘information value’ of a communicated message by computing the degree of randomness using entropy. This will include the process of determining the closeness of two distributions based on total distance variations. Next, the significance of relative entropy in analysing the product measures are illustrated. You will discover how the intrinsic measurements of distance are the key to an understanding of minimum and maximum convergence rates. The course also explains the significance of statistical inequalities in providing bounding measures and quantities that may be difficult or intractable to compute.

Next, the course highlights the various basic elements of statistical inferences. You will study the procedure for estimating and testing the hypothesis using different statistical methods. Following this, you will discover the significance of kullback-leibler divergence in measure the difference between two probability distributions. Subsequently, you will explore how mutual information is used for measuring a relationship between two random variables that are sampled simultaneously. In addition to this, you will learn how Fano’s inequality gives a lower bound on the mutual information between two random variables that take values on an element set. Following this, the various measures of information and their properties are highlighted. You will be taught how probability mass and density functions are used for determining discrete and continuous distributions. This will include the process of quantifying the information into smaller components to build a larger uncertainty using the chain rule. The functional measures of the shape of univariate distributions with respect to the concave and convex transform order are explained.

Finally, the course illustrates the significance of lower bounds for data compression and generating random numbers. This will include the process of deriving lower bounds for the minimum time required by an algorithm for distributed function over a network of point-to-point channels with finite capacity. In addition to this, the significance of strong converse theorem for discrete memoryless channels is explained. You will discover the process of ascertaining a threshold between perfectly reliable and completely unreliable communication. Following this, you will be taught about the process of establishing a strong converse in source coding for super-source networks. Lastly, you will study how an estimation risk can be lower bounded by the probability of error in testing problems using minmax bounds. ‘Diploma in the Foundations of Information Theory’ is an informative course that illustrates the mathematical treatment of the concepts, parameters and rules governing the transmission of messages through communication systems. Enrol in this course now and learn about the methods of measuring redundancy or efficiency of symbolic representation within a given language.

Start Course Now