Diploma in the Foundations of Information Theory

The notion and perception of data have gained significant attention in the contemporary world dominated by the Information Age. This course presents the scientific insights of quantification, storage, and communication of digital information. It explains the process of modelling uncertainty in information using various probability techniques. You will be taught about the procedure for measuring the amount of information in random variables and distributions. In addition, you will learn about ascertaining the ‘information value’ of a communicated message by computing the degree of randomness using entropy. This will include the process of determining the closeness of two distributions based on total distance variations. Next, the significance of relative entropy in analysing the product measures are illustrated. You will discover how the intrinsic measurements of distance are the key to an understanding of minimum and maximum convergence rates. The course also explains the significance of statistical inequalities in providing bounding measures and quantities that may be difficult or intractable to compute.

Next, it highlights the various basic elements of statistical inferences. You will study the procedure for estimating and testing the hypothesis using different statistical methods. Following this, you will discover the significance of Kullback-Leibler divergence in measuring the difference between two probability distributions. Subsequently, you will also explore how mutual information is used for measuring a relationship between two random variables that are sampled simultaneously. In addition, you will see how Fano’s inequality gives a lower bound on the mutual information between two random variables that take values on an element set. The various measures of information and their properties are also highlighted. You will be taught how probability mass and density functions are used for determining discrete and continuous distributions, including the process of quantifying the information into smaller components to build a larger uncertainty using the chain rule. The functional measures of the shape of univariate distributions with respect to the concave and convex transform order are explained.

Finally, the course illustrates the significance of lower bounds for data compression and generating random numbers, including the process of deriving lower bounds for the minimum time required by an algorithm for distributed function over a network of point-to-point channels with finite capacity. In addition, the significance of strong converse theorem for discrete memoryless channels is explained. You will discover the process of ascertaining a threshold between perfectly reliable and completely unreliable communication. Next, you will be taught about the process of establishing a strong converse in source coding for super-source networks. Lastly, study how an estimation risk can be lower bounded by the probability of error in testing problems using minimax bounds. ‘Diploma in the Foundations of Information Theory’ is an informative course that illustrates the mathematical treatment of the concepts, parameters and rules governing the transmission of messages through communication systems. Enrol in this course now and learn about the methods of measuring redundancy or efficiency of symbolic representation within a given language.