Module 1: Randomness and Entropy

Uncertainty and randomness

Randomness is the apparent or actual lack of pattern or predictability in events. A random sequence of events, symbols or steps often has no order and does not follow an intelligible pattern or combination.

Uncertainty refers to epistemic situations involving imperfect or unknown information. It applies to predictions of future events, to physical measurements that are already made, or to the unknown.

Total Variation Distance

The Total Variation Distance is a measure used for assessing the distances between distributions. It is also known as statistical distance, statistical difference or variational distance.

Measuring Randomness in a Random Variable

A Random Variable is a variable whose value is unknown or a function that assigns values to each of an experiment's outcomes.

The steps of measuring the randomness in a random variable are as follows:

Ascertaining the number of bits required to generate a random variable.
Generating samples of unbiased random variables using independent bits.

Typical Sets

The Typical Set is a set of sequences whose probability is close to two raised to the negative power of the entropy of their source distribution.