The principle of maximum entropy

Say that you are a statistician and are asked to come up with a probability distribution for the current state of knowledge on some particular topic you know little about. (This, in Bayesian statistics, is known as choosing a suitable prior.) To do this, the safest bet is coming up with the least informative distribution via the principle of maximum entropy.

This principle is clearly explained by Jaynes (1968): consider a die which has been tossed a very large number of times $N$. We expect the average to be 3.5, that is, we expect a distribution where $P_n = \frac{1}{6}$ for each $n$, see the figure below.

Instead, we are told that the average is $4.5$. How likely is it for each number $n = 1,2, \ldots, 6$ to come up for the next toss?

Since we know that $P$ always sums to 1, we have

$$\sum_{n=1}^6 P_n = 1.$$

We also know that the average is $4.5$, that is,

$$\sum_{n=1}^6 n \cdot P_n = 4.5.$$

We could satisfy these constraints by choosing $P_4 = P_5 = \frac{1}{2}$.

This is unlikely to be the distribution for our data since it can be derived in relatively few ways, namely: by throwing only $4$ and $5$, and in such a way that the throws average to $4.5$. A more likely distribution would be

This is still not the least informative distribution since it assumes $n = 1$ and $n = 2$ to be impossible events. Jaynes presents the straight line solution $P_n = (12n - 7)/210$,

This solution would also fail if the mean would have been higher, because then $P_0 = 0$ would occur again. The correct measure is the following information measure (Shannon, 1948) which is also known as information entropy,

$$S_I = - \sum_i p_i \log p_i.$$

We can find $p_i$ for $p_i = 1, 2, \ldots, 6$ by maximizing $S_I$ for given constraints. This problem, known as MaxEnt, is hard to solve manually since there are $6$ unknowns and various constraints. The solution can be approximated by rewriting it to a linear program.

Alternatively, analytic solutions exist for some subsets of this Shanon entropy maximization problem (Zabarankin and Uryasev, 2014) Here, we have that the mean is known (and nothing else), so the number of moments $m$ is $1$. Then, the maximum entropy distribution takes the form (Zabarankin and Uryasev, 2014; Eq. 5.1.7),

$$P_n = \frac{e^{\rho n}}{\sum_{n=1}^6 e^{\rho n}}, \: \text{ for } n = 1, 2, ..., 6.$$

This function satisfies $\sum_{n=1}^6 P_n = 1$ for any $\rho$. Now, we only have to find the $\rho$ for which the average is $4.5$. After some trial and error, you'll find that $\rho = 0.3715$ gives $\sum_{n=1}^6 n \cdot P_n \approx 4.501$.

This is the least informative distribution which satisfies the constraints. In other words, this is the distribution which can be obtained in the largest number of ways, given the constraints. For another example of maximum entropy distributions, see Chapter 10.1 of the book by McElreath (2020).

Trial and error