Index: The Book of Statistical Proofs ▷ Probability Distributions ▷ Univariate continuous distributions ▷ Continuous uniform distribution ▷ Maximum entropy distribution

Theorem: The continuous uniform distribution maximizes differential entropy for a random variable with a fixed range.

Proof: Without loss of generality, let us assume that the random variable $X$ is in the following range: $a \leq X \leq b$.

Let $g(x)$ be the probability density function of a continuous uniform distribution with minimum $a$ and maximum $b$ and let $f(x)$ be an arbitrary probability density function defined on the same support $\mathcal{X} = [a,b]$.

For a random variable $X$ with set of possible values $\mathcal{X}$ and probability density function $p(x)$, the differential entropy is defined as:

\[\label{eq:dent} \mathrm{h}(X) = - \int_{\mathcal{X}} p(x) \log p(x) \, \mathrm{d}x\]

Consider the Kullback-Leibler divergence of distribution $f(x)$ from distribution $g(x)$ which is non-negative:

\[\label{eq:kl-fg} \begin{split} 0 \leq \mathrm{KL}[f||g] &= \int_{\mathcal{X}} f(x) \log \frac{f(x)}{g(x)} \, \mathrm{d}x \\ &= \int_{\mathcal{X}} f(x) \log f(x) \, \mathrm{d}x - \int_{\mathcal{X}} f(x) \log g(x) \, \mathrm{d}x \\ &\overset{\eqref{eq:dent}}{=} - \mathrm{h}[f(x)] - \int_{\mathcal{X}} f(x) \log g(x) \, \mathrm{d}x \; . \end{split}\]

By plugging the probability density function of the continuous uniform distribution into the second term, we obtain:

\[\label{eq:int-fg-s1} \begin{split} \int_{\mathcal{X}} f(x) \log g(x) \, \mathrm{d}x &= \int_{\mathcal{X}} f(x) \log \frac{1}{b-a} \, \mathrm{d}x \\ &= \log \frac{1}{b-a} \int_{\mathcal{X}} f(x) \, \mathrm{d}x \\ &= -\log(b-a) \; . \end{split}\]

This is actually the negative of the differential entropy of the continuous uniform distribution, such that:

\[\label{eq:int-fg-s2} \int_{\mathcal{X}} f(x) \log g(x) \, \mathrm{d}x = -\mathrm{h}[\mathcal{U}(a,b)] = -\mathrm{h}[g(x)] \; .\]

Combining \eqref{eq:kl-fg} with \eqref{eq:int-fg-s2}, we can show that

\[\label{eq:cuni-maxent} \begin{split} 0 &\leq \mathrm{KL}[f||g] \\ 0 &\leq - \mathrm{h}[f(x)] - \left( -\mathrm{h}[g(x)] \right) \\ \mathrm{h}[g(x)] &\geq \mathrm{h}[f(x)] \end{split}\]

which means that the differential entropy of the continuous uniform distribution $\mathcal{U}(a,b)$ will be larger than or equal to any other distribution defined in the same range.

∎
Sources:

Metadata: ID: P412 | shortcut: cuni-maxent | author: JoramSoch | date: 2023-08-25, 11:20.