Index: The Book of Statistical Proofs ▷ General Theorems ▷ Information theory ▷ Shannon entropy ▷ Non-negativity

Theorem: The entropy of a discrete random variable is a non-negative number:

\[\label{eq:ent-nonneg} \mathrm{H}(X) \geq 0 \; .\]

Proof: The entropy of a discrete random variable is defined as

\[\label{eq:ent} \mathrm{H}(X) = - \sum_{x \in \mathcal{X}} p(x) \cdot \log_b p(x)\]

The minus sign can be moved into the sum:

\[\label{eq:ent-dev} \mathrm{H}(X) = \sum_{x \in \mathcal{X}} \left[ p(x) \cdot \left( - \log_b p(x) \right) \right]\]

Because the co-domain of probability mass functions is $[0,1]$, we can deduce:

\[\label{eq:nonneg} \begin{array}{rcccl} 0 &\leq &p(x) &\leq &1 \\ -\infty &\leq &\log_b p(x) &\leq &0 \\ 0 &\leq &-\log_b p(x) &\leq &+\infty \\ 0 &\leq &p(x) \cdot \left(-\log_b p(x)\right) &\leq &+\infty \; . \end{array}\]

By convention, $0 \cdot \log_b(0)$ is taken to be $0$ when calculating entropy, consistent with

\[\label{eq:lim-0log0} \lim_{p \to 0} \left[ p \log_b(p) \right] = 0 \; .\]

Taking this together, each addend in \eqref{eq:ent-dev} is positive or zero and thus, the entire sum must also be non-negative.

∎
Sources:

Metadata: ID: P57 | shortcut: ent-nonneg | author: JoramSoch | date: 2020-02-19, 19:10.