Index: The Book of Statistical Proofs ▷ Probability Distributions ▷ Univariate continuous distributions ▷ Normal distribution ▷ Differential entropy

Theorem: Let $X$ be a random variable following a normal distribution:

\[\label{eq:norm} X \sim \mathcal{N}(\mu, \sigma^2) \; .\]

Then, the differential entropy of $X$ is

\[\label{eq:norm-dent} \mathrm{h}(X) = \frac{1}{2} \ln\left( 2 \pi \sigma^2 e \right) \; .\]

Proof: The differential entropy of a random variable is defined as

\[\label{eq:dent} \mathrm{h}(X) = - \int_{\mathcal{X}} p(x) \, \log_b p(x) \, \mathrm{d}x \; .\]

To measure $h(X)$ in nats, we set $b = e$, such that

\[\label{eq:dent-nats} \mathrm{h}(X) = - \mathrm{E}\left[ \ln p(x) \right] \; .\]

With the probability density function of the normal distribution, the differential entropy of $X$ is:

\[\label{eq:norm-dent-s1} \begin{split} \mathrm{h}(X) &= - \mathrm{E}\left[ \ln \left( \frac{1}{\sqrt{2 \pi} \sigma} \cdot \exp \left[ -\frac{1}{2} \left( \frac{x-\mu}{\sigma} \right)^2 \right] \right) \right] \\ &= - \mathrm{E}\left[ - \frac{1}{2} \ln(2\pi\sigma^2) - \frac{1}{2} \left( \frac{x-\mu}{\sigma} \right)^2 \right] \\ &= \frac{1}{2} \ln(2 \pi \sigma^2) + \frac{1}{2} \, \mathrm{E}\left[ \left( \frac{x-\mu}{\sigma} \right)^2 \right] \\ &= \frac{1}{2} \ln(2 \pi \sigma^2) + \frac{1}{2} \cdot \frac{1}{\sigma^2} \cdot \mathrm{E}\left[ (x-\mu)^2 \right] \; . \end{split}\]

Note that $\mathrm{E}\left[ (x-\mu)^2 \right]$ corresponds to the variance of $X$ and the variance of the normal distribution is $\sigma^2$. Thus, we can proceed:

\[\label{eq:norm-dent-s2} \begin{split} \mathrm{h}(X) &= \frac{1}{2} \ln(2 \pi \sigma^2) + \frac{1}{2} \cdot \frac{1}{\sigma^2} \cdot \sigma^2 \\ &= \frac{1}{2} \ln(2 \pi \sigma^2) + \frac{1}{2} \\ &= \frac{1}{2} \ln(2 \pi \sigma^2 e) \; . \end{split}\]
∎
Sources:

Metadata: ID: P101 | shortcut: norm-dent | author: JoramSoch | date: 2020-05-14, 20:09.