Index: The Book of Statistical ProofsGeneral Theorems ▷ Information theory ▷ Kullback-Leibler divergence ▷ Definition

Definition: Let $X$ be a random variable with possible outcomes $\mathcal{X}$ and let $P$ and $Q$ be two probability distributions on $X$.

1) The Kullback-Leibler divergence of $P$ from $Q$ for a discrete random variable $X$ is defined as

\[\label{eq:KL-disc} \mathrm{KL}[P||Q] = \sum_{x \in \mathcal{X}} p(x) \cdot \log \frac{p(x)}{q(x)}\]

where $p(x)$ and $q(x)$ are the probability mass functions of $P$ and $Q$.

2) The Kullback-Leibler divergence of $P$ from $Q$ for a continuous random variable $X$ is defined as

\[\label{eq:KL-cont} \mathrm{KL}[P||Q] = \int_{\mathcal{X}} p(x) \cdot \log \frac{p(x)}{q(x)} \, \mathrm{d}x\]

where $p(x)$ and $q(x)$ are the probability density functions of $P$ and $Q$.

 
Sources:

Metadata: ID: D52 | shortcut: kl | author: JoramSoch | date: 2020-05-10, 20:20.