Index: The Book of Statistical ProofsGeneral Theorems ▷ Information theory ▷ Kullback-Leibler divergence ▷ Definition

Definition: Let $X$ be a random variable with possible outcomes $\mathcal{X}$ and let $P$ and $Q$ be two probability distributions on $X$.

1) The Kullback-Leibler divergence of $P$ from $Q$ for a discrete random variable $X$ is defined as

$\label{eq:KL-disc} \mathrm{KL}[P||Q] = \sum_{x \in \mathcal{X}} p(x) \cdot \log \frac{p(x)}{q(x)}$

where $p(x)$ and $q(x)$ are the probability mass functions of $P$ and $Q$.

2) The Kullback-Leibler divergence of $P$ from $Q$ for a continuous random variable $X$ is defined as

$\label{eq:KL-cont} \mathrm{KL}[P||Q] = \int_{\mathcal{X}} p(x) \cdot \log \frac{p(x)}{q(x)} \, \mathrm{d}x$

where $p(x)$ and $q(x)$ are the probability density functions of $P$ and $Q$.

By convention, $0 \cdot \log 0$ is taken to be zero when calculating the divergence between $P$ and $Q$.

Sources:

Metadata: ID: D52 | shortcut: kl | author: JoramSoch | date: 2020-05-10, 20:20.