Index: The Book of Statistical ProofsProbability Distributions ▷ Multivariate continuous distributions ▷ Multivariate normal distribution ▷ Covariance

Theorem: Let $x$ follow a multivariate normal distribution:

$\label{eq:mvn} x \sim \mathcal{N}(\mu, \Sigma) \; .$

Then, the covariance matrix of $x$ is

$\label{eq:mvn-cov} \mathrm{Cov}(x) = \Sigma \; .$

Proof:

1) First, consider a set of independent and standard normally distributed random variables:

$\label{eq:zi} z_i \overset{\mathrm{i.i.d.}}{\sim} \mathcal{N}(0,1), \quad i = 1,\ldots,n \; .$

Then, these variables together form a multivariate normally distributed random vector:

$\label{eq:z} z \sim \mathcal{N}(0_n, I_n) \; .$

Because the covariance is zero for independent random variables, we have

$\label{eq:zij-cov} \mathrm{Cov}(z_i,z_j) = 0 \quad \text{for all} \quad i \neq j \; .$

Moreover, as the variance of all entries of the vector is one, we have

$\label{eq:zi-var} \mathrm{Var}(z_i) = 1 \quad \text{for all} \quad i = 1, \ldots, n \; .$

Taking \eqref{eq:zij-cov} and \eqref{eq:zi-var} together, the covariance matrix of $z$ is

$\label{eq:z-cov} \mathrm{Cov}(z) = \left[ \begin{array}{ccc} 1 & \cdots & 0 \\ \vdots & \ddots & \vdots \\ 0 & \cdots & 1 \end{array} \right] = I_n \; .$

2) Next, consider an $n \times n$ matrix $A$ solving the equation $A A^\mathrm{T} = \Sigma$. Such a matrix exists, because $\Sigma$ is defined to be positive definite. Then, $x$ can be represented as a linear transformation of $z$:

$\label{eq:x-z} x = Az + \mu \sim \mathcal{N}(A 0_n + \mu, A I_n A^\mathrm{T}) = \mathcal{N}(\mu, \Sigma) \; .$

Thus, the covariance of $x$ can be written as:

$\label{eq:x-mean} \mathrm{Cov}(x) = \mathrm{Cov}( Az + \mu ) \; .$ $\label{eq:cov-inv} \mathrm{Cov}(x + a) = \mathrm{Cov}(x)$ $\label{eq:cov-scal} \mathrm{Cov}(Ax) = A \mathrm{Cov}(x) A^\mathrm{T} \; ,$

this becomes:

$\label{eq:mvn-cov-qed} \begin{split} \mathrm{Cov}(x) &= \mathrm{Cov}( Az + \mu ) \\ &\overset{\eqref{eq:cov-inv}}{=} \mathrm{Cov}(Az) \\ &\overset{\eqref{eq:cov-scal}}{=} A \, \mathrm{Cov}(z) A^\mathrm{T} \\ &\overset{\eqref{eq:z-cov}}{=} A I_n A^\mathrm{T} \\ &= A A^\mathrm{T} \\ &= \Sigma \; . \end{split}$
Sources:

Metadata: ID: P340 | shortcut: mvn-cov | author: JoramSoch | date: 2022-09-15, 08:41.