Index: The Book of Statistical Proofs ▷ Probability Distributions ▷ Multivariate continuous distributions ▷ Multivariate normal distribution ▷ Covariance

Theorem: Let $X$ follow a multivariate normal distribution:

\[\label{eq:mvn} X \sim \mathcal{N}(\mu, \Sigma) \; .\]

Then, the covariance matrix of $X$ is

\[\label{eq:mvn-cov} \mathrm{Cov}(X) = \Sigma \; .\]

Proof:

1) First, consider a set of independent and standard normally distributed random variables:

\[\label{eq:Zi} Z_i \overset{\mathrm{i.i.d.}}{\sim} \mathcal{N}(0,1), \quad i = 1,\ldots,n \; .\]

Then, these variables together form a multivariate normally distributed random vector:

\[\label{eq:Z} Z \sim \mathcal{N}(0_n, I_n) \; .\]

Because the covariance is zero for independent random variables, we have

\[\label{eq:Zij-cov} \mathrm{Cov}(Z_i,Z_j) = 0 \quad \text{for all} \quad i \neq j \; .\]

Moreover, as the variance of all entries of the vector is one, we have

\[\label{eq:Zi-var} \mathrm{Var}(Z_i) = 1 \quad \text{for all} \quad i = 1, \ldots, n \; .\]

Taking \eqref{eq:Zij-cov} and \eqref{eq:Zi-var} together, the covariance matrix of $Z$ is

\[\label{eq:Z-cov} \mathrm{Cov}(Z) = \left[ \begin{array}{ccc} 1 & \cdots & 0 \\ \vdots & \ddots & \vdots \\ 0 & \cdots & 1 \end{array} \right] = I_n \; .\]

2) Next, consider an $n \times n$ matrix $A$ solving the equation $A A^\mathrm{T} = \Sigma$. Such a matrix exists, because $\Sigma$ is defined to be positive definite. Then, $X$ can be represented as a linear transformation of $Z$:

\[\label{eq:X-Z} X = AZ + \mu \sim \mathcal{N}(A 0_n + \mu, A I_n A^\mathrm{T}) = \mathcal{N}(\mu, \Sigma) \; .\]

Thus, the covariance of $X$ can be written as:

\[\label{eq:X-mean} \mathrm{Cov}(X) = \mathrm{Cov}(AZ + \mu) \; .\]

With the invariance of the covariance matrix under addition

\[\label{eq:cov-inv} \mathrm{Cov}(X + a) = \mathrm{Cov}(X)\]

and the scaling of the covariance matrix upon multiplication

\[\label{eq:cov-scal} \mathrm{Cov}(AX) = A \mathrm{Cov}(X) A^\mathrm{T} \; ,\]

this becomes:

\[\label{eq:mvn-cov-qed} \begin{split} \mathrm{Cov}(X) &= \mathrm{Cov}(AZ + \mu) \\ &\overset{\eqref{eq:cov-inv}}{=} \mathrm{Cov}(AZ) \\ &\overset{\eqref{eq:cov-scal}}{=} A \, \mathrm{Cov}(Z) A^\mathrm{T} \\ &\overset{\eqref{eq:Z-cov}}{=} A I_n A^\mathrm{T} \\ &= A A^\mathrm{T} \\ &= \Sigma \; . \end{split}\]
∎
Sources:

Metadata: ID: P340 | shortcut: mvn-cov | author: JoramSoch | date: 2022-09-15, 08:41.