Index: The Book of Statistical Proofs ▷ General Theorems ▷ Probability theory ▷ Covariance ▷ Covariance matrix and expected values

Theorem: Let $X$ be a random vector. Then, the covariance matrix of $X$ is equal to the mean of the outer product of $X$ with itself minus the outer product of the mean of $X$ with itself:

\[\label{eq:covmat-mean} \Sigma_{XX} = \mathrm{E}(X X^\mathrm{T}) - \mathrm{E}(X) \mathrm{E}(X)^\mathrm{T} \; .\]

Proof: The covariance matrix of $X$ is defined as

\[\label{eq:covmat1} \Sigma_{XX} = \begin{bmatrix} \mathrm{E}\left[ (X_1-\mathrm{E}[X_1]) (X_1-\mathrm{E}[X_1]) \right] & \ldots & \mathrm{E}\left[ (X_1-\mathrm{E}[X_1]) (X_n-\mathrm{E}[X_n]) \right] \\ \vdots & \ddots & \vdots \\ \mathrm{E}\left[ (X_n-\mathrm{E}[X_n]) (X_1-\mathrm{E}[X_1]) \right] & \ldots & \mathrm{E}\left[ (X_n-\mathrm{E}[X_n]) (X_n-\mathrm{E}[X_n]) \right] \end{bmatrix}\]

which can also be expressed using matrix multiplication as

\[\label{eq:covmat2} \Sigma_{XX} = \mathrm{E}\left[ (X-\mathrm{E}[X]) (X-\mathrm{E}[X])^\mathrm{T} \right]\]

Due to the linearity of the expected value, this can be rewritten as

\[\label{eq:covmat-mean-qed} \begin{split} \Sigma_{XX} &= \mathrm{E}\left[ (X-\mathrm{E}[X]) (X-\mathrm{E}[X])^\mathrm{T} \right] \\ &= \mathrm{E}\left[ X X^\mathrm{T} - X \, \mathrm{E}(X)^\mathrm{T} - \mathrm{E}(X) \, X^\mathrm{T} + \mathrm{E}(X) \mathrm{E}(X)^\mathrm{T} \right] \\ &= \mathrm{E}(X X^\mathrm{T}) - \mathrm{E}(X) \mathrm{E}(X)^\mathrm{T} - \mathrm{E}(X) \mathrm{E}(X)^\mathrm{T} + \mathrm{E}(X) \mathrm{E}(X)^\mathrm{T} \\ &= \mathrm{E}(X X^\mathrm{T}) - \mathrm{E}(X) \mathrm{E}(X)^\mathrm{T} \; . \end{split}\]
∎
Sources:

Metadata: ID: P120 | shortcut: covmat-mean | author: JoramSoch | date: 2020-06-06, 05:31.