Index: The Book of Statistical Proofs ▷ Statistical Models ▷ Univariate normal data ▷ Multiple linear regression ▷ Symmetry of projection and residual-forming matrix

Theorem: The projection matrix and the residual-forming matrix are symmetric:

\[\label{eq:P-R-symm} \begin{split} P^\mathrm{T} &= P \\ R^\mathrm{T} &= R \; . \end{split}\]

Proof: Let $X$ be the design matrix from the linear regression model. Then, the matrix $X^\mathrm{T} X$ is symmetric, because

\[\label{eq:XTX-symm} (X^\mathrm{T} X)^\mathrm{T} = X^\mathrm{T} {X^\mathrm{T}}^\mathrm{T} = X^\mathrm{T} X \; .\]

Thus, the inverse of $X^\mathrm{T} X$ is also symmetric, i.e.

\[\label{eq:XTX-inv-symm} \left((X^\mathrm{T} X)^{-1}\right)^\mathrm{T} = (X^\mathrm{T} X)^{-1} \; .\]

1) The projection matrix for ordinary least squares is given by

\[\label{eq:P} P = X (X^\mathrm{T} X)^{-1} X^\mathrm{T} \; ,\]

such that

\[\label{eq:P-symm} \begin{split} P^\mathrm{T} &= \left( X (X^\mathrm{T} X)^{-1} X^\mathrm{T} \right)^\mathrm{T} \\ &= {X^\mathrm{T}}^\mathrm{T} \left((X^\mathrm{T} X)^{-1}\right)^\mathrm{T} X^\mathrm{T} \\ &= X (X^\mathrm{T} X)^{-1} X^\mathrm{T} \\ &\overset{\eqref{eq:P}}{=} P \; . \end{split}\]

2) The residual-forming matrix for ordinary least squares is given by

\[\label{eq:R} R = I_n - X (X^\mathrm{T} X)^{-1} X^\mathrm{T} = I_n - P \; ,\]

such that

\[\label{eq:R-symm} \begin{split} R^\mathrm{T} &= (I_n - P)^\mathrm{T} \\ &= I_n^\mathrm{T} - P^\mathrm{T} \\ &\overset{\eqref{eq:P-symm}}{=} I_n - P \\ &\overset{\eqref{eq:R}}{=} R \; . \end{split}\]
∎
Sources:

Metadata: ID: P399 | shortcut: mlr-symm | author: JoramSoch | date: 2022-12-22, 13:17.