Index: The Book of Statistical Proofs ▷ Statistical Models ▷ Univariate normal data ▷ Multiple linear regression ▷ Idempotence of projection and residual-forming matrix

Theorem: The projection matrix and the residual-forming matrix are idempotent:

\[\label{eq:P^2-R^2} \begin{split} P^2 &= P \\ R^2 &= R \; . \end{split}\]

Proof:

1) The projection matrix for ordinary least squares is given by

\[\label{eq:P} P = X (X^\mathrm{T} X)^{-1} X^\mathrm{T} \; ,\]

such that

\[\label{eq:P^2} \begin{split} P^2 &= X (X^\mathrm{T} X)^{-1} X^\mathrm{T} X (X^\mathrm{T} X)^{-1} X^\mathrm{T} \\ &= X (X^\mathrm{T} X)^{-1} X^\mathrm{T} \\ &\overset{\eqref{eq:P}}{=} P \; . \end{split}\]


2) The residual-forming matrix for ordinary least squares is given by

\[\label{eq:R} R = I_n - X (X^\mathrm{T} X)^{-1} X^\mathrm{T} = I_n - P \; ,\]

such that

\[\label{eq:R^2} \begin{split} R^2 &= (I_n - P) (I_n - P) \\ &= I_n - P - P + P^2 \\ &\overset{\eqref{eq:P^2}}{=} I_n - 2 P + P \\ &= I_n - P \\ &\overset{\eqref{eq:R}}{=} R \; . \end{split}\]
∎
Sources:

Metadata: ID: P135 | shortcut: mlr-idem | author: JoramSoch | date: 2020-07-22, 06:28.