Index: The Book of Statistical Proofs ▷ Statistical Models ▷ Univariate normal data ▷ Multiple linear regression ▷ Estimation, projection and residual-forming matrix

Theorem: Assume a linear regression model with independent observations

\[\label{eq:mlr} y = X\beta + \varepsilon, \; \varepsilon_i \overset{\mathrm{i.i.d.}}{\sim} \mathcal{N}(0, \sigma^2)\]

and consider estimation using ordinary least squares. Then, the estimated parameters, fitted signal and residuals are given by

\[\label{eq:mlr-est} \begin{split} \hat{\beta} &= E y \\ \hat{y} &= P y \\ \hat{\varepsilon} &= R y \end{split}\]

where

\[\label{eq:mlr-mat} \begin{split} E &= (X^\mathrm{T} X)^{-1} X^\mathrm{T} \\ P &= X (X^\mathrm{T} X)^{-1} X^\mathrm{T} \\ R &= I_n - X (X^\mathrm{T} X)^{-1} X^\mathrm{T} \end{split}\]

are the estimation matrix, projection matrix and residual-forming matrix and $n$ is the number of observations.

Proof:

1) Ordinary least squares parameter estimates of $\beta$ are defined as minimizing the residual sum of squares

\[\label{eq:ols} \hat{\beta} = \operatorname*{arg\,min}_{\beta} \left[ (y-X\beta)^\mathrm{T} (y-X\beta) \right]\]

and the solution to this is given by

\[\label{eq:b-est-qed} \begin{split} \hat{\beta} &= (X^\mathrm{T} X)^{-1} X^\mathrm{T} y \\ &\overset{\eqref{eq:mlr-mat}}{=} E y \; . \end{split}\]


2) The fitted signal is given by multiplying the design matrix with the estimated regression coefficients

\[\label{eq:y-est} \hat{y} = X\hat{\beta}\]

and using \eqref{eq:b-est-qed}, this becomes

\[\label{eq:y-est-qed} \begin{split} \hat{y} &= X (X^\mathrm{T} X)^{-1} X^\mathrm{T} y \\ &\overset{\eqref{eq:mlr-mat}}{=} P y \; . \end{split}\]


3) The residuals of the model are calculated by subtracting the fitted signal from the measured signal

\[\label{eq:e-est} \hat{\varepsilon} = y - \hat{y}\]

and using \eqref{eq:y-est-qed}, this becomes

\[\label{eq:e-est-qed} \begin{split} \hat{\varepsilon} &= y - X (X^\mathrm{T} X)^{-1} X^\mathrm{T} y \\ &= (I_n - X (X^\mathrm{T} X)^{-1} X^\mathrm{T}) y \\ &\overset{\eqref{eq:mlr-mat}}{=} R y \; . \end{split}\]
∎
Sources:

Metadata: ID: P75 | shortcut: mlr-mat | author: JoramSoch | date: 2020-03-09, 21:18.