Index: The Book of Statistical ProofsStatistical Models ▷ Univariate normal data ▷ Multiple linear regression ▷ Weighted least squares

Theorem: Given a linear regression model with correlated observations

$\label{eq:MLR} y = X\beta + \varepsilon, \; \varepsilon \sim \mathcal{N}(0, \sigma^2 V) \; ,$

the parameters minimizing the weighted residual sum of squares are given by

$\label{eq:WLS} \hat{\beta} = (X^\mathrm{T} V^{-1} X)^{-1} X^\mathrm{T} V^{-1} y \; .$

Proof: Let there be an $n \times n$ square matrix $W$, such that

$\label{eq:W-def} W V W^\mathrm{T} = I_n \; .$

Since $V$ is a covariance matrix and thus symmetric, $W$ is also symmetric and can be expressed as the matrix square root of the inverse of $V$:

$\label{eq:W-V} W V W = I_n \quad \Leftrightarrow \quad V = W^{-1} W^{-1} \quad \Leftrightarrow \quad V^{-1} = W W \quad \Leftrightarrow \quad W = V^{-1/2} \; .$

Left-multiplying the linear regression equation \eqref{eq:MLR} with $W$, the linear transformation theorem implies that

$\label{eq:MLR-W} Wy = WX\beta + W\varepsilon, \; W\varepsilon \sim \mathcal{N}(0, \sigma^2 W V W^T) \; .$

Applying \eqref{eq:W-def}, we see that \eqref{eq:MLR-W} is actually a linear regression model with independent observations

$\label{eq:MLR-W-dev} \tilde{y} = \tilde{X}\beta + \tilde{\varepsilon}, \; \tilde{\varepsilon} \sim \mathcal{N}(0, \sigma^2 I_n)$

where $\tilde{y} = Wy$, $\tilde{X} = WX$ and $\tilde{\varepsilon} = W\varepsilon$, such that we can apply the ordinary least squares solution giving

$\label{eq:WLS-qed} \begin{split} \hat{\beta} &= (\tilde{X}^\mathrm{T} \tilde{X})^{-1} \tilde{X}^\mathrm{T} \tilde{y} \\ &= \left( (WX)^\mathrm{T} WX \right)^{-1} (WX)^\mathrm{T} Wy \\ &= \left( X^\mathrm{T} W^\mathrm{T} W X \right)^{-1} X^\mathrm{T} W^\mathrm{T} W y \\ &= \left( X^\mathrm{T} W W X \right)^{-1} X^\mathrm{T} W W y \\ &\overset{\eqref{eq:W-V}}{=} \left( X^\mathrm{T} V^{-1} X \right)^{-1} X^\mathrm{T} V^{-1} y \end{split}$

which corresponds to the weighted least squares solution \eqref{eq:WLS}.

Sources:

Metadata: ID: P77 | shortcut: mlr-wls | author: JoramSoch | date: 2020-03-11, 11:22.