Index: The Book of Statistical ProofsStatistical Models ▷ Univariate normal data ▷ Multiple linear regression ▷ Deviance function

Theorem: Consider a linear regression model $m$ with correlation structure $V$

$\label{eq:mlr} m: \; y = X\beta + \varepsilon, \; \varepsilon \sim \mathcal{N}(0, \sigma^2 V) \; .$

Then, the deviance for this model is

$\label{eq:mlr-dev-v1} D(\beta,\sigma^2) = \mathrm{RSS}/\sigma^2 + n \cdot \left[ \log(\sigma^2) + \log(2\pi) \right]$

under uncorrelated observations, i.e. if $V = I_n$, and

$\label{eq:mlr-dev-v2} D(\beta,\sigma^2) = \mathrm{wRSS}/\sigma^2 + n \cdot \left[ \log(\sigma^2) + \log(2\pi) \right] + \log|V| \; ,$

in the general case, i.e. if $V \neq I_n$, where $\mathrm{RSS}$ is the residual sum of squares and $\mathrm{wRSS}$ is the weighted residual sum of squares.

Proof: The likelihood function for multiple linear regression is given by

$\label{eq:mlr-lf} \begin{split} p(y|\beta,\sigma^2) &= \mathcal{N}(y; X\beta, \sigma^2 V) \\ &= \sqrt{\frac{1}{(2\pi)^n |\sigma^2 V|}} \cdot \exp\left[ -\frac{1}{2} (y - X\beta)^\mathrm{T} (\sigma^2 V)^{-1} (y - X\beta) \right] \; , \end{split}$

such that, with $\lvert \sigma^2 V \rvert = (\sigma^2)^n \lvert V \rvert$, the log-likelihood function for this model becomes

$\label{eq:mlr-llf} \begin{split} \mathrm{LL}(\beta,\sigma^2) &= \log p(y|\beta,\sigma^2) \\ &= - \frac{n}{2} \log(2\pi) - \frac{n}{2} \log (\sigma^2) - \frac{1}{2} \log |V| - \frac{1}{2 \sigma^2} (y - X\beta)^\mathrm{T} V^{-1} (y - X\beta) \; . \end{split}$

The last term can be expressed in terms of the (weighted) residual sum of squares as

$\label{eq:mll-rss} \begin{split} - \frac{1}{2 \sigma^2} (y - X\beta)^\mathrm{T} V^{-1} (y - X\beta) &= - \frac{1}{2 \sigma^2} (Wy-WX\beta)^\mathrm{T} (Wy-WX\beta) \\ &= - \frac{1}{2 \sigma^2} \left( \frac{1}{n} \sum_{i=1}^{n} (W\varepsilon)_i^2 \right) = - \frac{\mathrm{wRSS}}{2 \sigma^2} \end{split}$

where $W = V^{-1/2}$. Plugging \eqref{eq:mll-rss} into \eqref{eq:mlr-llf} and multiplying with $-2$, we obtain the deviance as

$\label{eq:mlr-dev-v2-qed} \begin{split} D(\beta,\sigma^2) &= -2 \, \mathrm{LL}(\beta,\sigma^2) \\ &= -2 \left( - \frac{\mathrm{wRSS}}{2 \sigma^2} - \frac{n}{2} \log (\sigma^2) - \frac{n}{2} \log(2\pi) - \frac{1}{2} \log |V| \right) \\ &= \mathrm{wRSS}/\sigma^2 + n \cdot \left[ \log(\sigma^2) + \log(2\pi) \right] + \log|V| \end{split}$

which proves the result in \eqref{eq:mlr-dev-v2}. Assuming $V = I_n$, we have

$\label{eq:mll-rss-iid} \begin{split} - \frac{1}{2 \sigma^2} (y - X\beta)^\mathrm{T} V^{-1} (y - X\beta) &= - \frac{1}{2 \sigma^2} (y - X\beta)^\mathrm{T} (y - X\beta) \\ &= - \frac{1}{2 \sigma^2} \left( \frac{1}{n} \sum_{i=1}^{n} \varepsilon_i^2 \right) = - \frac{\mathrm{RSS}}{2 \sigma^2} \end{split}$

and

$\label{eq:mlr-logdet-V-iid} \frac{1}{2} \log|V| = \frac{1}{2} \log|I_n| = \frac{1}{2} \log 1 = 0 \; ,$

such that

$\label{eq:mlr-mll-v1-qed} D(\beta,\sigma^2) = \mathrm{RSS}/\sigma^2 + n \cdot \left[ \log(\sigma^2) + \log(2\pi) \right]$

which proves the result in \eqref{eq:mlr-dev-v1}. This completes the proof.

Sources:

Metadata: ID: P312 | shortcut: mlr-dev | author: JoramSoch | date: 2022-03-01, 08:42.