Corrected Akaike information criterion for multiple linear regression

Index: The Book of Statistical Proofs ▷ Statistical Models ▷ Univariate normal data ▷ Multiple linear regression ▷ Corrected Akaike information criterion

Theorem: Consider a linear regression model $m$

\[\label{eq:mlr} m: \; y = X\beta + \varepsilon, \; \varepsilon \sim \mathcal{N}(0, \sigma^2 V) \; .\]

Then, the corrected Akaike information criterion for this model is

\[\label{eq:mlr-aicc} \mathrm{AIC}_\mathrm{c}(m) = n \log\left( \frac{\mathrm{wRSS}}{n} \right) + n \left[ 1 + \log(2\pi) \right] + \log|V| + \frac{2 \, n \, (p+1)}{n-p-2}\]

where $\mathrm{wRSS}$ is the weighted residual sum of squares, $p$ is the number of regressors in the design matrix $X$ and $n$ is the number of observations in the data vector $y$.

Proof: The corrected Akaike information criterion is defined as

\[\label{eq:aicc} \mathrm{AIC}_\mathrm{c}(m) = \mathrm{AIC}(m) + \frac{2k^2 + 2k}{n-k-1}\]

where $\mathrm{AIC}(m)$ is the Akaike information criterion, $k$ is the number of free parameters in $m$ and $n$ is the number of observations.

The Akaike information criterion for multiple linear regression is given by

\[\label{eq:mlr-aic} \mathrm{AIC}(m) = n \log\left( \frac{\mathrm{wRSS}}{n} \right) + n \left[ 1 + \log(2\pi) \right] + \log|V| + 2 (p + 1)\]

and the number of free paramters in multiple linear regression is $k = p + 1$, i.e. one for each regressor in the design matrix $X$, plus one for the noise variance $\sigma^2$.

Thus, the corrected AIC of $m$ follows from \eqref{eq:aicc} and \eqref{eq:mlr-aic} as

\[\label{eq:mlr-aicc-qed} \begin{split} \mathrm{AIC}_\mathrm{c}(m) &= n \log\left( \frac{\mathrm{wRSS}}{n} \right) + n \left[ 1 + \log(2\pi) \right] + \log|V| + 2 \, k + \frac{2k^2 + 2k}{n-k-1} \\ &= n \log\left( \frac{\mathrm{wRSS}}{n} \right) + n \left[ 1 + \log(2\pi) \right] + \log|V| + \frac{2nk - 2k^2 - 2k}{n-k-1} + \frac{2k^2 + 2k}{n-k-1} \\ &= n \log\left( \frac{\mathrm{wRSS}}{n} \right) + n \left[ 1 + \log(2\pi) \right] + \log|V| + \frac{2nk}{n-k-1} \\ &= n \log\left( \frac{\mathrm{wRSS}}{n} \right) + n \left[ 1 + \log(2\pi) \right] + \log|V| + \frac{2 \, n \, (p+1)}{n-p-2} \\ \; . \end{split}\]

∎

Sources:

Claeskens G, Hjort NL (2008): "Akaike's information criterion"; in: Model Selection and Model Averaging, ex. 2.5, p. 67; URL: https://www.cambridge.org/core/books/model-selection-and-model-averaging/E6F1EC77279D1223423BB64FC3A12C37; DOI: 10.1017/CBO9780511790485.

Metadata: ID: P309 | shortcut: mlr-aicc | author: JoramSoch | date: 2022-02-11, 07:07.