Corrected Akaike information criterion for multiple linear regression

Index: The Book of Statistical Proofs ▷ Statistical Models ▷ Univariate normal data ▷ Multiple linear regression ▷ Corrected Akaike information criterion

Theorem: Consider a linear regression model $m$

\begin{matrix} (1) & m : y = X β + ε, ε \sim N (0, σ^{2} V) . \end{matrix}

$\label{eq:mlr} m: \; y = X\beta + \varepsilon, \; \varepsilon \sim \mathcal{N}(0, \sigma^2 V) \; .$

Then, the corrected Akaike information criterion for this model is

\begin{matrix} (2) & {A I C}_{c} (m) = n \log (\frac{w R S S}{n}) + n [1 + \log (2 π)] + \log | V | + \frac{2 n (p + 1)}{n - p - 2} \end{matrix}

$\label{eq:mlr-aicc} \mathrm{AIC}_\mathrm{c}(m) = n \log\left( \frac{\mathrm{wRSS}}{n} \right) + n \left[ 1 + \log(2\pi) \right] + \log|V| + \frac{2 \, n \, (p+1)}{n-p-2}$

where $\mathrm{wRSS}$ is the weighted residual sum of squares, $p$ is the number of regressors in the design matrix $X$ and $n$ is the number of observations in the data vector $y$ .

Proof: The corrected Akaike information criterion is defined as

\begin{matrix} (3) & {A I C}_{c} (m) = A I C (m) + \frac{2 k^{2} + 2 k}{n - k - 1} \end{matrix}

$\label{eq:aicc} \mathrm{AIC}_\mathrm{c}(m) = \mathrm{AIC}(m) + \frac{2k^2 + 2k}{n-k-1}$

where $\mathrm{AIC}(m)$ is the Akaike information criterion, $k$ is the number of free parameters in $m$ and $n$ is the number of observations.

The Akaike information criterion for multiple linear regression is given by

\begin{matrix} (4) & A I C (m) = n \log (\frac{w R S S}{n}) + n [1 + \log (2 π)] + \log | V | + 2 (p + 1) \end{matrix}

$\label{eq:mlr-aic} \mathrm{AIC}(m) = n \log\left( \frac{\mathrm{wRSS}}{n} \right) + n \left[ 1 + \log(2\pi) \right] + \log|V| + 2 (p + 1)$

and the number of free paramters in multiple linear regression is $k = p + 1$ , i.e. one for each regressor in the design matrix $X$ , plus one for the noise variance $\sigma^2$ .

Thus, the corrected AIC of $m$ follows from $\eqref{eq:aicc}$ and $\eqref{eq:mlr-aic}$ as

\begin{matrix} (5) & \begin{aligned} {A I C}_{c} (m) & = n \log (\frac{w R S S}{n}) + n [1 + \log (2 π)] + \log | V | + 2 k + \frac{2 k^{2} + 2 k}{n - k - 1} \\ = n \log (\frac{w R S S}{n}) + n [1 + \log (2 π)] + \log | V | + \frac{2 n k - 2 k^{2} - 2 k}{n - k - 1} + \frac{2 k^{2} + 2 k}{n - k - 1} \\ = n \log (\frac{w R S S}{n}) + n [1 + \log (2 π)] + \log | V | + \frac{2 n k}{n - k - 1} \\ = n \log (\frac{w R S S}{n}) + n [1 + \log (2 π)] + \log | V | + \frac{2 n (p + 1)}{n - p - 2} \\ . \end{aligned} \end{matrix}

$\label{eq:mlr-aicc-qed} \begin{split} \mathrm{AIC}_\mathrm{c}(m) &= n \log\left( \frac{\mathrm{wRSS}}{n} \right) + n \left[ 1 + \log(2\pi) \right] + \log|V| + 2 \, k + \frac{2k^2 + 2k}{n-k-1} \\ &= n \log\left( \frac{\mathrm{wRSS}}{n} \right) + n \left[ 1 + \log(2\pi) \right] + \log|V| + \frac{2nk - 2k^2 - 2k}{n-k-1} + \frac{2k^2 + 2k}{n-k-1} \\ &= n \log\left( \frac{\mathrm{wRSS}}{n} \right) + n \left[ 1 + \log(2\pi) \right] + \log|V| + \frac{2nk}{n-k-1} \\ &= n \log\left( \frac{\mathrm{wRSS}}{n} \right) + n \left[ 1 + \log(2\pi) \right] + \log|V| + \frac{2 \, n \, (p+1)}{n-p-2} \\ \; . \end{split}$

∎

Sources:

Claeskens G, Hjort NL (2008): "Akaike's information criterion"; in: Model Selection and Model Averaging, ex. 2.5, p. 67; URL: https://www.cambridge.org/core/books/model-selection-and-model-averaging/E6F1EC77279D1223423BB64FC3A12C37; DOI: 10.1017/CBO9780511790485.

Metadata: ID: P309 | shortcut: mlr-aicc | author: JoramSoch | date: 2022-02-11, 07:07.