Index: The Book of Statistical ProofsStatistical Models ▷ Univariate normal data ▷ Multiple linear regression ▷ Contrast-based t-test

Theorem: Consider a linear regression model

$\label{eq:mlr} y = X\beta + \varepsilon, \; \varepsilon \sim \mathcal{N}(0, \sigma^2 V)$

and a t-contrast on the model parameters

$\label{eq:tcon} \gamma = c^\mathrm{T} \beta \quad \text{where} \quad c \in \mathbb{R}^{p \times 1} \; .$

Then, the test statistic

$\label{eq:mlr-t} t = \frac{c^\mathrm{T} \hat{\beta}}{\sqrt{\hat{\sigma}^2 c^\mathrm{T} (X^\mathrm{T} V^{-1} X)^{-1} c}}$

with the parameter estimates

$\label{eq:mlr-est} \begin{split} \hat{\beta} &= (X^\mathrm{T} V^{-1} X)^{-1} X^\mathrm{T} V^{-1} y \\ \hat{\sigma}^2 &= \frac{1}{n-p} (y-X\hat{\beta})^\mathrm{T} V^{-1} (y-X\hat{\beta}) \end{split}$

follows a t-distribution

$\label{eq:mlr-t-dist} t \sim \mathrm{t}(n-p)$

under the null hypothesis

$\label{eq:mlr-t-h0} \begin{split} H_0: &\; c^\mathrm{T} \beta = 0 \\ H_1: &\; c^\mathrm{T} \beta > 0 \; . \end{split}$

Proof:

$\label{eq:b-est-dist} \hat{\beta} \sim \mathcal{N}\left( \beta, \, \sigma^2 (X^\mathrm{T} V^{-1} X)^{-1} \right) \; .$

Thus, the estimated contrast value $\hat{\gamma} = c^\mathrm{T} \hat{\beta}$ is distributed according to a univariate normal distribution:

$\label{eq:g-est-dist} \hat{\gamma} \sim \mathcal{N}\left( c^\mathrm{T} \beta, \, \sigma^2 c^\mathrm{T} (X^\mathrm{T} V^{-1} X)^{-1} c \right) \; .$

Now, define the random variable $z$ by dividing $\hat{\gamma}$ by its standard deviation:

$\label{eq:z} z = \frac{c^\mathrm{T} \hat{\beta}}{\sqrt{\sigma^2 c^\mathrm{T} (X^\mathrm{T} V^{-1} X)^{-1} c}} \; .$

Again applying the linear transformation theorem, this is distributed as

$\label{eq:z-dist} z \sim \mathcal{N}\left( \frac{c^\mathrm{T} \beta}{\sqrt{\sigma^2 c^\mathrm{T} (X^\mathrm{T} V^{-1} X)^{-1} c}}, \, 1 \right)$

and thus follows a standard normal distribution under the null hypothesis:

$\label{eq:z-dist-h0} z \sim \mathcal{N}(0, 1), \quad \text{if} \; H_0 \; .$

2) We also know that the residual sum of squares, divided the true error variance

$\label{eq:mlr-rss} v = \frac{1}{\sigma^2} \sum_{i=1}^{n} \hat{\varepsilon}_i^2 = \frac{\hat{\varepsilon}^\mathrm{T} \hat{\varepsilon}}{\sigma^2} = \frac{1}{\sigma^2} (y-X\hat{\beta})^\mathrm{T} V^{-1} (y-X\hat{\beta})$ $\label{eq:mlr-rss-dist} v \sim \chi^2(n-p) \; .$ $\label{eq:mlr-ind-v1} \hat{\beta} \quad \text{and} \quad \hat{\varepsilon} \quad \text{ind.}$

and thus, the estimated contrast values are also independent from the function of the residual sum of squares

$\label{eq:mlr-ind-v2} z = \frac{c^\mathrm{T} \hat{\beta}}{\sqrt{\sigma^2 c^\mathrm{T} (X^\mathrm{T} V^{-1} X)^{-1} c}} \quad \text{and} \quad v = \frac{\hat{\varepsilon}^\mathrm{T} \hat{\varepsilon}}{\sigma^2} \quad \text{ind.} \; ,$

the following quantity is, by definition, t-distributed

$\label{eq:mlr-t-s1} t = \frac{z}{\sqrt{v/(n-p)}} \sim \mathrm{t}(n-p), \quad \text{if} \; H_0$

and the quantity can be evaluated as:

$\label{eq:mlr-t-s2} \begin{split} t &\overset{\eqref{eq:mlr-t-s1}}{=} \frac{z}{\sqrt{v/(n-p)}} \\ &\overset{\eqref{eq:mlr-ind-v2}}{=} \frac{c^\mathrm{T} \hat{\beta}}{\sqrt{\sigma^2 c^\mathrm{T} (X^\mathrm{T} V^{-1} X)^{-1} c}} \cdot \sqrt{\frac{n-p}{\hat{\varepsilon}^\mathrm{T} \hat{\varepsilon} / \sigma^2}} \\ &= \frac{c^\mathrm{T} \hat{\beta}}{\sqrt{\frac{\hat{\varepsilon}^\mathrm{T} \hat{\varepsilon}}{n-p} \cdot c^\mathrm{T} (X^\mathrm{T} V^{-1} X)^{-1} c}} \\ &\overset{\eqref{eq:mlr-rss}}{=} \frac{c^\mathrm{T} \hat{\beta}}{\sqrt{\frac{(y-X\hat{\beta})^\mathrm{T} V^{-1} (y-X\hat{\beta})}{n-p} \cdot c^\mathrm{T} (X^\mathrm{T} V^{-1} X)^{-1} c}} \\ &\overset{\eqref{eq:mlr-est}}{=} \frac{c^\mathrm{T} \hat{\beta}}{\sqrt{\hat{\sigma}^2 c^\mathrm{T} (X^\mathrm{T} V^{-1} X)^{-1} c}} \; . \end{split}$

This means that the null hypothesis in \eqref{eq:mlr-t-h0} can be rejected when $t$ from \eqref{eq:mlr-t-s2} is as extreme or more extreme than the critical value obtained from Student’s t-distribution with $n-p$ degrees of freedom using a significance level $\alpha$.

Sources:

Metadata: ID: P391 | shortcut: mlr-t | author: JoramSoch | date: 2022-12-13, 10:16.