Partition of the mean squared error into bias and variance

Index: The Book of Statistical Proofs ▷ General Theorems ▷ Estimation theory ▷ Point estimates ▷ Partition of the mean squared error into bias and variance

Theorem: The mean squared error can be partitioned into variance and squared bias

\[\label{eq:MSE} \mathrm{MSE}(\hat{\theta}) = \mathrm{Var}(\hat{\theta}) + \mathrm{Bias}(\hat{\theta},\theta)^2\]

where the variance is given by

\[\label{eq:Var} \mathrm{Var}(\hat{\theta}) = \mathrm{E}_{\hat{\theta}}\left[ \left( \hat{\theta} - \mathrm{E}_{\hat{\theta}}(\hat{\theta}) \right)^2 \right]\]

and the bias is given by

\[\label{eq:Bias} \mathrm{Bias}(\hat{\theta},\theta) = \left( \mathrm{E}_{\hat{\theta}}(\hat{\theta}) - \theta \right) \; .\]

Proof: The mean squared error (MSE) is defined as the expected value of the squared deviation of the estimated value $\hat{\theta}$ from the true value $\theta$ of a parameter, over all values $\hat{\theta}$:

\[\label{eq:MSE-def} \mathrm{MSE}(\hat{\theta}) = \mathrm{E}_{\hat{\theta}}\left[ \left( \hat{\theta} - \theta \right)^2 \right] \; .\]

This formula can be evaluated in the following way:

\[\label{eq:MSE-ref1} \begin{split} \mathrm{MSE}(\hat{\theta}) &= \mathrm{E}_{\hat{\theta}}\left[ \left( \hat{\theta} - \theta \right)^2 \right] \\ &= \mathrm{E}_{\hat{\theta}}\left[ \left( \hat{\theta} - \mathrm{E}_{\hat{\theta}}(\hat{\theta}) + \mathrm{E}_{\hat{\theta}}(\hat{\theta}) - \theta \right)^2 \right] \\ &= \mathrm{E}_{\hat{\theta}}\left[ \left( \hat{\theta} - \mathrm{E}_{\hat{\theta}}(\hat{\theta}) \right)^2 + 2 \left( \hat{\theta} - \mathrm{E}_{\hat{\theta}}(\hat{\theta}) \right) \left( \mathrm{E}_{\hat{\theta}}(\hat{\theta}) - \theta \right) + \left( \mathrm{E}_{\hat{\theta}}(\hat{\theta}) - \theta \right)^2 \right] \\ &= \mathrm{E}_{\hat{\theta}}\left[ \left( \hat{\theta} - \mathrm{E}_{\hat{\theta}}(\hat{\theta}) \right)^2 \right] + \mathrm{E}_{\hat{\theta}}\left[ 2 \left( \hat{\theta} - \mathrm{E}_{\hat{\theta}}(\hat{\theta}) \right) \left( \mathrm{E}_{\hat{\theta}}(\hat{\theta}) - \theta \right) \right] + \mathrm{E}_{\hat{\theta}}\left[ \left( \mathrm{E}_{\hat{\theta}}(\hat{\theta}) - \theta \right)^2 \right] \; . \\ \end{split}\]

Because $\mathrm{E}_{\hat{\theta}}(\hat{\theta}) - \theta$ is constant as a function of $\hat{\theta}$, we have:

\[\label{eq:MSE-ref2} \begin{split} \mathrm{MSE}(\hat{\theta}) &= \mathrm{E}_{\hat{\theta}}\left[ \left( \hat{\theta} - \mathrm{E}_{\hat{\theta}}(\hat{\theta}) \right)^2 \right] + 2 \left( \mathrm{E}_{\hat{\theta}}(\hat{\theta}) - \theta \right) \mathrm{E}_{\hat{\theta}}\left[ \hat{\theta} - \mathrm{E}_{\hat{\theta}}(\hat{\theta}) \right] + \left( \mathrm{E}_{\hat{\theta}}(\hat{\theta}) - \theta \right)^2 \\ &= \mathrm{E}_{\hat{\theta}}\left[ \left( \hat{\theta} - \mathrm{E}_{\hat{\theta}}(\hat{\theta}) \right)^2 \right] + 2 \left( \mathrm{E}_{\hat{\theta}}(\hat{\theta}) - \theta \right) \left( \mathrm{E}_{\hat{\theta}}(\hat{\theta}) - \mathrm{E}_{\hat{\theta}}(\hat{\theta}) \right) + \left( \mathrm{E}_{\hat{\theta}}(\hat{\theta}) - \theta \right)^2 \\ &= \mathrm{E}_{\hat{\theta}}\left[ \left( \hat{\theta} - \mathrm{E}_{\hat{\theta}}(\hat{\theta}) \right)^2 \right] + \left( \mathrm{E}_{\hat{\theta}}(\hat{\theta}) - \theta \right)^2 \; . \\ \end{split}\]

This proofs the partition given by \eqref{eq:MSE}.

∎

Sources:

Wikipedia (2019): "Mean squared error"; in: Wikipedia, the free encyclopedia, retrieved on 2019-11-27; URL: https://en.wikipedia.org/wiki/Mean_squared_error#Proof_of_variance_and_bias_relationship.

Metadata: ID: P5 | shortcut: mse-bnv | author: JoramSoch | date: 2019-11-27, 14:26.