Index: The Book of Statistical Proofs ▷ General Theorems ▷ Probability theory ▷ Expected value ▷ Expected value minimizes squared error

Theorem: Let $X_1, \ldots, X_n$ be a collection of random variables with common mean $\mathrm{E}(X_i) = \mu$, $i = 1,\ldots,n$. Then, $\mu$ minimizes the mean squared error:

\[\label{eq:mean-mse} \mu = \operatorname*{arg\,min}_{a \in \mathbb{R}} \mathrm{E}\left[ (X_i - a)^2 \right] \; .\]

Proof: Using the linearity of expectation, we can simplify the objective function:

\[\label{eq:mse} \mathrm{E}\left[ (X_i - a)^2 \right] = \mathrm{E}\left[ X_i^2 - 2aX_i + a^2 \right] = a^2 - 2a\mu + \mathrm{E}(X_i^2) \; .\]

Setting the first derivative

\[\label{eq:dmse-da} \frac{\mathrm{d}}{\mathrm{d}a} \left[ a^2 - 2a\mu + \mathrm{E}(X_i^2) \right] = 2a - 2\mu\]

to zero to perform a derivative test, we obtain:

\[\label{eq:mean-mse-qed} 2a - 2\mu = 0 \quad \Leftrightarrow \quad a = \mu \; .\]

The second derivative is equal to 2, which is greater than 0. Since $a = \mu$ is the sole critical point, we can conclude that this value is the unique global minimum. This completes the proof that the expected value minimizes the mean squared error.

∎
Sources:

Metadata: ID: P469 | shortcut: mean-mse | author: salbalkus | date: 2024-09-13, 23:30.