Index: The Book of Statistical Proofs ▷ Model Selection ▷ Bayesian model selection ▷ Family evidence ▷ Derivation

Theorem: Let $f$ be a family of $M$ generative models $m_1, \ldots, m_M$ with model evidences $p(y \vert m_1), \ldots, p(y \vert m_M)$. Then, the family evidence can be expressed in terms of the model evidences as

\[\label{eq:FE-marg} \mathrm{FE}(f) = \sum_{i=1}^M p(y|m_i) \, p(m_i|f)\]

where $p(m_i \vert f)$ are the within-family prior model probabilities.

Proof: This a consequence of the law of marginal probability for discrete variables

\[\label{eq:prob-marg} p(y|f) = \sum_{i=1}^M p(y,m_i|f)\]

and the law of conditional probability according to which

\[\label{eq:prob-cond} p(y,m_i|f) = p(y|m_i,f) \, p(m_i|f) \; .\]

Since models are nested within model families, such that $m_i \wedge f \leftrightarrow m_i$, we have the following equality of probabilities:

\[\label{eq:prob-equal} p(y|m_i,f) = p(y|m_i \wedge f) = p(y|m_i) \; .\]

Plugging \eqref{eq:prob-cond} into \eqref{eq:prob-marg} and applying \eqref{eq:prob-equal}, we obtain:

\[\label{eq:ME-marg-qed} \mathrm{FE}(f) = p(y|f) = \sum_{i=1}^M p(y|m_i) \, p(m_i|f) \; .\]
∎
Sources:

Metadata: ID: P368 | shortcut: fe-der | author: JoramSoch | date: 2022-10-20, 10:47.