Definition: Maximum entropy prior distribution

Index: The Book of Statistical Proofs ▷ General Theorems ▷ Bayesian statistics ▷ Prior distributions ▷ Maximum entropy priors

Definition: Let $m$ be a generative model with likelihood function $p(y \vert \theta, m)$ and prior distribution $p(\theta \vert \lambda, m)$ using prior hyperparameters $\lambda$. Then, the prior distribution is called a “maximum entropy prior”, if

1) when $\theta$ is a discrete random variable, it maximizes the entropy of the prior probability mass function:

\[\label{eq:prior-maxent-disc} \lambda_{\mathrm{maxent}} = \operatorname*{arg\,max}_{\lambda} \mathrm{H}\left[ p(\theta \vert \lambda, m) \right] \; ;\]

2) when $\theta$ is a continuous random variable, it maximizes the differential entropy of the prior probability density function:

\[\label{eq:prior-maxent-cont} \lambda_{\mathrm{maxent}} = \operatorname*{arg\,max}_{\lambda} \mathrm{h}\left[ p(\theta \vert \lambda, m) \right] \; .\]

 

Sources:

• Wikipedia (2020): "Prior probability"; in: Wikipedia, the free encyclopedia, retrieved on 2020-12-02; URL: https://en.wikipedia.org/wiki/Prior_probability#Uninformative_priors.

Metadata: ID: D121 | shortcut: prior-maxent | author: JoramSoch | date: 2020-12-02, 18:13.