MIT 18.650 — Lecture 22

Bayesian Inference II

§ Computing the Bayes Estimator and MAP

Question #1. Determine how one might compute the Bayes estimator θ^Bayes:=∫θ⋅f(θ∣X1,…,Xn) dθ\hat{\theta}^{\text{Bayes}} := \int \theta \cdot f(\theta \mid X_1, \dots, X_n) \, \mathrm{d}\theta.

Here's the first idea: instead of computing an integral, just sample NN points according to the distribution f(θ∣X1,…,Xn)f(\theta \mid X_1, \dots, X_n) and take their average. But there's a problem—we know what f(θ∣X1,…,Xn)f(\theta \mid X_1, \dots, X_n) is proportional to, but we know nothing about the exact probabilities. How do we actually sample from this distribution?

Remark. To see why sampling from such a distribution might be hard, consider the following challenge.

Challenge. Randomly shuffle a deck of 5252 cards so that the probability of any given ordering is proportional to ∑n=152an⋅πn\sum_{n = 1}^{52} a_n \cdot \pi^n, where the nthn^{\text{th}} card has rank ana_n.

It's not computationally feasible to sum up all 52!52! weights.

It turns out that there's a universal solution to sampling from a distribution that we only know up to a proportion. The solution is Markov Chain Monte Carlo (MCMC). We won't explain it here; for further details, see Lecture 21 of 6.1220.

Question #2. Determine how one might compute the maximum a posteriori θ^MAP:=arg max⁡θf(θ∣X1,…,Xn)\hat{\theta}^{\text{MAP}} := \argmax_{\theta} f(\theta \mid X_1, \dots, X_n).

The simplifying trick is to maximize the log of the posterior instead, and do so via gradient ascent. This looks like:

θ(k+1)=θ(k)+ηk⋅[∇ℓn(θ(k))+∇log⁡f(θ(k))].\theta^{(k + 1)} = \theta^{(k)} + \eta_k \cdot \left[ \nabla \ell_n(\theta^{(k)}) + \nabla \log f(\theta^{(k)}) \right].

Question #3. Determine how one might construct a posterior interval.

Well, it's kinda obvious: a (1−α)(1 - \alpha) posterior interval is any interval over which the integral of the posterior is 1−α1 - \alpha.

§ Gaussian Conjugate Priors

Example. Suppose our data is sampled i.i.d. as X1,…,Xn∼N(θ,1)X_1, \dots, X_n \sim \mathcal{N}(\theta, 1), and our prior is θ∼N(0,σ2)\theta \sim \mathcal{N}(0, \sigma^2). Compute the posterior.

Solution: Plug it into the posterior formula.

f(θ∣X1,…,Xn)∝exp⁡(−12⋅∑i=1n(Xi−θ)2)⋅exp⁡(−12⋅θ2σ2)=exp⁡(−12[[n+1σ2]θ2−2θ∑i=1nXi])⋅exp⁡(−12⋅∑i=1nXi2)∝exp⁡(−[n+1σ2]2(θ−∑i=1nXi[n+1σ2])2)⋅[constant, non-functions of θ]\begin{align*}f(\theta \mid X_1, \dots, X_n) & \propto \exp\left( -\frac{1}{2} \cdot \sum_{i = 1}^n (X_i - \theta)^2 \right) \cdot \exp\left( -\frac{1}{2} \cdot \frac{\theta^2}{\sigma^2} \right) \\ & = \exp\left(-\frac{1}{2} \left[ \left[n + \frac{1}{\sigma^2}\right]\theta^2 - 2\theta \sum_{i = 1}^n X_i \right]\right) \cdot \textcolor{#008080}{ \exp\left(-\frac{1}{2} \cdot \sum_{i = 1}^n X_i^2\right) } \\ & \propto \exp\left( -\frac{\left[n + \frac{1}{\sigma^2}\right]}{2} \left( \theta - \frac{\sum_{i = 1}^n X_i}{ \left[n + \frac{1}{\sigma^2}\right] } \right)^2 \right) \cdot \textcolor{#008080}{\text{[constant, non-functions of $\theta$]}}\end{align*}

We can throw away the constant, non-functions of θ\theta that accumulate when completing the square. The result is:

posterior=N(∑i=1nXi[n+1σ2],1[n+1σ2]).\text{posterior} = \mathcal{N}\left( \frac{\sum_{i = 1}^n X_i}{\left[n + \frac{1}{\sigma^2}\right]}, \frac{1}{\left[n + \frac{1}{\sigma^2}\right]} \right).

There's not much insightful commentary to give here, other than pointing out the obvious conjugate prior.