MIT 18.650 — Lecture 21

Bayesian Inference I

§ Review: Likelihood and Multivariate Distributions

Suppose we have nn independent random variables X1,…,XnX_1, \dots, X_n, each with a PDF fi(xi∣θ)f_i(x_i \mid \theta) with a dependency on a parameter θ\theta. As an example, we might have something like Xi∼N(θ,i)X_i \sim \mathcal{N}(\theta, i).

If we're given the values X1,…,XnX_1, \dots, X_n, that lets us compute a likelihood function Ln(θ)L_n(\theta) that tells us how likely it would be to have observed this data given that the parameter had value θ\theta. (Recall Lecture 9.)

Ln(θ):=∏i=1nfi(Xi∣θ)=f(X1,…,Xn∣θ).L_n(\theta) := \prod_{i = 1}^n f_i(X_i \mid \theta) = f(X_1, \dots, X_n \mid \theta).

In the above, f(x1,…,xn∣θ)f(x_1, \dots, x_n \mid \theta) is the joint PDF. (Recall Lecture 6.)

§ Priors and Posteriors

Now suppose we have some prior belief about the distribution of the parameter θ\theta. As an example, perhaps θ\theta is a probability that we believe to be more likely to be high than low.

Definition. A prior is a PDF f(θ)f(\theta) that describes the probability distribution taken on by the parameter θ\theta.

We now read in some data X1,…,XnX_1, \dots, X_n that has a dependency on θ\theta. Bayes' rule tells us the following:

Pr(θ∣X1,…,Xn)=Pr(X1,…,Xn∣θ)⋅Pr(θ)Pr(X1,…,Xn)=Ln(θ)⋅f(θ)Pr(X1,…,Xn)∝Ln(θ)⋅f(θ).\mathrm{Pr}(\theta \mid X_1, \dots, X_n) = \frac{ \mathrm{Pr}(X_1, \dots, X_n \mid \theta) \cdot \mathrm{Pr}(\theta) }{ \mathrm{Pr}(X_1, \dots, X_n) } = \frac{L_n(\theta) \cdot f(\theta)}{\mathrm{Pr}(X_1, \dots, X_n)} \propto L_n(\theta) \cdot f(\theta).

The LHS is our updated, posterior belief about the distribution of the parameter θ\theta.

Remark. Notice how we don't care about the exact value of Pr(X1,…,Xn)\mathrm{Pr}(X_1, \dots, X_n), because it is effectively a normalizing constant that works only to make the integral of Pr(θ∣X1,…,Xn)\mathrm{Pr}(\theta \mid X_1, \dots, X_n) equal to one.

Definition. The posterior f(θ∣X1,…,Xn)f(\theta \mid X_1, \dots, X_n) is the probability distribution proportional to Ln(θ)⋅f(θ)L_n(\theta) \cdot f(\theta).

§ Conjugate Priors

Definition. The Beta distribution Beta(a,b)\mathrm{Beta}(a, b) (where a,b>0a, b > 0) has PDF f ⁣:[0,1]→R≥0f \colon [0, 1] \to \mathbb{R}^{\geq 0} given by f(p)∝pa−1⋅(1−p)b−1f(p) \propto p^{a - 1} \cdot (1 - p)^{b - 1}.

Remark. For example, the PDF of Beta(2,2)\mathrm{Beta}(2, 2) is f(p)=6⋅p⋅(1−p)f(p) = 6 \cdot p \cdot (1 - p).

Example. Suppose we sample data i.i.d. from Ber(θ)\mathrm{Ber}(\theta), and our prior is θ∼Beta(a,b)\theta \sim \mathrm{Beta}(a, b). In terms of the data X1,…,XnX_1, \dots, X_n, what is the posterior distribution?

Solution: Suppose AA of the samples are one, and the remaining B=n−AB = n - A samples are zero. Then we have:

f(θ∣X1,…,Xn)∝Ln(θ)⋅f(θ)∝[θA⋅(1−θ)B]⋅[θa−1⋅(1−θ)b−1]=θ(a+A)−1⋅(1−θ)(b+B)−1.\begin{align*}f(\theta \mid X_1, \dots, X_n) & \propto L_n(\theta) \cdot f(\theta) \\ & \propto \left[\theta^A \cdot (1 - \theta)^B\right] \cdot \left[\theta^{a - 1} \cdot (1 - \theta)^{b - 1}\right] \\ & = \theta^{(a + A) - 1} \cdot (1 - \theta)^{(b + B) - 1}.\end{align*}

Therefore, the posterior distribution is Beta(a+A,b+B)\mathrm{Beta}(a + A, b + B).

Definition. If the prior and posterior live in the same family of distributions, we say that we have a conjugate prior.

Remark. In the previous example, Beta(a,b)\mathrm{Beta}(a, b) was a conjugate prior. But this fact relies on the distributions of the sample data all being Xi∼Ber(θ)X_i \sim \mathrm{Ber}(\theta), rather than, say, Xi∼N(θ,i)X_i \sim \mathcal{N}(\theta, i).

§ Bayes Estimator and Max A Posteriori

As we read in more data, our posterior distribution becomes more informative. In fact, one can show that the standard deviation of the posterior decreases as 1/n1/\sqrt{n} with the number of samples nn.

Eventually, we might want to summarize the posterior distribution f(θ∣X1,…,Xn)f(\theta \mid X_1, \dots, X_n) with a single point estimate for θ\theta.

Definition. The Bayes estimator (or mean a posteriori) is the expected value of θ\theta under the posterior distribution.

Definition. The max a posteriori (MAP) is the mode of the posterior; that is, the value of θ\theta that maximizes f(θ∣X1,…,Xn)f(\theta \mid X_1, \dots, X_n).

Generally, the MAP is much easier to compute than the Bayes estimator, because it doesn't require computing any normalizing factors.

Remark. If we have no priors at all—that is, our prior f(θ)f(\theta) is the uniform distribution—then the max a posteriori is just the MLE. The spirit of Bayesian inference is that we must now account for priors.