MIT 18.650 — Lecture 13

Method of Moments and Bootstrap

§ Method of Moments Estimator

Let's briefly discuss one last method of defining an estimator; we won't go into too much detail.

Definition. Given a random variable $X$ , its $j^{\text{th}}$ moment is $\alpha_j := \mathbb{E}[X^j]$ .

This lets us define the Method of Moments Estimator.

MoM Estimator. We have (i.i.d.) samples $\{X_i\}_{i = 1}^n \sim f_{\theta}$ , where $\theta = (\theta_1, \theta_2, \dots, \theta_k)$ is a vector of all $k$ parameters of interest.

First, estimate the first $k$ moments of $X$ via $\hat{\alpha}_j = \frac{1}{n} \sum_{i = 1}^n X_i^j$ .
This gives us $k$ equations $\{\hat{\alpha}_j = \alpha_j\}$ , where the LHS of each equation is a sample moment, and the RHS is the corresponding population moment as a function of $(\theta_1, \theta_2, \dots, \theta_k)$ .
Since we have $k$ equations and $k$ unknowns, we can solve for $(\theta_1, \theta_2, \dots, \theta_k)$ . This solution is our estimate.

Remark. The typical estimator $\hat{\theta}_n := \bar{X}_n$ for the mean of a distribution is an example of the $k = 1$ case of an MoM estimator.

When solving the system of equations in closed form is not possible, we might resort to the Newton-Raphson method (e.g.), just as we did for computing the MLE.

It also turns out that the MoM estimator is consistent and asymptotically normal, with a computable asymptotic variance $\Sigma^{\mathrm{MoM}}$ ; roughly speaking, the asymptotic variance is found by applying the delta method to the inverse of the moment map. Note that the MoM estimator, unlike the MLE, need not attain the Cramér-Rao lower bound with equality.

§ Motivating Bootstrapping

We care about asymptotic variance of estimators because it's needed to write confidence intervals. Recall from Lecture 8:

Theorem. (Constructing CIs) Suppose $\hat{\theta}_n$ is an asymptotically normal estimator of $\theta$ with asymptotic variance $\sigma^2$ . Then:

C_n := \left( \hat{\theta}_n - z_{\alpha / 2} \frac{\sigma}{\sqrt{n}}, \hat{\theta}_n + z_{\alpha / 2} \frac{\sigma}{\sqrt{n}} \right)

has asymptotic coverage $(1 - \alpha)$ , where $z_{\alpha / 2}$ is such that $\Phi(z_{\alpha / 2}) - \Phi(-z_{\alpha / 2}) = 1 - \alpha$ .

There are three specific kinds of estimators we've discussed so far, and we know the asymptotic variance of all of them.

Given the estimator $\hat{\theta}_n = \bar{X}_n$ … its asymptotic variance is found by the CLT.
Given the estimator $\hat{\theta}_n = g(\bar{X}_n)$ … its asymptotic variance is found by the Delta method.
Given the estimator $\hat{\theta}_n = \hat{\theta}^{\mathrm{MLE}}$ … its asymptotic variance is found by the Fisher information.

In general, though, it might not be so easy to find the asymptotic variance.

Example. Let $\theta$ be the median of a distribution with a strictly positive PDF $f$ ; formally, the unique $\theta$ such that:
$\int_{-\infty}^{\theta} f(x) \, \mathrm{d}x = 0.5.$
We might naturally consider the estimator $\hat{\theta}_n := \mathrm{Median}(X_1, X_2, \dots, X_n)$ . But what is its asymptotic variance…?

We need another way to compute standard errors—that way is called bootstrapping.

§ Bootstrapping

The idea is to suppose we had $B$ batches of $n$ samples each. Then each of the $B$ batches gives its own value $\hat{\theta}_b$ for the estimator.

\{X_{1, 1}, X_{1, 2}, \dots, X_{1, n}\} \mapsto \hat{\theta}_1 ~~~ \parallel ~~~ \{X_{2, 1}, X_{2, 2}, \dots, X_{2, n}\} \mapsto \hat{\theta}_2 ~~~ \parallel ~~~ \dots ~~~ \parallel ~~~ \{X_{B, 1}, X_{B, 2}, \dots, X_{B, n}\} \mapsto \hat{\theta}_B

Remark. Previously, $\hat{\theta}_n$ denoted the value of the estimator that results from taking $n$ samples. Now, $\hat{\theta}_b$ denotes the value of the estimator from the $b^{\text{th}}$ batch of samples.

We could then use these $B$ values $\{\hat{\theta}_b\}_{b = 1}^B$ to estimate the variance of $\hat{\theta}_n$ like so:

\widehat{\mathbb{V}[\hat{\theta}_n]} = \frac{1}{B} \sum_{b = 1}^B \left( \hat{\theta}_b - \frac{1}{B} \sum_{c = 1}^B \hat{\theta}_c \right)^2

The issue, however, is that we might not have $Bn \gg n$ total samples from the population. The trick is to consider the following.

Definition. Given $n$ samples $\{X_1, \dots, X_n\}$ , the empirical distribution is the probability distribution:

\hat{\mathbb{P}}_n := \mathrm{Uniform}(\{X_1, \dots, X_n\}).

In other words, sampling from $\hat{\mathbb{P}}_n$ just returns one of the $n$ samples $\{X_i\}_{i = 1}^n$ uniformly at random.

Remark. Note that the distribution $\hat{\mathbb{P}}_n$ , itself, is random, because it depends on the values of random variables.

Definition. Given $n$ samples $\{X_1, \dots, X_n\}$ , a bootstrap sample is a collection of $m$ random variables $\{X_1^*, \dots, X_m^*\}$ sampled (i.i.d.) from $\hat{\mathbb{P}}_n$ .

Example. Given the $n = 3$ samples $\{5.2, 7.3, 6.8\}$ , one possible bootstrap sample of size $m = 4$ is $\{7.3, 7.3, 6.8, 7.3\}$ .

Thus, instead of independent real batches, we can just take $B$ bootstrap samples of size $n$ .

§ Example: Bootstrapping the Median

Let's use the estimator $\hat{\theta}_n := \mathrm{Median}(X_1, \dots, X_n)$ to estimate the median of a mystery distribution $\mathbb{P}$ , then use bootstrapping to estimate the asymptotic variance of $\hat{\theta}_n$ . Say we take $n = 5$ samples, and they come out as:

\{X_1, X_2, X_3, X_4, X_5\} = \{0.3, 0.5, 0.9, 1.4, 3.1\}.

Then the observed point estimate (call it $\hat{\theta}_0$ , for the original sample) is $\hat{\theta}_0 = 0.9$ . Now we draw $B = 4$ bootstrap samples of size $m = 5$ from the empirical distribution $\hat{\mathbb{P}}_5$ and compute the median of each.

\begin{align*}\{0.5,\, 3.1,\, 0.9,\, 0.5,\, 1.4\} &\mapsto \hat{\theta}_1 = 0.9 \\ \{0.3,\, 0.3,\, 1.4,\, 0.9,\, 0.3\} &\mapsto \hat{\theta}_2 = 0.3 \\ \{1.4,\, 0.9,\, 3.1,\, 1.4,\, 0.5\} &\mapsto \hat{\theta}_3 = 1.4 \\ \{0.9,\, 0.5,\, 0.3,\, 3.1,\, 0.9\} &\mapsto \hat{\theta}_4 = 0.9\end{align*}

Remark. The size $m = 5$ of the bootstrap samples really should be the same as the total number $n = 5$ of samples. Otherwise, our bootstrap samples would be describing the spread of $\hat{\theta}_m$ , not $\hat{\theta}_n$ .

The four medians average to $\frac{1}{4}(0.9 + 0.3 + 1.4 + 0.9) = 0.875$ , so their variance is:

v_{\mathrm{boot}} = \frac{(0.9 - 0.875)^2 + (0.3 - 0.875)^2 + (1.4 - 0.875)^2 + (0.9 - 0.875)^2}{4} \approx 0.1519.

Thus, our estimate of the asymptotic variance of $\hat{\theta}_n$ is $\hat{\sigma}^2 = n \cdot v_{\mathrm{boot}} \approx 0.76$ .

Remark. To be explicit about the exact reason why bootstrapping works…

For large enough $B$ , the law of large numbers says we should expect $v_{\mathrm{boot}} \approx \mathbb{V}_{\hat{\mathbb{P}}_n}[\hat{\theta}_n]$ .
For large enough $n$ , we should expect $\hat{\mathbb{P}}_n \approx \mathbb{P}$ , which yields $\mathbb{V}_{\hat{\mathbb{P}}_n}[\hat{\theta}_n] \approx \mathbb{V}_{\mathbb{P}}[\hat{\theta}_n]$ .
And if $\hat{\theta}_n$ is asymptotically normal with asymptotic variance $\sigma^2$ , then $\mathbb{V}_{\mathbb{P}}[\hat{\theta}_n] \approx \frac{\sigma^2}{n}$ .
- For this third bullet point, keep in mind the caveat from Lecture 8.

Altogether, this means $n \cdot v_{\mathrm{boot}}$ is indeed a reasonable estimator of the asymptotic variance $\sigma^2$ .

Next lecture, we'll see how computing this asymptotic variance will be helpful towards constructing confidence intervals.

Remark. Recall that a confidence interval only really cares about $\mathbb{V}_{\mathbb{P}}[\hat{\theta}_n]$ itself; the asymptotic variance $\sigma^2$ is just an alternative $n$ -scaled way of interpreting it.

Consequently, the next lecture on confidence intervals can avoid using the “third bullet point” and rely instead only on the approximation $v_{\mathrm{boot}} \approx \mathbb{V}_{\theta}[\hat{\theta}_n]$ .

The upshot of this is that the caveat from Lecture 8 will not be relevant to the “pivotal CI” at all! (Though it will still be relevant for the “normal approximation CI”.)