MIT 18.650 — Lecture 14

§ Bootstrap Properties

You can verify that all of the following are true.

Not very surprising, and not hard to prove.

§ Constructing CIs

Recall from Lecture 13 that, for any estimator $\hat{\theta}_n$, we can use bootstrapping to compute an estimate $v_{\mathrm{boot}}$ of its variance.

Let's say $\hat{\theta}_0 := \hat{\theta}_n(X_1, \dots, X_n)$ is the observed point estimate we retrieved after considering all $n$ samples. Then if $\hat{\theta}_n$ is asymptotically normal, we can write, say, $\hat{\theta}_0 \pm 1.96\sqrt{v_{\mathrm{boot}}}$ as a $95\%$ confidence interval.

But what if $\hat{\theta}_n$ isn't approximately normal? Here's what we do instead.

Approach (Pivotal CI). Suppose we have the following:

• We have $n$ samples $\{X_i\}_{i = 1}^n \sim \mathbb{P}_{\theta^*}$, producing an observed value $\hat{\theta}_0$ of $\hat{\theta}_n$.

• This lets us produce $B$ point estimates $\{\hat{\theta}_b\}_{b = 1}^{B}$.

  • Think of the $\{\hat{\theta}_b\}_{b = 1}^B$ as approximating the sampling distribution of $\hat{\theta}_n$ under $\mathbb{P}_{\theta^*}$.

• Our goal is to construct a $(1 - \alpha)$ CI for $\theta^*$.

Recall that CIs should be inherently random. We'd like to fix constants $(r_1, r_2)$ and take our CI to be $(\hat{\theta}_n - r_1, \hat{\theta}_n + r_2)$ so that:

$$1 - \alpha = \mathbb{P}_{\theta^*}(\hat{\theta}_n - r_1 \leq \theta^* \leq \hat{\theta}_n + r_2) = \mathbb{P}_{\theta^*}(\theta^* - r_2 \leq \hat{\theta}_n \leq \theta^* + r_1).$$

Note that the randomness in our CI and in the above comes from the randomness of $\hat{\theta}_n$.

If we knew the exact value of $\theta^*$ and the exact sampling distribution of $\hat{\theta}_n$ under $\mathbb{P}_{\theta^*}$, here's how we would pick $(r_1, r_2)$.

1. Find $q_{\alpha/2}$ such that $\mathbb{P}_{\theta^*}(\hat{\theta}_n < q_{\alpha/2}) = \frac{\alpha}{2}$, and find $q_{1 - \alpha/2}$ such that $\mathbb{P}_{\theta^*}(\hat{\theta}_n > q_{1 - \alpha/2}) = \frac{\alpha}{2}$.

2. Set $r_2 = \theta^* - q_{\alpha/2}$ and $r_1 = q_{1 - \alpha/2} - \theta^*$.

The problem, of course, is that we don't know $(q_{\alpha/2}, q_{1 - \alpha/2}, \theta^*)$. But we can estimate them using our bootstrap samples and $\hat{\theta}_0$.

1. Find $\hat{q}_{\alpha/2}$ such that $\frac{\alpha}{2}$ of the $\{\hat{\theta}_b\}_{b = 1}^B$ are at most $\hat{q}_{\alpha/2}$, and find $\hat{q}_{1 - \alpha/2}$ such that $\frac{\alpha}{2}$ of the $\{\hat{\theta}_b\}_{b = 1}^B$ are at least $\hat{q}_{1 - \alpha/2}$.

   • (Here, we estimate $q_{\alpha / 2}$ and $q_{1 - \alpha / 2}$ by estimating $\mathbb{P}_{\theta^*}$ using the $B$ bootstrap estimates $\{\hat{\theta}_b\}_{b = 1}^B$.)

2. Set $\hat{r}_2 = \hat{\theta}_0 - \hat{q}_{\alpha/2}$ and $\hat{r}_1 = \hat{q}_{1 - \alpha/2} - \hat{\theta}_0$.

   • (Here, we estimate $\theta^*$ by taking $\theta^* \approx \hat{\theta}_0$, recalling $\hat{\theta}_0$ was our $n$-sample point estimate for $\theta^*$.)

After retrieving $\hat{r}_1$ and $\hat{r}_2$ as above, we set our CI to be $(\hat{\theta}_0 - \hat{r}_1, \hat{\theta}_0 + \hat{r}_2)$.

You can also use the (inferior) percentile CI, which just returns $(\hat{q}_{\alpha/2}, \hat{q}_{1 - \alpha/2})$.

We discussed earlier in Lecture 8 why random CIs (the pivotal CI) are preferred over deterministic intervals (the percentile CI).

• Say the pivotal CI is a random interval $\mathcal{I}$. That tells us that $\mathrm{Pr}_{\theta^*}(\theta^* \in \mathcal{I}) = 1 - \alpha$.

• Say the percentile CI is a constant interval $\mathcal{I}$. That tells us that $\mathrm{Pr}_{\theta^*}(\hat{\theta}_n \in \mathcal{I}) = 1 - \alpha$.

Stated more explicitly, the pivotal CI is a random interval that captures the fixed $\theta^*$, whereas the percentile CI is a fixed interval that captures the random $\hat{\theta}_n$.

§ Example: Bootstrapping a Median CI

Recall the example from Lecture 13:

• Taking $n = 5$ samples $\{0.3, 0.5, 0.9, 1.4, 3.1\}$ gave a point estimate of $\hat{\theta}_0 = 0.9$.

• Taking $B = 4$ bootstrap samples gave $B = 4$ additional point estimates $\{\hat{\theta}_b\}_{b = 1}^4 = \{0.9,\, 0.3,\, 1.4,\, 0.9\}$ with $v_{\mathrm{boot}} \approx 0.1519$.

If we just assume that $\hat{\theta}_n$ is asymptotically normal, we get a $50\%$ CI immediately:

Now suppose we don't assume asymptotic normality. To compute a $50\%$ CI, we note:

• We have $\hat{q}_{0.25} = 0.3$ because $25\%$ of the $\{\hat{\theta}_b\}_{b = 1}^4$ are at or below $0.3$. Similarly, we have $\hat{q}_{0.75} = 1.4$.

• Therefore, we take $\hat{r}_2 = \hat{\theta}_0 - \hat{q}_{0.25} = 0.6$ and $\hat{r}_1 = \hat{q}_{0.75} - \hat{\theta}_0 = 0.5$.

And so our $50\%$ CIs are:

• The pivotal CI is the random interval $(\hat{\theta}_n - 0.5, \hat{\theta}_n + 0.6)$. Given the point estimate $\hat{\theta}_0 = 0.9$, this simplifies to $(0.4, 1.5)$.

• The percentile CI is just $(\hat{q}_{0.25}, \hat{q}_{0.75}) = (0.3, 1.4)$, which is estimating a constant interval.