MIT 18.650 — Lecture 8

Confidence Intervals

§ Asymptotic Normality

Definition. An estimator $\hat{\theta}_n$ of $\theta$ is asymptotically normal if there is some asymptotic variance $\sigma^2 > 0$ such that:

\sqrt{n}(\hat{\theta}_n - \theta) \rightsquigarrow \mathcal{N}(0, \sigma^2).

Informally, this is roughly equivalent to $\hat{\theta}_n \approx \theta + \frac{\sigma}{\sqrt{n}}Z$ , where $Z \sim \mathcal{N}(0, 1)$ .

Example. Given model $\{\mathrm{Ber}(p) \mid p \in (0, 1)\}$ and estimator $\hat{\theta}_n := \bar{X}_n$ of $\theta := p$ , determine its asymptotic variance.

Solution: Since $\mathbb{E}_p[X_i] = p = \theta$ and $\mathbb{V}_p[X_i] = p(1 - p)$ , the CLT says that the asymptotic variance is $p(1 - p)$ .

Example. Given model $\{\mathrm{Ber}(p) \mid p \in (0, 1)\}$ and estimator $\hat{\theta}_n := (\bar{X}_n)^2$ of $\theta := p^2$ , determine its asymptotic variance.

Solution: Invoke the delta method with $g(x) = x^2$ ; the asymptotic variance gets scaled by $g'(p)^2 = 4p^2$ , so it's $4p^3(1 - p)$ .

Remark. If $\hat{\theta}_n$ is asymptotically normal with asymptotic variance $\sigma^2$ , that tells us that $\hat{\theta}_n \approx \mathcal{N}(\theta, \frac{\sigma^2}{n})$ .

However, that does not imply that the variance of $\hat{\theta}_n$ equals—or even converges to—the variance of $\mathcal{N}(\theta, \frac{\sigma^2}{n})$ . Convergence in distribution does not imply convergence of moments.

Here's an explicit counterexample: consider an estimator $\hat{\theta}_n$ that happens to satisfy $\hat{\theta}_n := \theta + \frac{Y_n}{\sqrt{n}}$ , with the random variable $Y_n$ defined as below.

Z \sim \mathcal{N}(0, \sigma^2) ~~~ \parallel ~~~ Y_n := \begin{cases} Z & \text{ with probability } 1 - \frac{1}{n}. \\ n & \text{ with probability } \frac{1}{n}. \end{cases}

Then we have a convergence in distribution $Y_n \rightsquigarrow \mathcal{N}(0, \sigma^2)$ , implying that $\sqrt{n}(\hat{\theta}_n - \theta) \rightsquigarrow \mathcal{N}(0, \sigma^2)$ . In other words, the estimator $\hat{\theta}_n$ is asymptotically normal with asymptotic variance $\sigma^2$ . However, we may compute $\mathbb{V}[\hat{\theta}_n] = \frac{1}{n} \cdot \mathbb{V}[Y_n] \approx 1$ as $n \to \infty$ , which does not match the asymptotic variance $\sigma^2/n \to 0$ .

That said, in most well-behaved cases (read: under regularity conditions), it will generally be true that $\mathbb{V}[\hat{\theta}_n] \approx \frac{\sigma^2}{n}$ . So it is still generally fine to intuit $\sigma^2$ and $n \cdot \mathbb{V}[\hat{\theta}_n]$ as the same things.

§ Confidence Intervals

Definition. We say a $(1 - \alpha)$ confidence interval (CI) for a parameter $\theta$ is a random interval of the form $C_n = (A_n, B_n)$ such that, for any fixed value of $\theta$ , we have $\Pr_{\theta}(\theta \in (A_n, B_n)) \geq 1 - \alpha$ . We say $1 - \alpha$ is the coverage of the CI.

Remark. For any fixed $\theta$ , the endpoints $A_n$ and $B_n$ are themselves random variables whose distributions (usually) depend on $\theta$ .

If it turns out $(6, 7)$ is a $95\%$ confidence interval for $\theta$ , that does not mean $\Pr_{\theta}(\theta \in (6, 7)) \geq 0.95$ . That statement doesn't even make sense—either $\theta \in (6, 7)$ or $\theta \not \in (6, 7)$ , and there (usually) is no distribution on the possible values of $\theta$ .

Definition. A sequence of random intervals $\{C_n\}_{n = 1}^{\infty}$ has asymptotic coverage $1 - \alpha$ if $\liminf_{n \to \infty} \Pr_{\theta}(\theta \in C_n) \geq 1 - \alpha$ .

Theorem. (Constructing CIs) Suppose $\hat{\theta}_n$ is an asymptotically normal estimator of $\theta$ with asymptotic variance $\sigma^2$ . Then:

C_n := \left( \hat{\theta}_n - z_{\alpha / 2} \frac{\sigma}{\sqrt{n}}, \hat{\theta}_n + z_{\alpha / 2} \frac{\sigma}{\sqrt{n}} \right)

has asymptotic coverage $(1 - \alpha)$ , where $z_{\alpha / 2}$ is such that $\Phi(z_{\alpha / 2}) - \Phi(-z_{\alpha / 2}) = 1 - \alpha$ .

Proof: Just note that $\theta \in C_n$ is equivalent to $|\hat{\theta}_n - \theta| < z_{\alpha / 2} \frac{\sigma}{\sqrt{n}}$ , but also $\hat{\theta}_n - \theta \approx \mathcal{N}(0, \frac{\sigma^2}{n})$ . $\blacksquare$

Example. Given (i.i.d.) samples $X_1, \dots, X_n \sim \mathrm{Ber}(p)$ , construct confidence intervals for $p$ with asymptotic coverage $1 - \alpha$ .

Solution: The challenge is that $\sigma = \sqrt{p(1 - p)}$ depends on $p$ , but we don't know $p$ .

The fix is to conservatively bound $\sigma \leq \frac{1}{2}$ , which yields $C_n = \bar{X}_n \pm \frac{z_{\alpha / 2}}{2\sqrt{n}}$ .