MIT 18.650 — Lecture 7

Models and Estimators

§ Models

One of the biggest goals of statistics is the following.

Goal. Given some (i.i.d.) data $X_i \sim \mathbb{P}$ , derive an estimate $\hat{\mathbb{P}}$ for the distribution $\mathbb{P}$ .

This is an unreasonable goal without a model—a set of probability distributions that are candidates for $\mathbb{P}$ .

Definition. A model $\{\mathbb{P}_{\theta} \mid \theta \in \Theta\}$ is parametric if $\Theta$ is finite-dimensional, and nonparametric otherwise.

Example. The model $\{\mathrm{Exp}(\lambda) \mid \lambda \in \mathbb{R}^+\}$ is parametric, whereas $\{\text{PDF } f: [0, 1] \to \mathbb{R}^+\}$ is not.

If we assume a model like, say, $\{\mathcal{N}(\mu, \sigma^2) \mid \mu \in \mathbb{R},\sigma \in \mathbb{R}^+\}$ , we may also use notation like $\mathbb{E}_{\mu, \sigma^2}[X]$ .

§ Estimators

Definition. Given a parameter $\theta$ …

A point estimate $\hat{\theta}$ (or $\hat{\theta}_n$ ) is a single guess for the value of $\theta$ .
An estimator $\hat{\theta}$ is a random variable that is a function $\hat{\theta} = g(X_1, \dots, X_n)$ of the sample data.
We'll overload the notation $\theta \rightsquigarrow \hat{\theta}$ to say “ $\theta$ is estimated by $\hat{\theta}$ ”.

In what follows, assume the parameter $\theta$ is a real number.

Definition. The bias of an estimator $\hat{\theta}$ of $\theta$ is $\mathrm{bias}(\hat{\theta}) := \mathbb{E}_{\theta}[\hat{\theta}] - \theta$ .

We say $\hat{\theta}$ is unbiased if $\mathrm{bias}(\hat{\theta}) = 0$ .
We say $\hat{\theta}$ is asymptotically unbiased if $\lim_{n \to \infty} \mathrm{bias}(\hat{\theta}_n) = 0$ .

Definition. The standard error of an estimator $\hat{\theta}$ of $\theta$ is $\mathrm{se}(\hat{\theta}) := \sqrt{\mathbb{V}[\hat{\theta}]}$ , or just the standard deviation of $\hat{\theta}$ .

Definition. The mean squared error (MSE) of an estimator $\hat{\theta}$ of $\theta$ is $\mathrm{MSE}(\hat{\theta}) := \mathbb{E}_{\theta}[(\hat{\theta} - \theta)^2]$ .

Remark. Note that, in general, the bias, standard error, and MSE are all implicitly functions of the true, population value of $\theta$ .

For example, the equation $\mathrm{bias}(\hat{\theta}) := \mathbb{E}_{\theta}[\hat{\theta}] - \theta$ uses the notation $\mathbb{E}_{\theta}$ to reflect that in the definition $\hat{\theta} := g(X_1, \dots, X_n)$ of $\hat{\theta}$ , the random variables $\{X_i\}_{i = 1}^n$ are sampled (i.i.d.) from $\mathbb{P}_{\theta}$ . So there are implicit dependencies on the value of $\theta$ everywhere.

It turns out that the MSE is just a combination of bias and standard error.

Theorem. (MSE Equivalence) We always have $\mathrm{MSE}(\hat{\theta}) = \mathrm{bias}^2(\hat{\theta}) + \mathrm{se}^2(\hat{\theta})$ .

Proof: Just expand the definition of the MSE.

\begin{align*}\mathbb{E}_{\theta}[(\hat{\theta} - \theta)^2] & = \mathbb{E}_{\theta}[( \hat{\theta} - \mathbb{E}_{\theta}[\hat{\theta}] + \mathrm{bias}(\hat{\theta}) )^2] \\ & = \underbrace{\mathbb{E}_{\theta}[(\hat{\theta} - \mathbb{E}_{\theta}[\hat{\theta}])^2]}_{ \mathrm{se}^2(\hat{\theta}) } + \underbrace{2 \cdot \mathbb{E}_{\theta}[\hat{\theta} - \mathbb{E}_{\theta}[\hat{\theta}]] \cdot \mathrm{bias}(\hat{\theta})}_{ 0 } + \underbrace{\mathrm{bias}^2(\hat{\theta})}_{\mathrm{bias}^2(\hat{\theta})}. ~~~ \blacksquare\end{align*}

Remark. When $\mathrm{bias}(\hat{\theta}) = 0$ , the above is just the proof that $\mathbb{V}[X] = \mathbb{E}[(X - \mathbb{E}[X])^2]$ .

Definition. An estimator $\hat{\theta}_n$ of $\theta$ is consistent if $\hat{\theta}_n \convprob \theta$ as $n \to \infty$ .

Example. Given the model $\{\mathcal{N}(\mu, 1) \mid \mu \in \mathbb{R}\}$ , consider these three estimators for $\mu$ .

Take $\hat{\mu}_1 := \bar{X}_n$ . This is a consistent, unbiased estimator with standard error $1/\sqrt{n}$ and MSE $1/n$ .
Take $\hat{\mu}_2 := X_1$ . This is an inconsistent, unbiased estimator with standard error $1$ and MSE $1$ .
Take $\hat{\mu}_3 := 0$ . This is an inconsistent, biased estimator with standard error $0$ and MSE $\mu^2$ .

Theorem. (Consistency and MSE) If $\mathrm{MSE}(\hat{\theta}_n) \to 0$ as $n \to \infty$ , then $\hat{\theta}_n$ is consistent.

Proof: Recall the definition of convergence in probability; we would like the following:

Goal. For all $\epsilon > 0$ , we have $\Pr_{\theta}(|\hat{\theta}_n - \theta| > \epsilon) \to 0$ as $n \to \infty$ .

Well, taking $Y_n := (\hat{\theta}_n - \theta)^2$ , we are given that $\mathbb{E}_{\theta}[Y_n] \to 0$ as $n \to \infty$ . The finish is Markov's inequality:

\Pr_{\theta}(|\hat{\theta}_n - \theta| > \epsilon) = \Pr_{\theta}(Y_n > \epsilon^2) \leq \frac{\mathbb{E}_{\theta}[Y_n]}{\epsilon^2} \to 0 ~ \text{ as } n \to \infty. ~~~ \blacksquare