MIT 18.650 — Lecture 11

MLE Asymptotic Normality

§ Defining Score

We claim the MLE also achieves asymptotic normality. In fact, we claim we even know its asymptotic variance. It turns out, however, that we'll need a little more setup before we get to asymptotic variance.

Definition. Consider a $\underline{\text{statistical model}}$ with a parameter $\theta$ and varying PDFs given by $f_{\theta}(x)$ , dependent on $\theta$ . Given an observation $x$ and a proposed value $\theta$ of the parameter, the score is:

s(x; \theta) := \frac{\partial}{\partial\theta} \log f_{\theta}(x) = \frac{\frac{\partial}{\partial\theta} f_{\theta}(x)}{f_{\theta}(x)}.

The RHS comes from a single application of the chain rule. When $s(x; \theta)$ is positive, it means that $x$ would be better explained if the proposed $\theta$ were increased, and vice versa.

Remark. The notation is slightly confusing; as a simpler example, consider defining $g(x) := \frac{d}{dx} x^2$ and concluding $g(5) = 10$ .

Theorem. (Zero Mean Score) We have $\mathbb{E}_{\theta^*}[s(X; \theta^*)] = 0$ , where the $\mathbb{E}_{\theta^*}$ notation indicates that $X$ is sampled from the distribution $\mathbb{P}_{\theta^*}$ ; in other words, the proposed value $\theta^*$ is correct.

In this case, the score has some positives and negatives, but expectedly is overall neutral in how the proposal $\theta^*$ should change.

Proof: Just some integral manipulation.

\mathbb{E}_{\theta^*}[s(X; \theta^*)] = \int_{\mathcal{R}} f_{\theta^*}(x) \cdot \left[\frac{\frac{\partial}{\partial\theta} f_{\theta}(x)}{f_{\theta}(x)}\right]_{\theta = \theta^*} \mathrm{d}x = \int_{\mathcal{R}} \left[\frac{\partial}{\partial\theta} f_{\theta}(x)\right]_{\theta = \theta^*} \mathrm{d}x = \frac{\partial}{\partial\theta} \left[\int_{\mathcal{R}} f_{\theta}(x) \, \mathrm{d}x\right]_{\theta = \theta^*} = \frac{\partial}{\partial\theta} [1] = 0. ~~~ \blacksquare

§ Defining Fisher Information

Note that the definition of score doesn't care whether $x$ is sampled from a distribution that uses the proposed value $\theta$ for its parameter. The definition of Fisher information, on the other hand, does care.

Definition. Consider a $\underline{\text{statistical model}}$ with a parameter $\theta$ and varying PDFs given by $f_{\theta}(x)$ , dependent on $\theta$ . As a function of the true value of $\theta$ , the Fisher information of the resulting distribution is:

I(\theta) := \mathbb{V}_{\theta}\left[s(X; \theta)\right] = \mathbb{E}_{\theta}[s(X; \theta)^2].

As usual, the $\mathbb{E}_{\theta}$ and $\mathbb{V}_{\theta}$ notation indicates that $X \sim \mathbb{P}_{\theta}$ ; in other words, that $\theta$ is the “true value” of the parameter. It's because of this and the previous theorem that the above two expressions for $I(\theta)$ are equivalent.

When $I(\theta)$ is large, it means a single observation $x$ will tend to suggest very large changes to the proposed $\theta$ . In other words, singular observations $x$ carry more information.

Example. Given the model $\{\mathcal{N}(\theta, 9) \mid \theta \in \mathbb{R}\}$ , determine $I(\theta)$ . Do the same for $\{\mathcal{N}(\theta, 900) \mid \theta \in \mathbb{R}\}$ .

Solution: Recall that the PDF of $X \sim \mathcal{N}(\theta, 9)$ (with $\sigma = 3$ ) is given by:

f_{\theta}(x) = \frac{1}{3\sqrt{2\pi}} \exp\left(-\frac{(x - \theta)^2}{2 \times 3^2}\right) ~ \implies ~ s(x; \theta) = \frac{\partial}{\partial\theta} \log f_{\theta}(x) = \frac{x - \theta}{3^2}.

The Fisher information is therefore:

I(\theta) = \mathbb{V}_{\theta}\left[ \frac{X - \theta}{3^2} \right] = \left(\frac{1}{3^2}\right)^2 \mathbb{V}_{\theta}[X - \theta] = \left(\frac{1}{3^2}\right)^2(3^2) = \frac{1}{9}.

It's a constant! Expectedly, the Fisher information for $\{\mathcal{N}(\theta, 900) \mid \theta \in \mathbb{R}\}$ is constantly $\frac{1}{900}$ .

Notably, the Fisher information of the latter is smaller than the former. This should make sense; a single observation from $\{\mathcal{N}(\theta, 900) \mid \theta \in \mathbb{R}\}$ tells you a lot less about $\theta$ than a single observation from $\{\mathcal{N}(\theta, 9) \mid \theta \in \mathbb{R}\}$ does.

The Fisher information has one more alternative representation.

Theorem. (Equivalent Fisher) Fisher information equals the negative expected curvature of the log likelihood; that is,

I(\theta) = -\mathbb{E}_{\theta}\left[ \frac{\partial^2}{\partial\theta^2} \log f_{\theta}(X) \right].

Proof: Just a lot of calculus. By the quotient and chain rules, we have:

\frac{\partial^2}{\partial\theta^2} \log f_{\theta}(X) = \frac{\partial}{\partial\theta} \frac{\frac{\partial}{\partial\theta} f_{\theta}(X)}{f_{\theta}(X)} = \frac{f_{\theta}(X) \frac{\partial^2}{\partial\theta^2} f_{\theta}(X) - \left(\frac{\partial}{\partial\theta} f_{\theta}(X)\right)^2}{ \left(f_{\theta}(X)\right)^2 } = \frac{\frac{\partial^2}{\partial\theta^2} f_{\theta}(X)}{f_{\theta}(X)} - s(X; \theta)^2.

The expectation of the second term of the RHS is $-I(\theta)$ , so it suffices to show the expectation of the first term is zero.

\mathbb{E}_{\theta}\left[\frac{\frac{\partial^2}{\partial\theta^2} f_{\theta}(X)}{f_{\theta}(X)}\right] = \int_{\mathcal{R}} f_{\theta}(x) \cdot \left[ \frac{\frac{\partial^2}{\partial\theta^2} f_{\theta}(x)}{f_{\theta}(x)} \right] \, \mathrm{d}x = \int_{\mathcal{R}} \left[\frac{\partial^2}{\partial\theta^2} f_{\theta}(x)\right] \, \mathrm{d}x = \frac{\partial^2}{\partial\theta^2} \left[\int_{\mathcal{R}} f_{\theta}(x) \, \mathrm{d}x\right] = \frac{\partial^2}{\partial\theta^2} [1] = 0.

A very similar argument as before, actually… $\blacksquare$

Finally, when we have multiple samples, we can talk about “total Fisher information”.

Definition. Given $n$ samples $\{X_i\}_{i = 1}^n$ , recall $\ell_n(\theta) := \sum_{i = 1}^n \log f_{\theta}(X_i)$ is the log likelihood. Then the total Fisher information gained from these samples is:

I_n(\theta) := -\mathbb{E}_{\theta}\left[\frac{\partial^2}{\partial\theta^2}\ell_n(\theta)\right] = n I(\theta).

These two expressions are equal because all the $X_i$ are sampled from the same distribution $\mathbb{P}_{\theta}$ , so applying linearity of expectation yields $n$ copies of $I(\theta)$ summed together.

§ MLE Asymptotic Normality

Now we can finally get to asymptotic normality of the MLE.

Theorem. (MLE Asymptotic Normality) Under mild regularity conditions, the MLE $\hat{\theta}_n$ has asymptotic variance $\frac{1}{I(\theta^*)}$ ; that is,

\sqrt{n}(\hat{\theta}_n - \theta^*) \rightsquigarrow \mathcal{N}\left(0, \frac{1}{I(\theta^*)}\right).

Proof: Recall the log likelihood $\ell_n(\theta) := \sum_{i = 1}^n \log f_{\theta}(X_i)$ . The definition of the MLE $\hat{\theta}_n$ is that it maximizes $\ell_n(\theta)$ , meaning:

\frac{\partial}{\partial\theta} \ell_n(\hat{\theta}_n) = \sum_{i = 1}^n s(X_i; \hat{\theta}_n) = 0.

Recall that we proved the consistency of $\hat{\theta}_n$ earlier, meaning $\hat{\theta}_n \convprob \theta^*$ . Thus, it makes sense to take the Taylor approximation:

0 = \sum_{i = 1}^n s(X_i; \hat{\theta}_n) \approx \sum_{i = 1}^n s(X_i; \theta^*) + (\hat{\theta}_n - \theta^*) \left[ \frac{\partial}{\partial\theta} \sum_{i = 1}^n s(X_i; \theta) \right]_{\theta = \theta^*}.

Moving terms around yields the approximation:

\sqrt{n}(\hat{\theta}_n - \theta^*) \approx - \left(\frac{\frac{1}{\sqrt{n}}\cdot \sum_{i = 1}^n s(X_i; \theta^*)}{ \frac{1}{n} \cdot \sum_{i = 1}^n \left[\frac{\partial^2}{\partial\theta^2} \log f_{\theta}(X_i) \right]_{\theta = \theta^*} }\right).

All that setup pays off here; it turns out the numerator and denominator of this fraction can be expressed very nicely.

By “Zero Mean Score”, the definition of Fisher information, and CLT, the numerator converges ( $\rightsquigarrow$ ) to $\mathcal{N}(0, I(\theta^*))$ .
By “Equivalent Fisher” and LLN, the denominator converges ( $\convprob$ ) to $-I(\theta^*)$ .

Thus, by Slutsky's lemma, we have:

\sqrt{n}(\hat{\theta}_n - \theta^*) \rightsquigarrow -\left(\frac{\mathcal{N}(0, I(\theta^*))}{-I(\theta^*)}\right) = \mathcal{N}\left(0, \frac{1}{I(\theta^*)}\right).

And that's it. $\blacksquare$

Knowing that the asymptotic variance is $\frac{1}{I(\theta^*)}$ is useful for constructing confidence intervals using the MLE $\hat{\theta}_n$ , especially when the variance of $\hat{\theta}$ would not otherwise be easy to compute directly.

Remark. For the model $\{\mathrm{Ber}(p) \mid p \in (0, 1)\}$ , the Fisher information is $I(p) = \frac{1}{p(1 - p)}$ . CYU: Why is this expected?

§ The Cramér-Rao Lower Bound

As the following theorem explains, the MLE is in some sense the best estimator possible, as long as it is unbiased.

Theorem. (Cramér-Rao Lower Bound) Let $\hat{\theta}_n$ be any unbiased estimator of $\theta$ . Then $\mathbb{V}_{\theta^*}[\hat{\theta}_n] \geq \frac{1}{nI(\theta^*)}$ .

Proof: The finish of the proof will use the following theorem.

Theorem (Cauchy-Schwarz). For any random variables $U$ and $V$ , we have $\Cov(U, V)^2 \leq \Var(U) \cdot \Var(V)$ .

Proof: The point is that $\Var(U - \lambda V) \geq 0$ for all $\lambda$ . The LHS expands to a quadratic in $\lambda$ , so the discriminant of this quadratic must be nonpositive. The nonpositivity of the discriminant simplifies to the above theorem. $\square$

We'll invoke the above theorem on $U := \hat{\theta}_n$ and $V := \sum_{i = 1}^n s(X_i; \theta^*)$ .

Lemma 1. We have $\Var_{\theta^*}(V) = nI(\theta^*)$ .

Proof: The $X_i$ are independent, so variances add, and then this follows straight from the definition of Fisher information. $\square$

Now for the hard part.

Lemma 2. We have $\Cov_{\theta^*}(U, V) = 1$ .

Proof: Note that $\mathbb{E}_{\theta^*}\left[ \sum_{i = 1}^n s(X_i; \theta^*) \right] = 0$ by “Zero Mean Score”. Also note that we're considering the joint random variable $(X_1, \dots, X_n)$ , so our integrals will need to use the joint PDFs $\prod_{i = 1}^n f_{\theta^*}(x_i)$ . That said, the rest is computation.
$\begin{align*}\Cov_{\theta^*}(U, V) & = \mathbb{E}_{\theta^*}\left[\hat{\theta} \cdot \sum_{i = 1}^n s(X_i; \theta^*)\right] - \mathbb{E}_{\theta^*}[\hat{\theta}] \cdot \mathbb{E}_{\theta^*}\left[\sum_{i = 1}^n s(X_i; \theta^*)\right] \\ & = \mathbb{E}_{\theta^*}\left[\hat{\theta} \cdot \sum_{i = 1}^n s(X_i; \theta^*)\right] \\ & = \int \cdots \int_{\mathcal{R}} \left(\prod_{i = 1}^n f_{\theta^*}(x_i)\right) \cdot \hat{\theta}(x_1, x_2, \dots, x_n) \cdot \sum_{i = 1}^n s(x_i; \theta^*) \, \mathrm{d}x_1 \, \mathrm{d}x_2 \, \cdots \, \mathrm{d}x_n \\ & = \int \cdots \int_{\mathcal{R}} \left(\prod_{i = 1}^n f_{\theta^*}(x_i)\right) \cdot \hat{\theta}(x_1, x_2, \dots, x_n) \cdot \frac{ \left[\frac{\partial}{\partial\theta} \prod_{i = 1}^n f_{\theta}(x_i)\right]_{\theta = \theta^*} }{\prod_{i = 1}^n f_{\theta^*}(x_i)} \, \mathrm{d}x_1 \, \mathrm{d}x_2 \, \cdots \, \mathrm{d}x_n \\ & = \frac{\partial}{\partial\theta} \left[ \int \cdots \int_{\mathcal{R}} \left(\prod_{i = 1}^n f_{\theta}(x_i)\right) \cdot \hat{\theta}(x_1, x_2, \dots, x_n) \, \mathrm{d}x_1 \, \mathrm{d}x_2 \, \cdots \, \mathrm{d}x_n \right]_{\theta = \theta^*}.\end{align*}$
Here's the finish: $\hat{\theta}$ is unbiased, so the integral above evaluates to just $\theta$ , meaning the covariance is just $\frac{\partial}{\partial\theta}[\theta]_{\theta = \theta^*} = 1$ . $\square$

Now just plug Lemma 1 and Lemma 2 into the Cauchy-Schwarz theorem and win. $\blacksquare$