MIT 18.650 — Lecture 9

Introducing the MLE

§ Types of Parameters, Identifiability

Definition. In a statistical model, the parameters we care about are called parameters of interest, and those we don't are called nuisance parameters. A parameter $\theta$ is identifiable if distinct values of $\theta$ necessarily yield distinct probability distributions.

Example. In the model $\{\mathcal{N}(\mu, \sigma^2) \mid \mu \in \mathbb{R}, \sigma \in \mathbb{R}^+\}$ , we might say $\mu$ is a parameter of interest and $\sigma$ is a nuisance parameter.

Example. The response to a drug dosage $D$ is modeled by $\mathcal{N}\left(E_0 + \frac{D \times E_{\text{max}}}{D + \mathrm{ED}_{50}},\ \sigma^2\right)$ , for some parameter $\theta = (\sigma^2, E_0, E_{\text{max}}, \mathrm{ED}_{50})$ . In this case, $\theta$ is not identifiable.

§ Defining the Maximum Likelihood Estimator (MLE)

Consider some (i.i.d.) random variable samples $X_1, \dots, X_n \sim \mathbb{P}$ and a parameter $\theta$ . Furthermore, some new notation:

Notation. Given a parameter $\theta$ , let its true, population value be denoted by $\theta^*$ .

If the parameter $\theta$ can be expressed using $\mathbb{E}[\dots]$ only, it's not hard to define an okay estimator.

\text{(ex:)} ~~~~~~ \theta := \frac{\mu}{\sigma} = \frac{\mathbb{E}[X_i]}{(\mathbb{E}[X_i^2] - \mathbb{E}[X_i]^2)^{1/2}} ~ \implies ~ \hat{\theta}_n := \frac{\frac{1}{n}\sum_{i = 1}^n X_i}{\left( \frac{1}{n}\sum_{i = 1}^n X_i^2 - \left(\frac{1}{n}\sum_{i = 1}^n X_i\right)^2 \right)^{1/2}}.

More smartly, though, we should think to use the maximum likelihood estimator (MLE).

Definition. Let $f_{\theta}(x)$ denote the PDF corresponding to the distribution $\mathbb{P}_{\theta}$ , and consider some samples $X_1, \dots, X_n \sim \mathbb{P}_{\theta^*}$ .

The likelihood function is defined by $L_n(\theta) := \prod_{i = 1}^n f_{\theta}(X_i)$ .
The log likelihood function is just $\ell_n(\theta) := \log L_n(\theta) = \sum_{i = 1}^n \log f_{\theta}(X_i)$ .
The maximum likelihood estimator (MLE) $\hat{\theta}_n$ is the value of $\theta$ for which $L_n(\theta)$ (or $\ell_n(\theta)$ ) is maximized.

In other words, the MLE $\hat{\theta}_n$ is the value for the parameter $\theta$ whose distribution $\mathbb{P}_{\hat{\theta}_n}$ is most “compatible” with the samples taken from $\mathbb{P}_{\theta^*}$ , which would suggest that $\mathbb{P}_{\hat{\theta}_n}$ might be a good representation of $\mathbb{P}_{\theta^*}$ .

Remark. Reason through the following and check that they make sense.

For the model $\{\mathrm{Ber}(\theta) \mid \theta \in [0, 1]\}$ , the MLE is $\hat{\theta} := \frac{1}{n} \sum_{i = 1}^n X_i$ .
For the model $\{\mathcal{N}(\theta, 1) \mid \theta \in \mathbb{R}\}$ , the MLE is $\hat{\theta} := \frac{1}{n} \sum_{i = 1}^n X_i$ .
For the model $\{\mathrm{Unif}([0, \theta]) \mid \theta > 0\}$ , the MLE is $\hat{\theta} := \max(X_1, \dots, X_n)$ .

Note that $L_n(\theta)$ and $\ell_n(\theta)$ are random functions, since their behaviors are determined by random samples from $\mathbb{P}_{\theta^*}$ . If one seeks a likelihood function that is instead determined by $\theta^*$ itself, one may opt for…

Definition. The population log likelihood function is $\ell(\theta) := \mathbb{E}_{\theta^*}[\log f_{\theta}(X)]$ . (The $\mathbb{E}_{\theta^*}$ notation indicates that $X \sim \mathbb{P}_{\theta^*}$ .)