MIT 18.650 — Lecture 15

§ Hypothesis Testing

Definition. A test is a function $\Psi: \{\mathrm{data}\} \to \{0, 1\}$. Equivalently, a test is an estimator of a binary value.

• The rejection region of a test $\Psi$ is the set $\Psi^{-1}(1)$ of all data for which $\Psi(\mathrm{data}) = 1$.

• A test statistic is a function that summarizes the data and is sufficient for computing a test $\Psi$.

In other words, a test is a process of reading in data and answering a question about the population based on that data. Given a parametric model $\{\mathbb{P}_{\theta} \mid \theta \in \Theta\}$, the question a test will answer must look like “Does $\theta \in \Theta_0$ or $\theta \in \Theta_1$?”, where $\Theta = \Theta_0 \sqcup \Theta_1$.

Definition. The null hypothesis $H_0$ is the claim that $\theta \in \Theta_0$, and the alternative $H_1$ is the claim that $\theta \in \Theta_1$.

The distinction is that the null hypothesis is the “status quo” (e.g. “the drug does not outperform the placebo”).

§ Error Types

Of course, a test $\Psi$ only reads in sample data, so the answer that it gives won't always be correct.

Definition. A Type I error occurs if $H_0$ is true but $\Psi = 1$, and a Type II error occurs if $H_1$ is true but $\Psi = 0$.

Type I is conventionally the more serious error: it is worse to wrongly declare “the drug outperforms the placebo”. Note that the probability of a Type I / Type II error depends on the true population parameter $\theta$.

Definition. The size of a test $\Psi$ is $\mathrm{size}(\Psi) := \max_{\theta \in \Theta_0} \mathbb{P}_{\theta}(\Psi = 1)$: the maximum possible probability of a Type I error. A test $\Psi$ is said to have level $\alpha$ if $\mathrm{size}(\Psi) \leq \alpha$.

Definition. The power of a test $\Psi$ is the function $\beta(\theta) = \mathbb{P}_{\theta}(\Psi = 1)$.

In the above, the size of $\Psi$ is the $y$-intercept $\alpha$. A perfect test would have $\beta(\theta)$ be the Heaviside step function.