MIT 18.650 — Lecture 17

$p$ -Values

Note that the behavior of a test $\Psi: \{\mathrm{data}\} \to \{0, 1\}$ is a function of the level $\alpha$ that is set.

Definition. Given a test $\Psi$ and some inputted data $\{X_i\}_{i = 1}^n$ , the $p$ -value of $\Psi$ is the smallest level $\alpha$ at which $\Psi$ rejects $H_0$ .

Suppose we perform a test and get a $p$ -value of…

… less than $1\%$ . Then we say we have very strong evidence against $H_0$ .
… between $1\%$ and $5\%$ . Then we say we have strong evidence against $H_0$ .
… between $5\%$ and $10\%$ . Then we say we have weak evidence against $H_0$ .
… greater than $10\%$ . Then we say we have little or no evidence against $H_0$ .

Remark. In most settings, we can simplify the definition of a $p$ -value to just be “the probability of observing results as or more extreme than what we did, assuming that $H_0$ was true”.

Example. Suppose we observe $\bar{X}_n = 33.4$ with $n = 164$ and $\hat{\sigma} = 12$ , and we are doing a test on $H_0: \mu \leq 30$ and $H_1: \mu > 30$ . Using the Wald test $\Psi$ , our test statistic is:

T_n = \frac{\sqrt{n}}{\hat{\sigma}}(\bar{X}_n - 30) = \frac{\sqrt{164}}{12}(33.4 - 30) \approx 3.63.

The test $\Psi$ rejects $H_0$ if $3.63 \geq z_{\alpha}$ . The least $\alpha$ for which this occurs is $1 - \Phi(3.63) \approx 0.01\%$ , so the $p$ -value is $0.01\%$ , and we have very strong evidence against $H_0$ .