6.5 Hypothesis tests

Calculate a null tail probability (p-value) or compare the statistic with a pre-set rejection region. Reject $H_0$ when the p-value is at most $\alpha$ or the statistic lies in the rejection region; otherwise do not reject $H_0$.

A two-sided p-value of 0.04 gives sufficient evidence against $H_0$ at 5% but not at 1%. The final statement must name the contextual claim in $H_1$, such as evidence that a mean has changed.

To test whether a coin has probability greater than $0.5$ of heads, use $H_0:p=0.5$, $H_1:p>0.5$. If 15 heads occur in 20 tosses, the direct p-value is $$P(X\ge15\mid X\sim B(20,0.5))=\sum_{r=15}^{20}\binom{20}{r}(0.5)^{20}.$$ Compare this value with the stated significance level.

For a suitable binomial null model use $N(np_0,np_0(1-p_0))$; for a suitable Poisson null model use $N(\lambda_0,\lambda_0)$. Apply continuity correction to the observed/critical integer boundary before standardising.

For a two-tailed discrete test, construct both extreme regions under $H_0$ so their combined null probability does not exceed the significance level. Discreteness may make the actual significance smaller than the nominal level.

A normal population has known $\sigma=10$. Test $H_0:\mu=50$ against $H_1:\mu>50$ using $n=100$ and $\bar x=51.8$. Then $SE=1$ and $z=1.8$, giving upper-tail p-value $P(Z\ge1.8)\approx0.0359$. Reject at 5% and conclude there is evidence that the population mean exceeds 50.

The same $z=1.8$ would not reject a two-sided test at 5%, because the two-sided 5% critical values are approximately $\pm1.96$. Always formulate $H_1$ before using the data.

Type I is a false positive: evidence is declared when the null state is actually true. Type II is a false negative: the test fails to detect a specified departure from the null.

For a fixed test rule, the Type I probability is evaluated under the null parameter. The Type II probability depends on which alternative parameter value is actually true; it is not one universal number.

Suppose $X_1,\ldots,X_{25}$ are normal with known $\sigma=10$, and a 5% upper-tail test of $H_0:\mu=50$ rejects when $\bar X>53.29$ (since $50+1.645(10/5)=53.29$). Under $H_0$, $$P(\text{Type I})=P(\bar X>53.29\mid\mu=50)\approx0.05.$$

If the true mean is instead 54, the same fixed rule makes a Type II error when $\bar X\le53.29$. Hence $$P(\text{Type II at }\mu=54)=P\left(Z\le\frac{53.29-54}{2}\right)=P(Z\le-0.355)\approx0.361.$$

For a binomial rule ‘reject $H_0:p=p_0$ when $X\ge c$’, Type I is $P_{p_0}(X\ge c)$ and Type II at $p=p_1$ is $P_{p_1}(X<c)$. For a Poisson test, use the same regions with $Po(\lambda_0)$ and the stated alternative $Po(\lambda_1)$. Evaluate these discrete probabilities directly when required.