S2.4 - Hypothesis tests

A statistical investigation begins by defining exactly what is being studied. The population is the complete set of units of interest; a census seeks data from every population unit, while a sample survey collects data from only a subset.

To study the fill mass of all cans produced during one shift, the population is every can from that shift, one can is a sampling unit, and the production list or numbered stream may provide the frame. Measuring 100 selected cans is a sample survey; measuring every can is a census.

A statistic is a quantity calculated entirely from sample observations. It may estimate or test a population feature, but its formula cannot contain an unknown population parameter. Thus $\bar X$, the sample range and the number of successes are statistics; $(X_1-\mu)/\sigma$ is not when $\mu$ and $\sigma$ are unknown.

If the same random-sampling procedure were repeated with a fixed sample size, the statistic would usually change. Its sampling distribution lists every possible value of the statistic together with its probability under the stated population model.

Suppose two independent Bernoulli observations are drawn from a population with success probability $0.5$, and let $T$ be the sample proportion of successes. The four equally likely ordered samples give$$T=0,\ \frac12,\ 1$$with probabilities$$P(T=0)=\frac14,\quad P\left(T=\frac12\right)=\frac12,\quad P(T=1)=\frac14.$$This probability model is the sampling distribution of $T$, not the distribution of a single observation.

A test statistic is a chosen statistic whose sampling distribution is known under the null hypothesis. That distribution lets an observed sample value be judged as ordinary or unusually extreme.

A hypothesis test asks whether sample evidence is sufficiently unusual under a stated model to justify rejecting that model in favour of a specific alternative. It can therefore refine a mathematical model, but it cannot prove either hypothesis.

State both hypotheses before using the data. Assume H0, identify the sampling distribution of a suitable test statistic, and calculate the probability of the observed value or values at least as supportive of H1. If this p-value is no greater than alpha—or the statistic lies in the precomputed critical region—reject H0; otherwise do not reject H0.

A historical breakdown rate is $8$ per week and a refurbishment is claimed to have changed it. Use$$H_0:\lambda=8,\qquad H_1:\lambda\ne8.$$If the observed count is outside the two-tailed critical region, do not reject $H_0$: there is insufficient evidence that the mean breakdown rate changed.

A critical region is the set of test-statistic values that cause rejection of the null hypothesis. It is chosen from the sampling distribution under H0 so that its total probability—the actual significance level—does not exceed the stated level and is as close as the discrete distribution permits.

For an upper-tailed test with $J\sim Po(9)$ under $H_0$, tables give$$P(J\le13)=0.9261,\qquad P(J\le14)=0.9585.$$Hence$$P(J\ge14)=0.0739>0.05,\qquad P(J\ge15)=0.0415\le0.05.$$The 5% critical region is therefore $J\ge15$. An observed value of 15 or more is significant at the 5% level.

Because a discrete tail changes in jumps, the actual significance is often below the nominal level. Moving the boundary one integer further inward here would make the rejection probability too large; moving it outward would be valid but unnecessarily conservative.

The wording of the alternative hypothesis fixes the tail structure before the sample is inspected. A directional claim uses one tail; a claim of any change uses both tails.

For a one-tailed test, the critical-region probability is placed in the stated tail. For a two-tailed test at nominal level alpha, choose lower and upper regions with probabilities as close as practical to alpha/2 while keeping their combined probability no greater than alpha. Discreteness may prevent equal tails or an exact total.

Testing whether a historical proportion $0.35$ has changed requires $H_0:p=0.35$ and $H_1:p\ne0.35$, so both unusually few and unusually many successes oppose $H_0$. Testing whether it has fallen uses $H_1:p<0.35$ and only the lower tail.

For a Binomial test, the parameter is a population success probability p; for a Poisson test, it is the mean event rate lambda for a stated exposure. Under H0, that exact parameter determines the sampling distribution used for the test.

Use this order: (1) define the parameter and state H0 and H1; (2) translate the observed count and direction into the correct tail or tails; (3) calculate an exact tail probability from tables/formulae or construct a critical region; (4) compare with alpha; (5) reject or do not reject H0 and give a contextual conclusion.

Suppose 8 of 40 items fail and the historical failure proportion is $0.35$. To test whether it is lower, use$$H_0:p=0.35,\qquad H_1:p<0.35,\qquad X\sim B(40,0.35).$$The lower-tail probability is$$P(X\le8)=0.0303<0.05.$$Reject $H_0$: there is sufficient evidence at the 5% level that the failure proportion is lower than $0.35$.

Use an approximation only when suitable, preserving the null model: a rare-event Binomial may use $Po(np_0)$, while a sufficiently spread Binomial or Poisson count may use a matched Normal distribution with continuity correction. For example, six weeks at a null Poisson rate of 6 per week gives total mean 36 and may be approximated by $N(36,36)$.

For a discrete two-tailed test, construct both rejection tails and add their null probabilities to obtain the actual significance. Do not automatically double one observed-tail probability unless that rule is justified by the chosen test; discrete tails are often unequal.