Why this matters
You can never survey everyone, so you survey some people and reason back. That reasoning is always uncertain, and the whole discipline here is about measuring the uncertainty rather than hiding it. The most common misuse of statistics is not bad arithmetic; it is treating a significant result as a certain one.
The idea
Statistical inference means inferring the population's properties from a sample. It splits in two: estimation puts a number on a property (how tall is the average student?), and hypothesis testing judges a claim (is this method better?).
Point estimation gives one value: sample mean 168 cm, so estimate the population mean at 168 cm. Easy to read, but it shows none of its own uncertainty. Interval estimation gives a range, say 166 cm to 170 cm. That range is the confidence interval, and its confidence level says how often such intervals contain the true value over repeated surveys.
Read the level carefully. A 95 percent confidence level means that if you repeated the same survey method 100 times, about 95 of the resulting intervals would contain the true population mean. It is a statement about the procedure, not about this one interval.
Hypothesis testing sets up two hypotheses: the null hypothesis H0 (no difference, no relationship) and the alternative hypothesis H1 (there is one).
You assume the thing you want to deny, which feels backwards until you see why. Showing directly that "the new method works" is hard. Showing that "if it made no difference, data like ours would be very unlikely" is tractable, and it gets you to the same place.
The procedure: set H0 and H1; choose a significance level (0.05 or 0.01 are usual); compute the p-value; compare.
| Condition | Judgment |
|---|---|
| p-value < significance level | Reject H0, adopt H1 (statistically significant) |
| p-value >= significance level | Do not reject H0 (cannot be called significant) |
Which test depends on the data. A t-test asks whether two groups' means differ. A chi-squared test asks whether two categorical variables are related.
Two errors are always possible. A Type I error rejects a true null (a false alarm); a Type II error keeps a false one (a miss). Lowering the significance level reduces Type I errors and tends to increase Type II errors. They trade off; you cannot minimise both.
Finally, three things a p-value is not. It is not the probability that the null hypothesis is true. A small p-value does not mean the effect is large. And the threshold is a convention chosen by the field, not a boundary between true and false.
Picture it
flowchart TD
A["Set H0 (no difference) and H1"] --> B["Choose significance level, e.g. 0.05"]
B --> C[Compute the p-value from the data]
C --> D{p-value < significance level?}
D -- yes --> E["Reject H0: statistically significant"]
D -- no --> F["Do not reject H0"]
E --> G{But is the effect large enough to matter?}Worked example
Two website designs are tested on 100,000 visitors, split evenly. The old one is clicked by 0.4 percent, the new one by 0.5 percent. The test returns a small p-value.
The statistics are behaving correctly. With 50,000 people per side, a difference of 0.1 percentage points is far larger than random variation would comfortably produce, so it is statistically significant, and "these designs perform identically" is not a good explanation of the data.
Now the separate question: does it matter? The new design gets one extra click per thousand visitors. Whether that is worth rebuilding the page depends on what a click is worth and what the change costs, and no statistical test answers that.
This is the trap the sample size sets. Significance says "this difference is probably not noise". It says nothing about size. Make the sample large enough and almost any real difference, however tiny, becomes significant. That is exactly why statistically significant and practically important must be judged separately.
Your turn
Try the practice questions for this lesson. Judge statements about estimation and testing, match each term to its definition, and decide whether a significant result is automatically an important one.
Take it further
A club claims its new online quiz tool raises average test scores. Should you trust its summary, or test the claim properly?
- Frame the hypotheses. The club reports its group averaged 74 against 70 for a control group of the same size. Write the null and the alternative hypothesis.
- Judge the evidence. Suppose the test returns a p-value of 0.03 against a significance level of 0.05. Say what that does and does not justify concluding.
- Decide. Recommend whether the school should adopt the tool, separating whether the difference is statistically significant from whether it is large enough to matter.
Four marks on a weekly test may be significant and still not be worth changing how the school teaches.
Recap
- Point estimation gives a value; interval estimation gives a confidence interval and shows the uncertainty.
- Testing assumes the null hypothesis and asks how unlikely the data would then be.
- Reject H0 when the p-value is below the significance level; a t-test compares means, a chi-squared test compares categories.
- Type I and Type II errors trade off against each other.
- Significant is not the same as important, and a p-value is not the probability that H0 is true.