A hypothesis test makes a decision using imperfect sample information. Two kinds of incorrect decisions are possible. Understanding their consequences helps explain significance levels, power, and why failing to reject a null hypothesis does not establish that it is true.
Separate truth from the test decision
A Type I error occurs when a true null hypothesis is rejected. A Type II error occurs when a false null hypothesis is not rejected. The definitions depend on the hypotheses, so write those first before describing an error in context.
| Actual situation | Reject the null | Fail to reject the null |
|---|---|---|
| Null is true | Type I error | Correct decision |
| Null is false | Correct detection | Type II error |
The table's rows describe reality, which is usually unknown. Its columns describe the test's decision, which is based on a random sample. A particular study cannot ordinarily identify whether its own decision is an error simply from its P-value.
Example: describe errors in a safety test
Suppose says a batch's defect proportion is 0.02, and says it exceeds 0.02. Rejecting the null triggers an expensive inspection.
A Type I error means concluding that the defect rate exceeds 2% when it is actually 2%, causing an unnecessary inspection. A Type II error means failing to detect a rate above 2%, allowing a worse-than-standard batch to pass without that inspection.
These consequences are different. The cost of an unnecessary inspection and the cost of missed defects help inform the choice of significance level and sample size. The names “Type I” and “Type II” do not rank the errors by seriousness.
Alpha controls false alarms under the null
The significance level is the probability of a Type I error under the specified null model for a properly calibrated test, or an upper bound in some discrete procedures. A 5% level means that repeated tests on data generated under the null would reject about 5% of the time.
It is not a 5% probability that this particular rejection is wrong. That would require additional information about how often the null is true and the behavior under alternatives.
Choose alpha before seeing the result. Moving the threshold after observing the P-value undermines the advertised long-run error control.
Power depends on a specific alternative
For a specified false parameter value, let be the probability of a Type II error. Power is the probability of rejecting the null when that particular alternative is true.
There is not generally one universal power value for every false null. A large departure from the null is easier to detect than a tiny departure. A power statement should name the effect size or true parameter value used in its calculation.
For the defect example, detecting a true rate of 10% is easier than detecting 2.5% with the same sample and decision rule.
Example: compute power from a rejection rule
Suppose a sample mean is normally distributed with known population standard deviation 10 and sample size 100. Test against at 5%. The sample mean's standard deviation is 1, and rejection occurs when
If the true mean is 53, then the power is Thus for this alternative.
If the true mean were 50.5, power would be much smaller because the rejection threshold is farther into that alternative distribution's upper tail. The rule remains fixed; the true distribution changes.
null
alternative
Normal sample-mean distributions with standard deviation 1 under null mean 50 and alternative mean 53. The rejection threshold is 51.645; the alternative right tail is shaded.
The rejection threshold stays fixed. Under the alternative mean 53, most sample means lie to its right, giving high power.
Describe this figure
Normal sample-mean distributions with standard deviation 1 under null mean 50 and alternative mean 53. The rejection threshold is 51.645; the alternative right tail is shaded.
Curves: null; alternative.
Lines: reject above.
How design choices affect power
Increasing sample size reduces standard error and generally increases power for a fixed effect and alpha. A larger true effect or less measurement variability also increases power. Increasing alpha makes rejection easier and increases power, but also raises the Type I error rate.
Lowering alpha without increasing sample size generally reduces power. You cannot simultaneously reduce both error risks merely by demanding stronger evidence; additional information, such as a larger or more precise sample, is usually needed.
Try it yourself
A test has power 0.85 against a specified alternative. What is the Type II error probability? If alpha changes from 0.05 to 0.01 with everything else fixed, what generally happens to power?
Answer: A smaller alpha makes rejection harder, so power generally decreases against the same alternative. The false-alarm risk falls, but missed detections become more likely.
What to remember
Type I means rejecting a true null; Type II means missing a false null. Alpha describes null-model false alarms, while power describes detection under a specified alternative.