The meaning of "p"

p = Probability of obtaining test-statistic this extreme, given that H0 is true.

Inference Rule: Reject H0 if test-statistic is "large" enough.

THEREFORE: If we choose to reject H0 when p < 0.05, then the chance of having a False Alarm is no more than 0.05 (1 time in 20).

When p < 0.05, we commonly say that the effect (in the case of the t-test, the difference between the groups on the dependent variable) is statistically significant.

True state of affairs H0 H0: Null Hypothesis ("no effect") HA: Alternative Hypothesis ("some effect") Correct Non-Rejection [of H0] Miss (Type II Error) False Alarm (Type I Error) Correct Rejection [of H0]

Note: It should be clear that p is defined relative to only half of the table of possible outcomes: when H0 is actually true. By concentrating on p, our primary concern is avoiding false alarms. The other side of the table, where H0 is actually false, concerns our ability to avoid missing an actual effect, i.e. the power of the test.