You want to see if the means of two groups are different. You compare the means statistically and get a $p$ value of 0.05 when testing at a significance level of 0.07. What is the conclusion?

1. You reject the null hypothesis.
   Thus you cannot reject that the two population means are the same.
2. You fail to reject the null hypothesis.
   Thus you cannot reject that the two population means are the same.
3. You reject the null hypothesis.
   Thus you reject that the two population means are the same.
4. You fail to reject the null hypothesis.
   Thus you reject that the two population means are the same.
5. Help!!

What is a $p$ value anyway?

The $p$ value is the probability of having obtained a result at least as extreme as the one found with our sample if the null hypothesis were true. --- Kirkwood & Sterne

IF the null hypothesis is true
AND all the other assumptions about the model are also true
THEN the $p$ value expresses the probability of observing something as extreme as what you have in your sample.

What is a $p$ value anyway?

The $p$ value is the probability of having obtained a result at least as extreme as the one found with our sample if the null hypothesis were true. --- Kirkwood & Sterne

IF the null hypothesis is true
AND all the other assumptions about the model are also true
THEN the $p$ value expresses the probability of observing something as extreme as what you have in your sample.

Roughly: the $p$ value is a number that measures how surprised you are.

The epistemology of science

From ancient times ...

... to recent times

What are the problems with p values?

They try to answer the "wrong" question

A researcher typically wants to know if the hypothesis holds:

$$P(H | D)$$

but the p value computes

$$P(D \text{ "or more extreme" } | H)$$

The p value is used in the wrong way

Typically used as a decision rule:

$$p \text{ value} \left\{\begin{array}{ll}<0.05 & \text{reject} - \text{"significant"} \\ \geq 0.05 & \text{not reject} - \text{"not significant" or "no association"} \end{array} \right.$$

• Arbitrary threshold for continuous scale
• Significant does not mean clinically relevant
• Non-significance does not mean that $H_0$ is true - only that there was insufficient evidence to reject it ("absence of evidence is not evidence of absence").

The p value contains two types of information

Unrealistic null hypothesis

Compare two treatments with effects $\mu_1$ and $\mu_2$

$$H_0 : \mu_1 = \mu_2$$

When do we really believe that the effects of two treatments are exactly the same?

Hard to believe in most public health or social science research.

Use confidence intervals

The CI is defined as the values of $H_0$ that are not rejected.

Fully defined from (infinitely many) p values

... interpretation of the CI

Bayes factors

Lower the significance level

Bayesian analysis

Let's put things into perspective

Publication bias

Are p values bad?

• No

• Medicine / public health has moved forward in leaps and bounds in the last 100 years.

• "Assessing the Statistical Analyses Used in Basic and Applied Social Psychology After Their p-Value Ban" (2019)

• The function of significance tests is to prevent you from making a fool of yourself, and not to make unpublishable results publishable

The sceptical p value

Held (2019): The sceptical p value. Focus on statistical evaluation of replication studies.

Recommendations

Embrace uncertainty!

Know your tools!

• Report effect sizes and CIs (and perhaps p values)
• Put as much energy into discussing clinical relevance as statistical results.
• Abandon dichotomizing and "statistically significant"
• Never conclude: "no difference" eller "no effect"

Who said this?

Some people hate the very name of statistics, but I find them full of beauty and interest. Whenever they are not brutalized, but delicately handled by the higher methods, and are warily interpreted, their power of dealing with complicated phenomena is extraordinary. They are the only tools by which an opening can be cut through the formidable thicket of difficulties that bars the path of those who pursue the Science of man.

- Francis Galton (1894)

References