Critical issues in statistical causal inference for observational physics education research

Recent critiques of physics education research (PER) studies have revoiced the critical issues when drawing causal inferences from observational data where no intervention is present. In response to a call for a “causal reasoning primer” in PER, this paper discusses some of the fundamental issues in statistical causal inference. In reviewing these issues, we discuss well-established causal inference methods commonly applied in other fields and discuss their application to PER. Using simulated data sets, we illustrate (i) why analysis for causal inference should control for confounders but not control for mediators and colliders and (ii) that multiple proposed causal models can fit a highly correlated dataset. Finally, we discuss how these causal inference methods can be used to represent and explain existing issues in quantitative PER. Throughout, we discuss a central issue in observational studies: A good quantitative model fit for a proposed causal model is not sufficient to support that proposed model over alternative models. To address this issue, we propose an explicit role for observational studies in PER that draw statistical causal inferences: Proposing future intervention studies and predicting their outcomes. Mirroring the way that theory can motivate experiments in physics, observational studies in PER can predict the causal effects of interventions, and future intervention studies can test those predictions directly.

Figure 1

Fundamental causal structures: (a) chain—where $Y$ acts as a mediator, (b) fork—where $Y$ acts as a confounder, and (c) collider—where $Y$ acts as a collider.

Figure 2

Causal diagram illustrating the relationship between academic GPA, college scholarship, and athletic talent.

Figure 3

Causal diagram depicting the relationships among variables $X$, $Y$, and $Z$, where $X$ causes $Z$, and $X$ and $Z$ jointly cause $Y$ with corresponding causal coefficients indicated on the arrows.

Figure 4

(a) Causal diagram illustrating the relationship between $X$ and $Y$, where $X$ causes $Y$. $C_1$, $C_2$, and $C_3$ are common causes of both $X$ and $Y$. The labels on the edges represent the causal coefficients. (b) A randomized experiment (or intervention) on $X$ breaks the causal dependence of confounders on $X$.

Figure 5

Six acyclic causal models that fit the same set of correlations between the variables in Table 3.

Figure 6

A proposed causal diagram representing the causal structure of how Pre E-CLASS scores, major, instruction, and URM status predict the Post E-CLASS scores.

Figure 7

Causal diagram illustrating the relationships between smoking, low birth weight, alternative low birth weight causes, and mortality as seen in the study [57].

Figure 8

A proposed causal diagram illustrating the causal structure of how concept inventory (CI) and attitude survey (AS) predict final course grade (G).

Figure 9

This is a reproduction of Fig. 2 of Ref. [78]. Causal diagram of the cross-lagged path model between self-efficacy and academic performance at time 1 and time 2.

Figure 10

Models, as shown in Ref. [82], represent theorized relationships between gender and physics identity mediated through three motivational variables: self-efficacy (SE), interest (INT), and perceived recognition (PR). (a) In model 1, the three mediating variables are noncausally associated with one another. (b–d) In models 2–4, there are causal associations between the three motivational variables: one of the three motivational variables is a common cause of the others, and another variable is a common effect of the others.