Policy recommendations from causal inference in physics education research

Sound educational policy recommendations require valid estimates of causal effects, but observational studies in physics education research sometimes have loosely specified causal hypotheses. The connections between the observational data and the explicit or implicit causal conclusions are sometimes misstated. The link between the causal conclusions reached and the policy recommendations made is also sometimes loose. Causal graphs are used to illustrate these issues in several papers from Physical Review Physics Education Research. For example, the core causal conclusion of one paper rests entirely on the choice of a causal direction although an unstated plausible alternative gives an exactly equal fit to the data.

Figure 1

The top graph is the DAG from Ref. [13], using abbreviated variable names. The bottom one is the Markov equivalent graph described in the text. The graphs were drawn using the online DAGitty tool.

Figure 2

This is the DAG form of the ADMG used in Ref. [14], with their bidirectional arrow shown instead as an explicit unmeasured U. The variable “Physics Career Choice,” connected only by an arrow from identity, is omitted for simplicity.

Figure 3

A DAG schematically representing the causal issues addressed in Ref. [25]. Variables shown by lightly filled ovals are unmeasured. The combined effects of social variables and gender are presumably strongly nonlinear, i.e., the effects of gender depend on social context including bias. Since it happens that in this sample the variable HS courses is approximately independent of gender [25], the arrow between them could be erased. The observational result reported is that no arrow is needed from gender to postscore. Other demographic variables have similar diagrams but with poorly measured HS courses.