Framework for evaluating statistical models in physics education research

Across the field of education research there has been an increased focus on the development, critique, and evaluation of statistical methods and data usage due to recently created, very large datasets and machine learning techniques. In physics education research (PER), this increased focus has recently been shown through the 2019 Physical Review PER Focused Collection examining quantitative methods in PER. Quantitative PER has provided strong arguments for reforming courses by including interactive engagement, demonstrated that students often move away from scientistlike views due to science education, and has injected robust assessment into the physics classroom via concept inventories. The work presented here examines the impact that machine learning may have on physics education research, presents a framework for the entire process including data management, model evaluation, and results communication, and demonstrates the utility of this framework through the analysis of two types of survey data.

Figure 1

A framework for building and evaluating statistical models in physics education research.

Figure 2

Distribution of pre- and postscores for FCI and FMCE courses from the LASSO dataset ($N=6688$). In both the pre- and postscore cases, the distribution of scores are right skewed normally distributed. Data come from multiple institutions across multiple courses ($N_{\text{courses}}=90$).

Figure 3

A comparison of the out-of-sample predicted distributions of post-test scores from the LASSO dataset. Curves represent the kernel density estimates of the predicted post-test scores. The gradient boosted model is closer to the true distribution than the ordinary least squares model and the mixed effects model.

Figure 4

Residuals plots for the LASSO study. For the OLS model and the xgboost model, the residuals are normally distributed with the majority being within $\pm 10$ points of the true value (see Appendix Fig. 6). The mixed effects model does not produce normally distributed residuals.

Figure 5

Receiver operator characteristic curves for the E-CLASS models predicting whether a student is in an introductory course or a BFY course. The more area under a curve, the better its performance. Overall, the boosted model ($\mathrm{AUC}=0.798$) performs approximately 2% better than the maximum likelihood model ($\mathrm{AUC}=0.773$) and considerably better than the mixed effects model ($\mathrm{AUC}=0.581$). However, the overall scores are still below an excellent discrimination ($\mathrm{AUC}>0.8$) as identified by Hosmer Jr et al. [68]. In this figure, the curves represent the average ROC curve for all 5 cross validation folds.

Figure 6

Residual distributions for the LASSO study.