Using machine learning to identify the most at-risk students in physics classes

Machine learning algorithms have recently been used to predict students’ performance in an introductory physics class. The prediction model classified students as those likely to receive an A or B or students likely to receive a grade of C, D, F or withdraw from the class. Early prediction could better allow the direction of educational interventions and the allocation of educational resources. However, the performance metrics used in that study become unreliable when used to classify whether a student would receive an A, B, or C (the ABC outcome) or if they would receive a D, F or withdraw (W) from the class (the DFW outcome) because the outcome is substantially unbalanced with between 10% to 20% of the students receiving a D, F, or W. This work presents techniques to adjust the prediction models and alternate model performance metrics more appropriate for unbalanced outcome variables. These techniques were applied to three samples drawn from introductory mechanics classes at two institutions ($N=7184$, 1683, and 926). Applying the same methods as the earlier study produced a classifier that was very inaccurate, classifying only 16% of the DFW cases correctly; tuning the model increased the DFW classification accuracy to 43%. Using a combination of institutional and in-class data improved DFW accuracy to 53% by the second week of class. As in the prior study, demographic variables such as gender, underrepresented minority status, first-generation college student status, and low socioeconomic status were not important variables in the final prediction models.

Figure 1

Variable importance of the optimized model predicting DFW for sample 2 using institutional data and data available in class at the end of week 2. Error bars are one standard deviation in length.

Figure 2

Predicted probability of earning an A, B, or C for sample 1 disaggregated by the actual grade received in the class. The figure plots the probability density of each outcome. The order of the peaks in the lower figure from left to right is W, F, D, C.

Figure 3

Decision tree for predicting the DFW outcome for sample 2 using institutional data and data available in class at the end of week 2.