Rubric-based holistic review represents a change from traditional graduate admissions approaches in physics

Rubric-based admissions are claimed to help make the graduate admissions process more equitable, possibly helping to address the historical and ongoing inequities in the U.S. physics graduate school admissions process that have often excluded applicants from minoritized races, ethnicities, genders, and backgrounds. Yet, no studies have examined whether rubric-based admissions methods represent a fundamental change in the admissions process or simply represent a new tool that achieves the same outcome. To address that, we developed supervised machine learning models of graduate admissions data collected from our department over a seven-year period. During the first four years, our department used a traditional admission process and switched to a rubric-based process for the following three years, allowing us to compare which parts of the applications were driving admissions decisions. We find that faculty focused on applicants’ physics GRE scores and grade-point averages when making admissions decisions before the implementation of the rubric. While we were able to develop a sufficiently good model whose results we could trust for the data before the implementation of the rubric, we were unable to do so for the data collected after the implementation of the rubric, despite multiple modifications to the algorithms and data such as implementing Tomek Links. Our inability to model the second dataset despite being able to model the first combined with model comparison analyses suggests that rubric-based admission does change the underlying process. These results suggest that rubric-based holistic review is a method that could make the graduate admission process in physics more equitable.

Figure 1

Plot A shows Fig. 2 of Young and Caballero [24] with the Tomek Links marked. Filled square points represent Tomek Links. Plot B shows the same plot after the Tomek Links have been removed. Data for which either the GPA or physics GRE score is missing are not plotted.

Figure 2

Averaged AUC feature importances over 30 trials for dataset 0. Physics GRE score, Quantitative GRE score, and undergraduate GPA, appearing in orange, were the factors found to be meaningful and hence predictive of being admitted for dataset 0.

Figure 3

Comparison of the testing AUC when (a) dataset 0 is used to train the model and (b) when dataset 1a is used to train the model. Training refers to the training AUC for the model. All error bars are 1 standard error. Results were averaged over 30 trials. When training on dataset 0, we are able to produce an acceptable AUC when testing on dataset 0 but not dataset 1a. When training on dataset 1a, we are not able to produce an acceptable AUC in either case.

Figure 4

Comparison of the testing accuracy when (a) dataset 0 is used to train the model and (b) when dataset 1a is used to train the model. The null accuracy is shown in cyan with the shorter in height error bars. All error bars are 1 standard error. Results were averaged over 30 trials. When training on dataset 0, we are able to produce an acceptable accuracy when testing on dataset 0 but not dataset 1a. When training on dataset 1a, we could not produce an acceptable accuracy in either case.

Figure 5

Raincloud plots showing the distribution of physics GRE scores of all applicants before and after the implementation of the rubric, only admitted applicants, and only nonadmitted applicants. Only the distributions of physics GRE scores for nonadmitted applicants before and after the implementation of the rubric were found to be statistically different.

Figure 6

Raincloud plots showing the distribution of grade-point averages of all applicants before and after the implementation of the rubric, only admitted applicants, and only nonadmitted applicants. None of the distributions of GPAs for applicants before and after the implementation of the rubric were found to be statistically different.

Figure 7

Raincloud plots showing the distribution of quantitative GRE scores of all applicants before and after the implementation of the rubric, only admitted applicants, and only nonadmitted applicants. None of the distributions of quantitative GRE scores for applicants before and after the implementation of the rubric were found to be statistically different.