Investigating institutional influence on graduate program admissions by modeling physics Graduate Record Examination cutoff scores

Despite limiting access to applicants from underrepresented racial and ethnic groups, the practice of using hard or soft Graduate Record Examination (GRE) cutoff scores in physics graduate program admissions is still a popular method for reducing the pool of applicants. The present study considers whether the undergraduate institutions of applicants have any influence on the admissions process by modeling a physics GRE cutoff score with application data from admissions offices of five universities. Two distinct approaches based on inferential and predictive modeling are conducted. While there is some disagreement regarding the relative importance between features, the two approaches largely agree that including institutional information significantly aids the analysis. Both models identify cases where the institutional effects are comparable to factors of known importance such as gender and undergraduate GPA. As the results are stable across many cutoff scores, we advocate against the practice of employing physics GRE cutoff scores in admissions.

Figure 1

A comparison of the P-GRE distribution between national data [22] and data used in this study. The analysis is primarily concerned with domestic applicants (yellow curve).

Figure 2

Estimated P-GRE distributions by racial and ethnic groups (number of applicants indicated in parenthesis). Note that the combined distribution normalizes the differences between the combined groups.

Figure 3

Summary of odds ratio significance levels of 19 independent models: Each P-GRE cutoff score (horizontal coordinate) denotes an independent maximum graduates logistic regression model that includes possible duplicates (see Sec. 3c). In each model, the fields of the statistically significant features are marked with a symbol indicating the Bonferroni-corrected significance level: circle indicates $\alpha =0.05$, triangle indicates $\alpha =0.01$, and diamond indicates $\alpha =0.001$. Blank fields indicate statistically insignificant results. The dotted separation lines categorize the features as applicant-related, institution-wide metrics and concerning Physics departments.

Figure 4

Overall performance of the conditional inference forest. The standard errors of the $K$-fold ($K=10$) estimates are indicated by the error bars. While the ABOVE class imbalance is very high for lower cutoffs, the accuracy standard errors are always above the imbalance level. The AUC score is mostly above 0.9, which Hosmer et al. categorizes as “outstanding” [31].

Figure 5

Conditional inference forest feature importance measures. The standard errors of the $K$-fold ($K=10$) estimates are indicated by the error bars. Features included in the CIF are listed in the legend in decreasing order of average (across all cutoffs) decrease in AUC upon removal.

Figure 6

Conditional inference forest feature elimination procedure. The standard errors of the $K$-fold ($K=10$) estimates are indicated by the error bars. The features are eliminated from left to right, where the named feature is currently the least important feature, and thus the next to be dropped from the model.

Figure 7

Performance comparison between conditional inference forests with all features, with two institutional features, and without institutional features (only U-GPA, gender, and race). The standard errors of the $K$-fold ($K=10$) estimates are indicated by the error bars. The significant improvement in performance by the simple addition of two institutional features suggests that the contribution from the institutional features is captured by a few features.