Rubric-based holistic review: A promising route to equitable graduate admissions in physics

As systematic inequities in higher education and society have been brought to the forefront, graduate programs are interested in increasing the diversity of their applicants and enrollees. Yet, structures in place to evaluate applicants may not support such aims. One potential solution to support those aims is rubric-based holistic review. Starting in 2018, our physics department implemented a rubric-based holistic review process for all applicants to our graduate program. The rubric assessed applicants on 18 metrics covering their grades, test scores, research experiences, noncognitive competencies, and fit with the program. We then compared faculty’s ratings of applicants by admission status, sex, and undergraduate program over a three-year period. We find that the rubric scores show statistically significant differences between admitted and nonadmitted students as hoped. We also find that differences in rubric scores based on sex or undergraduate program reflected known systematic inequities such as applicants from smaller and less prestigious undergraduate universities scoring lower on the physics GRE and women performing more volunteer work in academia. Our results then suggest rubric-based holistic review as a possible route to making graduate admissions in physics more equitable.

Figure 1

Faculty ratings of domestic applicants on 18 constructs. In the plot, a larger, darker circle means that more applicants are in that bin. While many applicants are in each level of the academic preparation and test score constructs, few applicants are in the “low” bin of the research, noncognitive skills, and program fit constructs.

Figure 2

Faculty ratings of domestic applicants on 18 constructs split by whether the applicant was admitted. Ratings that are statistically different between the two plots are marked on the right side of the plot, with a corrected $p$ value $<0.001$ represented by “***,” $<0.01$ by “**,” and $<0.05$ by “*.” The distribution of ratings of all constructs is statistically different for admitted applicants compared to nonadmitted applicants. Overall, most admitted applicants were rated “high” while most nonadmitted applicants were rated “medium”.

Figure 3

Faculty ratings of domestic applicants (admitted and nonadmitted) on 18 constructs split by whether the applicant was male or female. Ratings that are statistically different between the two plots are marked on the right side of the plot, with a corrected $p$ value $<0.001$ represented by ***, $<0.01$ by **, and $<0.05$ by *. Only three of the constructs showed statistical differences between males and females: physics GRE score where males scored higher and community contributions and diversity contributions where females scored higher.

Figure 4

Faculty ratings of domestic applicants (admitted and nonadmitted) on 18 constructs split by whether the applicant attended a more selective or less selective undergraduate university. Ratings that are statistically different between the two plots are marked on the right side of the plot, with a corrected $p$ value $<0.001$ represented by ***, $<0.01$ by **, and $<0.05$ by *. Only the general GRE and physics GRE scores showed differences.

Figure 5

Faculty ratings of domestic applicants (admitted and nonadmitted) on 18 constructs split by whether the applicant attended a university with a larger or smaller physics program. Ratings that are statistically different between the two plots are marked on the right side of the plot, with a corrected $p$ value $<0.001$ represented by ***, $<0.01$ by **, and $<0.05$ by *. Only the physics GRE score and conscientiousness showed differences between the groups of applicants, with the latter dependent on how larger physics program is defined.