Values affirmation replication at the University of Illinois

Values affirmation exercises have been implemented in many contexts to combat stereotype threat in students from marginalized populations; the exercises are intended to fortify students by prompting them to self-affirm their values in short writing activities. Within the physics education research community, the style of intervention was underlined by a positive result from the University of Colorado Boulder; researchers were able to use the intervention to minimize the achievement gap between men and women in an introductory physics course. These results inspired a replication experiment in two physics courses at the University of Illinois in Urbana-Champaign, and this article provides some history and context of these interventions, describes our specific implementation, and reveals that we were unable to reproduce the positive results despite thorough attention to the details of the replication. Our findings suggest that the values affirmation exercises are not understood at a level where they should be considered a positive intervention to help marginalized populations.

Figure 1

Adjusted exam scores for students in the Miyake et al. study, by gender and treatment condition [27]. Note that the vertical axis begins at 55%. Reprinted with permission from AAAS.

Figure 2

Average exam scores for men and women in treatment (values affirmation) and control conditions for the first exam in (a) Physics 100 and (b) Physics 212. Note that the vertical axis begins at 0.55.

Figure 3

Histogram of bootstrapped exam average ranges for men and women in treatment (values affirmation) and control conditions in Physics 100 and 212. In all of these plots, the control condition is light orange with diagonal hatching up and to the right (//) and the treatment condition is magenta with spots; dark orange with both patterns shows their overlap.

Figure 4

Histogram of bootstrapped difference in exam scores (treatment–control average) for women due to the affirmation activity, from (a) unadjusted scores and (b) adjusted scores in Physics 212. The Physics 100 histogram is shown in reddish purple with diagonal hatching up and to the right (//) and the Physics 212 histogram is bluish purple with diagonal hatching down and to the right (\\); dark purple with cross hatching shows their overlap.

Figure 5

Summary of effects within each gender and treatment. Points indicate mean value of treatment–control values and error bars are 95% confidence intervals. CO 1 refers to the first Colorado experiment [27] and CO 2 refers to the second implementation by Colorado [101]. The values for Colorado are adjusted scores, and their error bars are 95% confidence intervals, calculated from their reported standard errors. Note that within each experiment from Colorado, the exam and FMCE score effects are not independent; each pair of scores (exam and FMCE) is from the same group of students. The combined value for women’s gains was calculated with a random effects model.

Figure 6

Final grades of students by gender and treatment or control condition in the original Colorado implementation [27]. Reprinted with permission from AAAS.

Figure 7

Distributions of students’ final grades within each gender and treatment condition for (a) Physics 100 and (b) Physics 212. Each of these fractions is calculated out of the total number of students in each category ($\mathrm{gender}\times \mathrm{condition}$).

Figure 8

Fraction of men and women responding for each level of endorsement to the statement, “According to my own personal beliefs, I expect men to generally do better in physics than women.” Students selecting 1 strongly disagree with the statement (a rejection of the stereotype), while students selecting 5 strongly agree with the statement (an endorsement of the stereotype).

Figure 9

Students’ exam scores by treatment and gender group, also partitioned by high and low stereotype endorsement, for the first Colorado implementation [27]. Reprinted with permission from AAAS.

Figure 10

Average exam grades for men and women in each treatment group, based on their classification of being a high or low endorser of gender stereotype in physics.

Figure 11

Average exam grades for men and women in each treatment group, based on their classification of being a high or low endorser of gender stereotype in physics.

Figure 12

Histogram of bootstrapped exam average ranges in Physics 100 for men and women in treatment (values affirmation) and control conditions with low and high endorsement of gender stereotype in physics. In all of these plots, the control condition is light orange with diagonal hatching up and to the right (//) and the treatment condition is magenta with spots; dark orange with both patterns shows their overlap.

Figure 13

Histogram of bootstrapped exam average ranges in Physics 212 for men and women in treatment (values affirmation) and control conditions with low and high endorsement of gender stereotype in physics. In all of these plots, the control condition is light orange with diagonal hatching up and to the right (//) and the treatment condition is magenta with spots; dark orange with both patterns shows their overlap.

Figure 14

Histogram of bootstrapped exam gains for treatment participants over control participants, comparing low and high stereotype endorsement. These gains were calculated by subtracting the average exam scores for those in the control condition from exam scores for those in the treatment condition, meaning positive gains imply higher scores from the affirmation exercise while negative gains show lower scores in those who completed the affirmation. In these plots, high endorsers are plotted in red with cross hatching and low endorsers are plotted in light green with spots; dark green with both patterns shows their overlap.

Figure 15

Histogram of bootstrapped difference in exam scores (treatment–control average) for high stereotype endorsing women due to the affirmation activity, from (a) unadjusted scores and (b) adjusted scores in Physics 212. The Physics 100 histogram is shown in reddish purple with diagonal hatching up and to the right (//) and the Physics 212 histogram is bluish purple with diagonal hatching down and to the right (\\); dark purple with cross hatching shows their overlap.

Figure 16

Summary of effects within each gender and treatment, partitioned by stereotype endorsement. “Endorse” values refer to student averages of those who had high endorsement of the stereotype, while “Reject” values refer to low endorsement. Points indicate mean value of treatment–control average values and error bars are 95% confidence intervals. CO refers to the first Colorado experiment [27]. The values for Colorado are adjusted scores, and their error bars are 95% confidence intervals, calculated from their reported standard errors. Note that within Colorado’s experiment, the exam and FMCE score effects are not independent; the pair of scores (exam and FMCE) is from the same group of students.