Beyond normalized gain: Improved comparison of physics educational outcomes

This study proposes methods of reporting results of physics conceptual evaluations that more fully characterize the range of outcomes experienced by students with differing levels of prior preparation, allowing for more meaningful comparison of the outcomes of educational interventions within and across institutions. Factors leading to variation in post-test scores on the Force and Motion Conceptual Evaluation (FMCE) across different instructors, semesters, and course models in a sample collected in introductory calculus-based mechanics at a large, eastern land-grant university were examined. The sample was collected over nine years and contains a total of $N=4409$ matched pretest and post-test records. The data showed a systematic semester-by-semester variation in both pretest scores and ACT or SAT mathematics percentile scores. Neither the normalized gain nor Cohen’s $d$ removed the semester-to-semester variation observed in post-test scores. The local average curve plotting post-test scores against pretest scores, which we call a conceptual growth curve, allowed for the characterization of outcomes for students with different pretest scores. Regression models were used to produce an approximation to this curve. By using either the full curve or a mathematical approximation developed through linear regression, the post-test score that would be observed if a class enrolled students with a given level of prior preparation measured by pretest scores can be predicted. This predicted post-test score can then be used to calculate the predicted normalized gain if desired. These methods rely on using the natural variation of incoming student preparation at one institution to predict how a class would perform if it enrolled students with different prior preparation. The study presents an example of converting the outcomes at an institution with a weakly prepared student population to the outcomes which would have been observed if the course enrolled a more prepared student population; converting the outcomes for a different student population dramatically changed the interpretation of how the class studied was functioning.

Figure 1

The average ACT/SAT mathematics percentile (ACTM%), FMCE pretest, post-test, normalized gain, and Cohen’s $d$ per semester.

Figure 2

(a) The normalized gain plotted against FMCE pretest percentage. (b) The normalized gain plotted against ACT/SAT mathematics percentile (ACTM%). Each point represents the average of an individual course lecture section. The line is the regression line using the full dataset.

Figure 3

FMCE post-test scores vs pretest scores. The local average is plotted in black with its 95% confidence interval. The Hake Model for the normalized gain is plotted in green. The background is a heat map showing the density of the data at each point.

Figure 4

Conceptual growth curve comparing LA instruction (black) with coordinated learning CL instruction (orange) including the 95% confidence interval.

Figure 5

FMCE post-test scores vs pretest scores. The local average is plotted in black. The Hake model for the normalized gain is plotted in green. The linear model is plotted in blue. The quadratic model is plotted in orange.

Figure 6

Predicted FMCE post-test percentage vs observed FMCE post-test percentage by class section. The line has slope one. Classes above the line outperformed predictions; classes below the line under performed predictions.