Uncovering the hidden meaning of cross-curriculum comparison results on the Force Concept Inventory

In a recent study, Caballero and colleagues conducted a large-scale evaluation using the Force Concept Inventory (FCI) to compare student learning outcomes between two introductory physics curricula: the Matter and Interactions (M&I) mechanics course and a pedagogically-reformed-traditional-content (PRTC) mechanics course. Using a conventional approach to detect between-curriculum differences, the researchers found that students in the PRTC course outperformed their M&I peers on nearly all the FCI items. However, this conventional approach does not control for the levels of student mastery of the tested concepts, and hence falls short of revealing deep information about the key differences between the two curricula. To this end, we use a stratum-specific method, common for detecting differential item functioning, to analyze a subset of the empirical data in the study of Caballero et al. By comparing students of equal mastery levels measured by the FCI (stratum-specific analysis), we find that the M&I students in fact demonstrate an advantage on the questions regarding Newton’s first and second laws. In addition, the previously reported underperformance of the M&I students on kinematics diminishes in our stratum-specific analysis. Results from this study show better alignment with students’ exposure to relevant topics in their respective courses. Both our findings and methodological approach provide useful implications for research on effective curriculum evaluation.

Figure 1

The organizing structures of traditional mechanics courses and M&I modern mechanics. (a) The traditional canon of linearly sequenced force and motion concepts; arrows indicate the topical progression in traditional courses. (b) The hierarchical organization of force and motion concepts in M&I modern mechanics; arrows indicate the topical encapsulation from an overarching principle to specific topics.

Figure 2

Conventional analysis of differences in student performance on the individual FCI items between the PRTC course and the M&I course: $\mathrm{\Delta }g=g_{\text{PRTC}}-g_{\mathrm{M}\&\mathrm{I}}$. Here, each $g$ represents an item normalized gain.

Figure 3

MH D-DIF analysis. (a) MH D-DIF for pretest, (b) MH D-DIF for post-test. A positive MH D-DIF indicates an item in favor of the PRTC course; a negative MH D-DIF indicates an item in favor of the M&I course. The numerical value on top of each bar shows the corresponding significance level ($p$ value). Those with both $|\mathrm{MH}\mathrm{D}\text{−}\mathrm{DIF}|>1$ and $p\le 0.01$ are highlighted with color.

Figure 4

Item characteristic curves for the FCI items with a significant DIF. The circular and triangular marks represent empirical observations of correct portions for the M&I and PRTC courses, respectively. The solid and dashed curves are regression lines for the M&I and PRTC courses, respectively.

Figure 5

MH D-DIF values and significance levels for the FCI items. Items containing significant DIF are highlighted.