Item-level gender fairness in the Force and Motion Conceptual Evaluation and the Conceptual Survey of Electricity and Magnetism

Gender gaps on the most widely used conceptual inventories created by physics education researchers have been extensively studied. Most of the research exploring the consistent gender gaps has been performed at the student level using the total evaluation score; less research has been performed examining these assessments at the item level and this research has been predominately restricted to the Force Concept Inventory (FCI). Many studies have identified subsets of FCI items as unfair to either men or women. An item is fair if men and women of equal ability in conceptual physics score equally on the item. This study explored the item-level gender fairness of the Force and Motion Conceptual Evaluation (FMCE) and the Conceptual Survey of Electricity and Magnetism (CSEM). Classical test theory and differential item functioning (DIF) analysis were employed to examine item fairness. Fairness was investigated with four large post-test samples, two for the FMCE ($n_1=3016$, $n_2=3719$) and two for the CSEM ($n_1=2014$, $n_2=2657$). Men and women performed significantly differently on the majority of FMCE items but with no more than a small effect size. There were fewer items in the CSEM where men and women performed differently. Using DIF analysis, which assumes that overall test score is an accurate measure of ability, only one item in the FMCE demonstrated large DIF in both samples with that item unfair to women. One additional item showed large DIF in a single sample, also unfair to women. Only one item in the CSEM demonstrated large DIF. The item was unfair to men but this result was not consistent across all samples. The number of large DIF items identified in both the FMCE and the CSEM was substantially smaller than the number of large DIF items identified in the FCI by previous studies.

Figure 1

CTT difficulty results for the FMCE. The top two panels are sample 1 (a) FMCE pretest and (b) FMCE post-test. The bottom two panels are sample 3A (c) FMCE pretest and (d) FMCE post-test. A line of slope 1 is drawn to allow comparison of male and female difficulty. Error bars represent 1 standard deviation in each direction.

Figure 2

FMCE post-test percentage vs FMCE pretest: (a) Sample 1 and (b) sample 3A. The number next to the data point is the number of students within each pretest range.

Figure 3

CTT difficulty results for the CSEM. The top two panels are sample 2 (a) CSEM pretest and (b) CSEM post-test. The bottom two panels are sample 3B (c) CSEM pretest and (d) CSEM post-test. A line of slope 1 is drawn to allow comparison of male and female difficulty. Error bars represent 1 standard deviation in each direction.

Figure 4

CSEM post-test percentage vs CSEM pretest: (a) Sample 2 and (b) sample 3B. The number next to the data point is the number of students within each pretest range.