Replicating analyses of item response curves using data from the Force and Motion Conceptual Evaluation

Ishimoto, Davenport, and Wittmann have previously reported analyses of data from student responses to the Force and Motion Conceptual Evaluation (FMCE), in which they used item response curves (IRCs) to make claims about American and Japanese students’ relative likelihood to choose certain incorrect responses to some questions. We have used an independent dataset of over 6,500 American students’ responses to the FMCE to generate IRCs to test their claims. Converting the IRCs to vectors, we used dot product analysis to compare each response item quantitatively. For most questions, our analyses are consistent with Ishimoto, Davenport, and Wittmann, with some results suggesting more minor differences between American and Japanese students than previously reported. We also highlight the pedagogical advantages of using IRCs to determine the differences in response patterns for different populations to better understand student thinking prior to instruction.

Figure 1

Item response curves for 6584 American students (the RSW dataset) for item 17 on the FMCE. $\mathrm{NR}=\text{no response}$. The error bars on E (the correct answer) and B (the most commonly chosen incorrect answer) are the 95% confidence intervals of 10 000 bootstrapped sample datasets based on the RSW data described in Sec. 3c. For visual clarity, specific plot markers and error bars are omitted for less frequently chosen answers.

Figure 2

The fractional score distributions on the FMCE are shown for three separate datasets: 1531 Japanese students (IDWJ) and 2347 American students (IDWA) used in Ishimoto, Davenport, and Wittmann [30]; 6584 American students (RSW) obtained from PhysPort and other sources [7, 15, 16, 37].

Figure 3

The IRC dot products for all three pairs of populations shown in Figs. 5, 6, and 7. The IRCs of the American students (RSW and IDWA) are quite similar, and thus the IRC dot product values of the Japanese students (IDWJ) with either set of American students (RSW or IDWA) are similar for each item. IRC dot products are also shown for each population compared to a simulated set of random responses.

Figure 4

Distributions of the IRC dot product values for each of the comparisons shown in Fig. 3. The RSW-IDWA comparison has a very narrow distribution near unity. The RSW-IDWJ and IDWJ-IDWA distributions are very similar and lower than RSW-IDWA. Each comparison to a simulated dataset of random responses has a distribution that is noticeably lower than any of the comparisons between real datasets.

Figure 5

The IRC dot product for each item on the FMCE is shown in blue for the IDW American (IDWA) and Japanese (IDWJ) datasets [30]. The black error bars represent the randomized trial confidence intervals described in Sec. 3b3.

Figure 6

The IRC dot product for each item on the FMCE is shown in green for our American dataset (RSW) and the IDW American dataset (IDWA). The green error bars around the data points represent the IRC dot product confidence intervals described in Sec. 3c. The black error bars represent the randomized trial confidence intervals described in Sec. 3b3. The vertical range of (a) matches Figs. 5 and 7. Plot (b) contains the same information as (a), but a smaller vertical range is chosen to allow better visualization.

Figure 7

The IRC dot product for each item on the FMCE is shown in red for our American dataset (RSW) and the IDW Japanese dataset (IDWJ). The red error bars around the data points represent the IRC dot product confidence intervals described in Sec. 3c. The black error bars represent the randomized trial confidence intervals described in Sec. 3b3.

Figure 8

Item 21 IRCs: the solid lines show the IRCs for the RSW dataset, and the dashed lines show the IRCs for the IDWJ dataset. $\mathrm{NR}=\text{no response}$. The error bars on E, F, and H RSW IRCs are the 95% confidence intervals of 10 000 bootstrapped sample datasets based on RSW data described in Sec. 3c.

Figure 9

Item 32 IRCs: the solid lines show the IRCs for the RSW dataset, and the dashed lines show the IRCs for the IDWJ dataset. $\mathrm{NR}=\text{no response}$. The error bars on B, E, and F RSW IRCs are the 95% confidence intervals of 10 000 bootstrapped sample datasets based on RSW data described in Sec. 3c.

Figure 10

Item 11 IRCs: the solid lines show the IRCs for the RSW dataset, and the dashed lines show the IRCs for the IDWJ dataset. $\mathrm{NR}=\text{no response}$. The error bars on A and G RSW IRCs are the 95% confidence intervals of 10 000 bootstrapped sample datasets based on RSW data described in Sec. 3c.