Analyzing false positives of four questions in the Force Concept Inventory

In this study, we analyze the systematic error from false positives of the Force Concept Inventory (FCI). We compare the systematic errors of question 6 (Q.6), Q.7, and Q.16, for which clearly erroneous reasoning has been found, with Q.5, for which clearly erroneous reasoning has not been found. We determine whether or not a correct response to a given FCI question is a false positive using subquestions. In addition to the 30 original questions, subquestions were introduced for Q.5, Q.6, Q.7, and Q.16. This modified version of the FCI was administered to 1145 university students in Japan from 2015 to 2017. In this paper, we discuss our finding that the systematic errors of Q.6, Q.7, and Q.16 are much larger than that of Q.5 for students with mid-level FCI scores. Furthermore, we find that, averaged over the data sample, the sum of the false positives from Q.5, Q.6, Q.7, and Q.16 is about 10% of the FCI score of a midlevel student.

Figure 1

Question 7 (Q.7) of the FCI [1]. [Question statement] A steel ball is attached to a string and is swung in a circular path in a horizontal plane as illustrated in the accompanying figure. At the point $P$ indicated in the figure, the string suddenly breaks near the ball. If these events are observed from directly above as in the figure, which path would the ball most closely follow after the string breaks?

Figure 2

Subquestions of the FCI Q.7. [Question statement] Which of the eight choices best represents the direction of the following variables, just after the string breaks? If you think a variable is zero or not at all horizontal, write 9; 1. force acting on the ball; 2. velocity of the ball.

Figure 3

Distribution of raw scores on the FCI ($N=1110$). The values on the horizontal axis designate the highest score in a given bin.

Figure 4

$P_{\mathrm{raw}}^i$ (vertical axis, white circle) and $P_{\mathrm{true}}^i$ (vertical axis, black triangle) of Q.5, 6, 7, and 16 for each group of students with a given $S_{\mathrm{raw}}$ (horizontal axis). The error bars ($P_{\mathrm{raw}}^i$, gray with caps; $P_{\mathrm{true}}^i$, black without caps) indicate the Clopper-Pearson 95% confidence intervals of the dependent variable.

Figure 5

$P_{\mathrm{fp}}^i$ (vertical axis) of Q.5, 6, 7, and 16 for each group of students with a given $S_{\mathrm{raw}}$ (horizontal axis). The error bars indicate the Clopper-Pearson 95% confidence intervals of the dependent variable.

Figure 6

The ratio of the sum of the systematic errors of Q.5, Q.6, Q.7, and Q.16 to a given raw score $S_{\mathrm{raw}}$ for each group of students with that $S_{\mathrm{raw}}$. The error bars indicate the square-and-add intervals of the dependent variable for each question.