Assessing class-wide consistency and randomness in responses to true or false questions administered online

We have developed simple data-mining algorithms to assess the consistency and the randomness of student responses to problems consisting of multiple true or false statements. In this paper we describe the algorithms and use them to analyze data from introductory physics courses. We investigate statements that emerge as outliers because the class has a preference for the incorrect answer and also those that emerge as outliers because the students are randomly changing their responses. These outliers are found to include several statements that are known in the literature to expose student misconceptions. Combining this fact with comments made by students and results of complementary assessments provides evidence that the tendency of a group of students to change their answer to a true or false statement or to remain consistent can serve as indicators of whether the class has understood the relevant concept. Our algorithms enable teachers to employ problems of the type described as a tool to identify specific aspects of a course that require improvement. They also enable researchers to employ such problems in experiments designed to probe aspects of students’ thought processes and behavior. Additionally, our results demonstrate that at least one category of research-inspired problems (ranking tasks) can be adapted to the linked true or false format and productively used as an assessment tool in an online setting.

Figure 1

An example of the type of problem discussed in this paper. Each problem consists of five true-false statements.

Figure 2

Cumulative probability distribution of the overall consistency score [$C_{\mathrm{total}}$ defined in Eq. (3)] by subsample. The total number of statements used to generate the distributions are 250 for UWP 1350 and 257 for MIT 8.011.

Figure 3

Cumulative probability distribution of the overall randomness score [$R_{\mathrm{total}}$ defined in Eq. (6)] by subsample. The total number of statements used to generate the distributions are 250 for UWP 1350 and 257 for MIT 8.011.

Figure 4

Overall randomness [$R_{\mathrm{total}}$ defined in Eq. (6)] vs overall consistency [$C_{\mathrm{total}}$ defined in Eq. (3)] for the 250 statements in the UWP 1350 subsample. The dashed lines indicate the strong randomness cut of 0.23 and the strong (incorrect) consistency cut of $-0.08$ discussed in Sec. 3. Points outlined with symbols are discussed in Sec. 5.

Figure 5

Overall randomness [$R_{\mathrm{total}}$ defined in Eq. (6)] vs overall consistency [$C_{\mathrm{total}}$ defined in Eq. (3)] for the 257 statements in the MIT 8.011 subsample. The dashed lines indicate the strong randomness cut of 0.23 and the strong (incorrect) consistency cut of $-0.08$ discussed in Sec. 3. Points outlined with symbols are discussed in Sec. 5.

Figure 6

Pictures from the problem labeled with circles (○) in Figs. 4, 5. The problem is adapted from Ref. 4.

Figure 7

Pictures from one of the problems labeled with upward-pointing triangles (▵) in Fig. 4. The horizontal component of the initial velocity is always $20\mathrm{m}/\mathrm{s}$ in this problem. The problem is adapted from Ref. 4.

Figure 8

Pictures from one of the problems labeled with upward-pointing triangles (▵) in Fig. 4. The vertical component of the initial velocity is always $20\mathrm{m}/\mathrm{s}$ in this problem. The problem is adapted from Ref. 4.

Figure 9

Cumulative probability distribution of the overall consistency score [$C_{\mathrm{total}}$ defined in Eq. (3)] for two distinct populations of students enrolled in UWP 2240. One population was not required to post comments to the problems while the other was required to post one comment per week. An identical set of 136 statements was used to create the distribution for each population.

Figure 10

Cumulative probability distribution of the overall randomness score [$R_{\mathrm{total}}$ defined in Eq. (6)] for two distinct populations of students enrolled in UWP 2240. One population was not required to post comments to the problems while the other was required to post one comment per week. An identical set of 136 statements was used to create the distribution for each population.

Figure 11

The eight projectile pairs used for the linked true or false problem described in the text. The figures were adapted from Ref. 13.

Figure 12

Overall randomness versus overall consistency for eight statements of the form “[the higher trajectory] will have a longer time of flight than [the lower trajectory]” that correspond to the eight projectile pairs shown in Fig. 11. This statement should be answered “true” in each case.

Figure 13

Overall randomness versus overall consistency for eight statements of the form “[both trajectories] will have equal times of flight” that correspond to the eight projectile pairs shown in Fig. 11. This statement should be answered “false” in each case.