Online test administration results in students selecting more responses to multiple-choice-multiple-response items

We developed the Physics Inventory of Quantitative Literacy (PIQL) to assess students’ quantitative reasoning in introductory physics contexts. The PIQL includes several “multiple-choice-multiple-response” (MCMR) items (i.e., multiple-choice questions for which more than one response may be selected) as well as traditional single-response multiple-choice items. In this paper, we discuss differences in performance on MCMR items that seems to result from differences in administration method (paper versus online). In particular, we find a tendency for “clickiness” in online administration: students choose more responses to MCMR items when taking the electronic version of the assessment. Student performance on single-response multiple-choice items was not affected by administration method. These results suggest that MCMR items may provide a unique opportunity to probe differences in online and on-paper administration of low-stakes assessments.

Figure 1

PIQL MCMR item that probes understanding of the negative sign in the context of mechanical work. The correct answers are d and g.

Figure 2

Average number of answer choices selected by Institution 1 students for the six MCMR PIQL items. Items 1, 2, 5, and 6 have one correct answer choice; item 3 has three correct answer choices, and item 4 has two.

Figure 3

Fraction of student responses that include 1–6 answer choices for each item. Items 1 and 6 have six answer choices; 2, 3, and 5 have five answer choices; and 4 has seven answer choices. (No student chose all seven of item 4’s answer choices.).

Figure 4

Average number of answer choices selected online by Institution 1 students for the six MCMR PIQL items with items grouped or ungrouped.

Figure 5

Average number of answer choices selected on paper by Institutions 1 and 2 students for the six MCMR PIQL items with items grouped or ungrouped.

Figure 6

Item difficulty for the PIQL’s four MCMR items with a single correct answer choice.

Figure 7

Fraction of student responses in each category of our four-level scoring scheme for MCMR items.