Validation and analysis of the coupled multiple response Colorado upper-division electrostatics diagnostic

Standardized conceptual assessment represents a widely used tool for educational researchers interested in student learning within the standard undergraduate physics curriculum. For example, these assessments are often used to measure student learning across educational contexts and instructional strategies. However, to support the large-scale implementation often required for cross-institutional testing, it is necessary for these instruments to have question formats that facilitate easy grading. Previously, we created a multiple-response version of an existing, validated, upper-division electrostatics diagnostic with the goal of increasing the instrument’s potential for large-scale implementation. Here, we report on the validity and reliability of this new version as an independent instrument. These findings establish the validity of the multiple-response version as measured by multiple test statistics including item difficulty, item discrimination, and internal consistency. Moreover, we demonstrate that the majority of student responses to the new version are internally consistent even when they are incorrect and provide an example of how the new format can be used to gain insight into student difficulties with specific content in electrostatics.

Figure 1

A sample item from the CMR CUE. The prompt has been paraphrased; see Ref. [8] for the full prompt and a detailed discussion of question development and scoring.

Figure 2

Distributions of scores ($N=421$) on the CMR CUE from 15 courses at the institutions described in Table 1. These data did not pass a statistical test for normality (Anderson-Darling test, $p=0.03$).

Figure 3

Average scores for each item on the CMR CUE ($N=421$). Error bars represent a 95% confidence interval (double the standard error on the mean). Score distributions for each individual item are not necessarily normally distributed.

Figure 4

Item-test correlations (as measured by Pearson’s $r$) for each of the items on the CMR CUE. For $N=421$, any correlation greater than 0.09 is significant at the $p<0.05$ level [20]; thus, item-test correlations are statistically significant for all items. The conventional cutoff for an acceptable correlation (0.2) is marked as a bold line.

Figure 5

(a) Method selections for $N=421$ students on Q5 (Fig. 1). (b) Reasoning selections for the subset of students who selected each of the three most common methods: direct integration ($N=24$), Gauss’s law ($N=176$), and superposition ($N=191$). Number of responses for each reasoning element is proportional to the area of the circle.

Figure 6

Fraction of students coded as “consistent” (at least one consistent reasoning element and no inconsistent ones), “inconsistent” (any inconsistent reasoning elements), or “neither” (neither consistent nor inconsistent reasoning elements) on each of the questions that have consistency checks. Note that there are no consistency checks on questions 8, 11, 14, and 15, but there are two possible consistency checks in Q12: between subparts ii and iii, and between subparts iii and iv.