Validation of a coupled, multiple response assessment for upper-division thermal physics

Currently there are no assessment instruments available for upper-division thermal physics, though several introductory assessments are currently available. Notably missing from these introductory assessment are items targeting statistical mechanics. This leaves a gap in the content that can be assessed by upper-division thermal physics faculty. In this paper, we present a new, upper-division thermal physics assessment that explicitly addresses both classical thermodynamics and statistical mechanics: The Upper-level Statistical and Thermodynamics Evaluation for Physics (U-STEP). We piloted the U-STEP in multiple classes across various institutions during three semesters, and collected over 600 student responses in total. Here, we present multiple measures of validity and reliability for the U-STEP. We utilized classical test theory to determine item difficulties and discriminations, and differential item functioning to identify potential bias in assessment items that can manifest as performance gaps between different genders and races. The completed U-STEP will be the first upper-division thermal physics assessment available, and one of the first standardized physics assessments to explicitly address identification of possible item bias during the development process.

Figure 1

A CMR item included on the U-STEP addressing the concept of work (item 1). The item is composed of a multiple choice (MC) prompt, followed by a prompt for reasoning. The MC portion of the item asks only for direction of energy flow due to work, as opposed to the sign of work, to accommodate for different sign conventions used across different classes (i.e., based on defining work as done on or by the system). Note that “not determinable” was included here to account for the possibility that, for example, a student believes that the fact that this gas is nonideal means we cannot determine work from a P-V diagram.

Figure 2

Scoring scheme (left) and application examples (right) for item 1 of the U-STEP (Fig. 1). Multiple choice (MC) options (A–D) are in the top row of the scoring scheme, while the leftmost column lists multiple-response (MR, reasoning) options. Students select one MC answer and as many MR options as they desire in order to support their response. All other entries within the table are scores assigned to each response. The * indicates the correct MC response. See Supplemental Material [50] to see other items and their scoring schemes.

Figure 3

Item 4 of the U-STEP: (a) The figure presented with the item in Spring 2020 and Fall 2020 and (b) the figure presented in Spring 2021. When the item used (a), a large fraction of students chose C for (ii), which aligns with the longest column being the most probable. After the figure changed from (a) to (b), more students selected D for (ii), increasing both the item difficulty and discrimination drastically (see Table 2).

Figure 4

Scree plot from EFA for the U-STEP showing a clear elbow after the first factor. A horizontal line is shown for an eigenvalue of 1, as it is suggested to only keep factors with eigenvalues greater than 1.