Development and validation of a conceptual multiple-choice survey instrument to assess student understanding of introductory thermodynamics

We discuss the development and validation of the long version of a conceptual multiple-choice survey instrument called the Survey of Thermodynamic Processes and First and Second Laws-Long suitable for introductory physics courses. This version of the survey instrument is a longer version of the original shorter version developed and validated earlier. The 19 contexts including the exact wording of all of the problem situations posed in the two versions of the survey instrument are identical and the difference between the long and short versions of the instrument is only in the multiple-choice options. In particular, in the longer version of the survey instrument, there are no alternative conceptions explicitly embedded in the four multiple-choice options students choose from and the questions asked in a given context in one item of the shorter survey instrument were split into several items focusing, e.g., on different thermodynamic variables. After the development and validation of the longer version of the survey instrument, the final version was administered in 12 different in-person classes (four different institutions) in which students answered the questions in-class on paper scantron forms with the instructor as the proctor and 12 different in-person classes (five different institutions) in which students answered the questions online on Qualtrics within a two-hour period. This longer version of the survey instrument was administered to introductory physics students in various traditionally taught calculus-based and algebra-based classes before and after traditional lecture-based instruction in relevant concepts. It was also administered to upper-level undergraduates majoring in physics and Ph.D. students taught traditionally for bench marking purposes and for concurrent validity, which involved comparing advanced students’ performance with those of introductory students for whom the survey is intended. Similar to the shorter version, we find that although the longer version of the survey instrument focuses on thermodynamics concepts covered in introductory courses, it is challenging even for advanced students. A comparison with the baseline data on the longer version of the validated survey instrument presented here can help instructors evaluate the effectiveness of innovative pedagogies designed to help students develop a solid grasp of these concepts.

Figure 1

(a). Topics for items 1–26 [topics for items 27–78 are in (b) and (c)]. Along with other physics faculty members who teach thermodynamics, we identified topics involved in each item used for one possible method of solving the problem (note that several other approaches can be used to solve different problems). An “R” indicates that a concept is required to solve the problem using the cited solution method or the information is provided in the question. (b) Topics for items 27–53. (c) Topics for items 54–78.

Figure 2

Point biserial coefficient for each item for in-class administration.

Figure 3

Point biserial coefficient for each item for online administration in Qualtrics.