Development and validation of a conceptual survey instrument to evaluate students’ understanding of thermodynamics

We discuss the development and validation of a conceptual multiple-choice survey instrument called the survey of thermodynamic processes and first and second laws (STPFaSL) suitable for introductory physics courses. The survey instrument uses common student difficulties with these concepts as resources in that the incorrect answers to the multiple-choice questions were guided by them. After the development and validation of the survey instrument, the final version was administered at six different institutions. It was administered to introductory physics students in various traditionally taught calculus-based and algebra-based classes in paper-pencil format before and after traditional lecture-based instruction in relevant concepts. We also administered the survey instrument to upper-level undergraduates majoring in physics and Ph.D. students for benchmarking and for content validity and compared their performance with those of introductory students for whom the survey is intended. We find that although the survey instrument focuses on thermodynamics concepts covered in introductory courses, it is challenging even for advanced students. A comparison with the base line data on the validated survey instrument presented here can help instructors evaluate the effectiveness of innovative pedagogies designed to help students develop a solid grasp of these concepts.

Figure 1

A schematic diagram of the STPFaSL instrument development process.

Figure 2

PBC for each item. The solid line represents the mean value for all items.

Figure 3

Average percentage scores for students in algebra-based introductory physics courses on the STPFaSL instrument by topic before and after instruction (for the pretest, the number of students $N=218$, blue, and for the post-test, $N=382$, red). The blue and red horizontal lines show the averages on the entire survey instrument before and after instruction, respectively.

Figure 4

Average percentage scores for students in calculus-based introductory physics courses on the STPFaSL instrument by topic before and after instruction (for the pretest, the number of students $N=704$, blue, and for the post-test, $N=505$, red). The blue and red horizontal lines show the averages on the entire survey instrument before and after instruction, respectively.

Figure 5

Average performance on the STPFaSL instrument by topic for $N=147$ upper-level undergraduates and Ph.D. students. For completeness, error bars on each topic score (very small black vertical lines) indicate the sample error of the mean topic score, assuming each topic is independent. For comparing pairs of topics whose coverage on the STPFaSL instrument is minimally overlapping, e.g., comparison of performance on questions involving Irreversible and Reversible processes, an assumption of independence may be appropriate, but otherwise the topics and their errors should not be directly compared within a population. Another pair of topics for which there is minimal overlap involves the second law problems and problems requiring knowledge of the state variables. The blue horizontal line shows the average on the entire instrument.

Figure 6

Average percentage scores for first-year physics Ph.D. students individually and in pairs on the STPFaSL instrument by topic (for individual Ph.D. students, the number of students $N=45$, blue, and for the paired test, $N=21$, red). The blue and red horizontal lines show the averages for those groups on the entire survey instrument.