Validity of peer grading using Calibrated Peer Review in a guided-inquiry, conceptual physics course

Constructing and evaluating explanations are important science practices, but in large classes it can be difficult to effectively engage students in these practices and provide feedback. Peer review and grading are scalable instructional approaches that address these concerns, but which raise questions about the validity of the peer grading process. Calibrated Peer Review (CPR) is a web-based system that scaffolds peer evaluation through a “calibration” process where students evaluate sample responses and receive feedback on their evaluations before evaluating their peers. Guided by an activity theory framework, we developed, implemented, and evaluated CPR-based tasks in guided-inquiry, conceptual physics courses for future teachers and general education students. The tasks were developed through iterative testing and revision. Effective tasks had specific and directed prompts and evaluation instructions. Using these tasks, over 350 students at three universities constructed explanations or analyzed physical phenomena, and evaluated their peers’ work. By independently assessing students’ responses, we evaluated the CPR calibration process and compared students’ peer reviews with expert evaluations. On the tasks analyzed, peer scores were equivalent to our independent evaluations. On a written explanation item included on the final exam, students in the courses using CPR outperformed students in similar courses using traditional writing assignments without a peer evaluation element. Our research demonstrates that CPR can be an effective way to explicitly include the science practices of constructing and evaluating explanations into large classes without placing a significant burden on the instructor.

Figure 1

Schematic of the Calibrated Peer Review process. CPR uses peer review with a training, or calibration, stage to prepare students for reviewing their peers’ work. Students cannot proceed between stages until a deadline passes (indicated by dashed lines), but students can work within the stage at their own pace (indicated by solid lines). When appropriate, the system provides background material, calibration texts, evaluation questions, and other students’ texts.

Figure 2

The elements of the activity system and their interrelations.

Figure 3

Histograms of the difference between the average weighted text score and expert score on the magnetism ($N=313$, upper) and Newton’s second law ($N=312$, lower) CPR tasks.

Figure 4

Research scores plotted versus average weighted peer scores for the magnetism ($N=313$, upper) and Newton’s second law ($N=312$, lower) CPR tasks.

Figure 5

Average score on the final exam explanation item for LEP ($N=334$) and PET ($N=324$) students. Error bars are SEM.

Figure 6

Histogram of post-test written explanation scores for LEP ($N=334$) and PET ($N=324$) students.

Figure 7

Content assessment average percentage normalized gain for LEP ($N=326$) and PET ($N=309$) students ($p=0.15$; $t=1.456$ $df=633$).