Peer assessment of student-produced mechanics lab report videos

We examine changes in students’ rating behavior during a semester-long sequence of peer evaluation laboratory exercises in an introductory mechanics course. We perform a quantitative analysis of the ratings given by students to peers’ physics lab reports, and conduct interviews with students. We find that peers persistently assign higher ratings to lab reports than do experts, that peers begin the semester by giving high ratings most frequently and end the semester with frequent middle ratings, and that peers go through the semester without much change in the frequency of low ratings. We then use student interviews to develop a model for student engagement with peer assessment. This model is based on two competing influences which appear to shape peer evaluation behavior: a strong disinclination to give poor ratings with a complementary preference to give high ratings when in doubt, and an attempt to develop an expertlike criticality when assessing peers’ work.

Figure 1

Peer assessments of physics lab report videos taken from the beginning (pre) and end (post) of the semester show that the mean peer grade becomes significantly lower and closer to the mean expert grade by 5 points, about half a letter grade on a standard 100-point grading scale ($p\ll 0.01$, Cohen’s $d=-0.45$). A two-sample KS test shows distributions of expert grades do not significantly change ($p=0.86$). Means are determined from 565 peer grades of a sample of 204 physics lab report videos selected from the first and last labs of the semester, along with expert grades for each of those videos. Error bars show 95% confidence intervals.

Figure 2

Distributions of differences between peer and expert grades show that peer grades become more accurate, but not more precise relative to expert grades. Two distinct grading schemes are considered: (a) In grading scheme one, many peer videos are each graded by few peers. (b) In grading scheme two, 20 training-phase videos are each graded by all peers. The mean grade differences shift significantly toward zero over the course of the semester, indicating peer grades increase in accuracy. The widths of the distributions do not change significantly, indicating peer grades do not become more precise. See Table 1 for $N$ values and full tabulation of results.

Figure 3

Expert rating distributions (a) do not change significantly from pre to post, while peer rating distributions (b) change dramatically. Five ratings on a five-point poor to excellent scale determine the numerical value of each peer or expert grade. Excellent changes from being among the most frequent peer ratings to among the least frequent, and good ratings become substantially more frequent. Figure shows 2820 peer ratings of a sample of 204 physics lab report videos selected from the first and last labs of the semester, along with expert ratings for each of those videos. Error bars show 95% confidence intervals.

Figure 4

Peer ratings for physics-content rubric items (2, 3, and 4) show a greater pre-to-post decline than peer ratings for nonphysics-content rubric items (1 and 5). Consistent among all rubric items was a large decline in peer excellent ratings. Expert ratings were relatively stable across rubric items, with the exception of item 1 (organization and structure). Figure shows 2820 peer ratings of a sample of 204 physics lab report videos selected from the first and last labs of the semester, along with expert ratings for each of those videos, sorted by the rubric item to which each rating was assigned. Error bars show 95% confidence intervals. See Table 2 for a tabulation of comparisons.

Figure 5

Peer rating distributions in two different classroom contexts—on campus and online—show the same pre (a) and post (b) trends. The on-campus offering was conducted in 2014, and comprised 338 enrolled Georgia Tech students. The online offering was conducted in 2016, and comprised 21 enrolled Georgia Tech students. Peer ratings of 10 videos evaluated by students in both course offerings are shown. The similarity of the distributions indicates good overall repeatability, and suggests that the students in both classroom contexts are generally comparable with respect to their peer evaluation behavior. Error bars show 95% confidence intervals.