How the introduction of self-assessment rubrics helped students and teachers in a project laboratory course

Scientific abilities rubrics developed at Rutgers University are an assessment and self-assessment tool. They consist of tables, each representing a broad scientific ability (for example, “to collect and analyze data”), the listing of subabilities (for example, “independent and dependent variables are identified”) and the criteria to assess to which degree the subability has been developed. We introduced scientific abilities rubrics to assess student work in the project laboratory, a project-based course where groups of students solve open-ended experimental physics problems. They submit a report in the form of a web page, which is evaluated and returned to the students with feedback on what needs to be improved. These iterations are repeated until the report is deemed acceptable. We present our experience with the rubrics, and the development of a new rubric for the web report. We show that the introduction of the rubrics reduced the workload of the graders, while increasing the quality of the reports. Based on these results we conclude that using scientific abilities rubrics is a very efficient way of assessing experimental project-based work. We also present an analysis of which abilities, assessed by the rubrics, appear to be the most difficult to develop.

Figure 1

The plot of total words versus words in the first feedback times the number of iterations. The coefficient in the fit was used to estimate the total words in the feedback for those reports, for which we could not retrieve all the feedback.

Figure 2

An estimate for the number of words used to write all the feedback for each group. In the color graphic, red indicates the prerubrics period (before 2014-F), green indicates the intermediate period (the semester 2014-F), and blue indicates the rubrics period (from 2015-S on). Fifty percent of all reports fall within the respective shaded areas (the interquartile ranges). The lines indicate the respective medians. The labels on the horizontal axis indicate the year and semester [spring (S) and fall (F)].

Figure 3

Accepted old reports scored with the new rubrics. The figure shows the average score over all subabilities of each report.

Figure 4

The scores given to the old reports on each rubric item shown in a box-and-whiskers type diagram. The black line represents the median, it indicates the value under which at least 50% of all reports scored. The gray area is the interquartile range with at least 50% of the scores, and the whiskers are the remaining at most 50% of the scores. The outliers are represented as separate points (gray circles). Rubric C was only used for one report, so there is no distribution of scores. Examples: On rubric item B6, the majority of the scores were 0 (at least 50%—median), while scores reached up to 3. Rubric items G2 and G3 have both at least 50% of the scores between 0 and 1, but G2 has only one outlier with a score of 3, while G3 has between 25% and 50% of the scores between 1 and 3. Therefore, although the median is the same, the distribution of G2 is peaked towards lower scores, while of G3 towards higher scores. Both, however, contain at least one score 3.

Figure 5

The number of iterations needed before a report in the rubrics period received a full score on a particular rubric item represented as a box-and-whiskers type diagram. The difficulty levels are determined as described in the text. The threshold is indicated by the dotted line.

Figure 6

Comparison of our predictions and the measured difficulties of students. The difficulty levels are as described in the text. The labels O indicate the overridden values, as described in the text.

Figure 7

The rubric for the ability to design and conduct an observational experiment, which corresponds to our rubric for the ability to design and conduct an experiment to investigate a phenomenon.