Assessing student written problem solutions: A problem-solving rubric with application to introductory physics

Problem solving is a complex process valuable in everyday life and crucial for learning in the STEM fields. To support the development of problem-solving skills it is important for researchers and curriculum developers to have practical tools that can measure the difference between novice and expert problem-solving performance in authentic classroom work. It is also useful if such tools can be employed by instructors to guide their pedagogy. We describe the design, development, and testing of a simple rubric to assess written solutions to problems given in undergraduate introductory physics courses. In particular, we present evidence for the validity, reliability, and utility of the instrument. The rubric identifies five general problem-solving processes and defines the criteria to attain a score in each: organizing problem information into a Useful Description, selecting appropriate principles (Physics Approach), applying those principles to the specific conditions in the problem (Specific Application of Physics), using Mathematical Procedures appropriately, and displaying evidence of an organized reasoning pattern (Logical Progression).

Figure 1

Problem corresponding to student solutions in Figs. 2 and 3. It can be solved in at least two different ways. One uses Newton’s second law to relate the force on a CO molecule to its acceleration, and then kinematics to relate this acceleration to the final speed. A second method uses conservation of energy to relate the work done on a CO molecule by the electric field to its kinetic energy (and thus its speed) when it exits the box. The correct answer is $1160\mathrm{N}/\mathrm{C}$.

Figure 2

Example of applying the MAPS rubric to a student solution.

Figure 3

Example of applying the MAPS rubric to a second student solution.

Figure 4

First (main) problem-solving task used in student interviews.

Figure 5

Scatter plot of total rubric score vs TA grade for all eight midterm problems ($N=918$). The correlation ($\rho$) between the two scores is 0.82 ($p<0.0001$). Points are shifted by a small random number so as not to mask clusters of scores.

Figure 6

Rubric scores from (a) 160 student solutions and (b) 159 solutions written by an instructor or taken from a textbook solution manual. This rubric was identical to the final rubric but used a maximum score of 4.

Figure 7

Graph of score agreement (weighted kappa) for two raters as a function of number of solutions scored for eight problems. The solutions were scored sequentially in time. The ninth data point is a rescoring of the first problem initially scored as the first data point.