Toward AI grading of student problem solutions in introductory physics: A feasibility study

Solving problems is crucial for learning physics, and not only final solutions but also their derivations are important. Grading these derivations is labor intensive, as it generally involves human evaluation of handwritten work. AI tools have not been an alternative, since even for short answers, they needed specific training for each problem or set of problems. Extensively pretrained AI systems offer a potentially universal grading solution without this specific training. This feasibility study explores an AI-assisted workflow to grade handwritten physics derivations using MathPix and GPT-4. We were able to successfully scan handwritten solution paths and achieved an R-squared of 0.84 compared to human graders on a synthetic dataset. The proposed workflow appears promising for formative feedback, but for final evaluations, it would best be used to assist human graders.

Figure 1

Possible workflow for AI-assisted grading of handwritten work.

Figure 2

Prompt and solution 1 for this study.

Figure 3

Solutions 2 through 5 to the prompt in Fig. 2.

Figure 4

Solutions 6 through 10 to the prompt in Fig. 2.

Figure 5

Solutions 11 through 14 to the prompt in Fig. 2.

Figure 6

Solutions 15 through 20 to the prompt in Fig. 2.

Figure 7

Solutions 21 through 25 to the prompt in Fig. 2.

Figure 8

Example of a grading prompt.

Figure 9

Example of a prompt to summarize the grading feedback.

Figure 10

Illustration of a possible future implementation of the first two steps in Fig. 1. A handwritten solution was scanned into PDF (left panel), translated into LaTeX using Mathpix [9], and corrected using GPT-4 (right panel).

Figure 11

Illustration of a possible future implementation of the first two steps in Fig. 1. A handwritten solution was scanned into PDF (left panel), translated into LaTeX using Mathpix [9], and corrected using GPT-4 (right panel).

Figure 12

Correlation between manual and AI-grading rubric scores.

Figure 13

Correlation between manual and AI-grading total scores. For each data point, the solution number and the standard deviations of the AI grading (vertical error bars) and human grading (horizontal error bars) are indicated.

Figure 14

Dendrograms and heat maps of similarities between the rubric grades obtained from manual grading (top panel) and AI grading (middle panel), as well as the AI similarity-of-approach measure (bottom panel). The cluster cuts are indicated by purple lines; the dashed line indicates possible cuts.