What students learn when studying physics practice exam problems

We developed a web-based tool to provide students with access to old exam problems and solutions. By controlling the order in which students saw the problems, as well as their access to solutions, we obtained data about student learning by studying old exam problems. Our data suggest that in general students learn from doing old exam problems, and that having access to the problem solutions increases their learning. However, the data also suggest the depth of learning may be relatively shallow. In addition, the data show that doing old exam problems provides important formative assessment about the student’s overall preparedness for the exam and their particular areas of strength and weakness.

Figure 1

Student interaction with exam-prep tool. The highest usage was during the 24 h prior to the exam. Of the 1100 students enrolled in the course, 769 tried at least one question, 494 tried at least half, and 160 did all of the questions.

Figure 2

Results for the 22 groups of paired problems. The horizontal axis is the baseline score for the problem from the two groups that saw the question first, the vertical axis shows performance by students seeing the problem second. These two groups are separated based on if they had access to the complete solution for the paired problem or only the answer. See the appendix for more information.

Figure 3

There are 160 students who answered all problems in the tool. Each diamond represents a student’s score on the exam-prep tool and the exam. The horizontal axis is the students’ scores on the practice problems. The vertical axis represents the students’ scores on the exam. Most points lie above the dashed line, showing students performed better on the exam than on the exam-prep tool.

Figure 4

Thirteen of the questions given on the exam were matched with 13 of the paired problems on the exam-prep tool. Students that got both practice problems correct scored 15% higher on the matched problem on average than students that got both practice problems wrong.

Figure 5

Student relative performance on exam problems for topics the exam-prep tool predicted they would perform well on (diamonds), compared with topics the tool predicted they would not perform well on (squares). Early on, the predictive power is very weak, perhaps because students had time to work on the material they did not know. However, on the exam day the predictive power was quite good, showing a 20% difference in student performance.

Figure 6

Student performance on exam problems for topics the exam-prep tool predicted they would perform well on (diamonds), and topics the tool predicted they would not perform well on (squares). The students are binned based on their grade in the prior semester (mechanics) physics course. For $\mathrm{B}+$ through $\mathrm{C}-$ students, performance on the practice tool is a better predictor of performance then their performance in the prior physics course.