To draw or not to draw? Examining the necessity of problem diagrams using massive open online course experiments

Previous research on problem diagrams suggested that including a supportive diagram, one that does not provide necessary problem solving information, may bring little, or even negative, benefit to students’ problem solving success. We tested the usefulness of problem diagrams on 12 different physics problems (6A/B experiments) in our massive open online course. By analyzing over 8000 student responses in total, we found that including a problem diagram that contains no significant additional information only slightly improves the first attempt correct rate for the few most spatially complex problems, and has little impact on either the final correct percentage or the time spent on solving the problem. On the other hand, in half of the cases, removing the diagram significantly increased the fraction of students’ drawing their own diagrams during problem solving. The increase in drawing behavior is largely independent of students’ physics abilities. In summary, our results suggest that for many physics problems, the benefit of a diagram is exceedingly small and may not justify the effort of creating one.

Figure 1

Experiment design. Each experiment consisted of a pair of problems. The same design was used for all 6 experiments conducted.

Figure 2

Percentage of first attempt correct for each problem. The rightmost column is the percentage correct aggregated over all 12 problems. *Difference is significant at the 0.05 level. ** Difference is significant at the 0.01 level. (chi-squared test).

Figure 3

Comparing median time on task (seconds) for each problem. **Significant at $p=0.01$ level, Mann-Whitney U test.

Figure 4

Comparing the drawing behavior between the two conditions. (a) Percentage of students who chose to draw a diagram during problem solving. The rightmost columns are the drawing percentage aggregated over all 12 problems. *Difference is significant at the 0.05 level. ** Difference is significant at the 0.01 level. (chi-squared test) (b) The difference in percentage between the two conditions on each problem.

Figure 5

Comparing the percentage of first attempt correct rate between the two conditions for “drawers” (a) and “nondrawers” (b). *Difference is significant at the 0.05 level. ** Difference is significant at the 0.01 level. (chi-squared test).

Figure 6

The impact of DG and NDG condition on students with different skill backgrounds averaged over all 6 experiments. (a) The impact on performance as measured by first attempt correct rate. (b) The impact on drawing behavior.

Figure 7

Dendrogram of cluster analysis on expert’s ratings of problems. The height is a metric for measuring the relative distance between items and clusters. The three major clusters are largely independent of either the clustering algorithm or the choice of distance metric.

Figure 8

Radar graph for the average rating of each problem cluster.

Figure 9

Diagram and problem text of problem P4.

Figure 10

Diagram and problem text for problem P8.

Figure 11

Diagram and problem text for problems P5, P11, and P12.