Gaze patterns enhance response prediction: More than correct or incorrect

Eye tracking enables the reconstruction of eye movements and thus the analysis of visual information selection and integration processes during problem solving. In this way, learner-specific difficulties can be identified and problem-solving process can be adapted accordingly. For such an adaptation, the prediction of response behavior plays a crucial role. To predict whether a problem is solved correctly or incorrectly, the segmentation of the visual stimulus into specific areas of interest (AOIs) is particularly crucial for the quality of a prediction based on eye-tracking data. In the study presented here, the gaze data of $N=115$ students were analyzed while solving the Test of Understanding Graphs in Kinematics (TUG-K), a validated test instrument whose items include graphs of position, velocity, and acceleration versus time. For selected items, response accuracy was predicted based on visual attention using multiple logistic regression analysis, examining the influence of AOI segmentation. The prediction quality could be significantly improved when the diagram was not considered as contiguous AOI, but when it was divided into solution-relevant and solution-irrelevant areas. To verify that the AOIs selected by the regression algorithm are indeed relevant to the solution process, an expert rating was performed, which showed moderate to good agreement between the AOIs rated by the experts as relevant to the correct solution and the AOIs selected by the algorithm. There are also pairs of items in the TUG-K that require the same mathematical solution procedure but differ in the physical context. This opened the possibility to investigate a new approach. Based on response accuracy and allocation of visual attention to one item, the response accuracy of the other item of the pair was predicted. It could be shown that the prediction quality based on visual attention was significantly higher than the prediction based on response accuracy. This demonstrates the added value of collecting process-based data versus product-based data for prediction and thus for learner-specific adaptation. The results of this study indicate, first, that only certain areas are crucial for a correct solution when extracting information from diagrams and, second, that the application of mathematical procedures plays a crucial role in interpreting graphs of different physics quantities. These findings thus provide insight into the visual strategies involved in interpreting kinematic diagrams and can also serve as a basis for eye tracking-based adaptation of problem-solving processes, in which adaptation can occur even before an incorrect answer is given.

Figure 1

Example item from the TUG-K. The task has been translated verbatim into German and reads as follows in the English original: The following figure shows the position versus time graph of an object. The velocity of the object at $t=2\mathrm{s}$ is:. The axis labels are also written in German and mean time (“Zeit”) and position (“Ort”).

Figure 2

Segmentation of the diagram of an exemplary item into different micro-AOIs. The axis labels are written in German and mean time (Zeit) and position (Ort).

Figure 3

Comparison of McFadden’s pseudo $R^2$ for intra-item prediction based on the different datasets. The limit from which good prediction quality can be assumed is shown as a dashed line.

Figure 4

Predictive AOIs of item 18 (red: longer TVD with incorrect answer, green: longer TVD with correct answer). The axis labels are written in German and mean time (Zeit) and position (Ort).

Figure 5

Comparison of McFadden’s pseudo $R^2$ for inter-item prediction based on the different datasets.