Differences in visual attention between those who correctly and incorrectly answer physics problems

This study investigated how visual attention differed between those who correctly versus incorrectly answered introductory physics problems. We recorded eye movements of 24 individuals on six different conceptual physics problems where the necessary information to solve the problem was contained in a diagram. The problems also contained areas consistent with a novice-like response and areas of high perceptual salience. Participants ranged from those who had only taken one high school physics course to those who had completed a Physics PhD. We found that participants who answered correctly spent a higher percentage of time looking at the relevant areas of the diagram, and those who answered incorrectly spent a higher percentage of time looking in areas of the diagram consistent with a novicelike answer. Thus, when solving physics problems, top-down processing plays a key role in guiding visual selective attention either to thematically relevant areas or novice-like areas depending on the accuracy of a student’s physics knowledge. This result has implications for the use of visual cues to redirect individuals’ attention to relevant portions of the diagrams and may potentially influence the way they reason about these problems.


I. INTRODUCTION
Often diagrams in physics problems contain information that is both relevant to the solution of the problem and information that is irrelevant.Students commonly use this irrelevant information as they reason their way to an incorrect answer, when, in fact, they should simply ignore it.The use of irrelevant information in student answers has been observed in many studies, such as those by McDermott looking at common student difficulties in understanding motion [1,2].
A convenient way of measuring what learners pay attention to is to measure their eye movements.In the current study, we measure students' saccades (i.e., when eyes are in motion) and fixations (i.e., when eyes are stationary at a specific spatial location) to measure where they attend in physics problems.Attention is guided by two sources of information, one internal and the other external, referred to as top-down and bottom-up information, respectively.Bottom-up information is based on the physical characteristics of stimuli on which a person fixates, and the visual processes that work on such information tend to be very fast and involve primitive brain areas early in the visual stream [3,4].The influence of bottom-up information on attention is generally explained in terms of the relative perceptual salience of elements of the visual stimuli [5][6][7].Perceptually salient regions of an image tend to be those with relatively greater contrast in terms of color, orientation, intensity, or motion compared to the other image elements.Perceptually salient elements are argued to automatically capture attention through primitive visual mechanisms [8,9].Itti, Koch, and Niebur [5,6] have created a computational algorithm to produce a salience map of a scene or diagram, using contrasts of light intensity, orientation (e.g., of lines), and color.Such salience maps have been found to predict significantly greater than chance where people will fixate as they view images [10,11] though top-down factors (described below) have been shown to have even larger effects on where people fixate [12][13][14].Models of the effects of saliency on eye movements argue that one's attention first selects the location with highest salience, this location is then fixated, and after the information there has been sufficiently processed, one's attention moves to the next most salient spatial location.Carmi and Itti [7] studied the effects of saliency as a function of viewing time and found that their perceptual salience model best predicted the first six or seven fixations when viewing a scene (see also Parkhurst, Law, and Niebur [10]).For the average viewer, this is equivalent to about the first 2 sec of viewing.This suggests that bottom-up processes are more dominant in the first 2 sec of viewing, with top-down processes exerting a greater influence on eye movements thereafter.
In the domain of physics, it has been proposed by Heckler [15] that the consistent wrong answer pattern by novices on introductory physics problems is in part a result of their attention being directed to the most perceptually salient and plausibly relevant features in a problem.He explains that the most salient features capture attention through perceptual processes and less salient features have little opportunity to be considered, leading to an incorrect answer.Student answer patterns are cited as evidence for these perceptually driven responses; however, no eye-movement data supporting this hypothesis are provided.
However, some researchers [13] have found that perceptual saliency, as assessed by Itti's model, did poorly in accounting for the paths that viewers' eyes took when given a search task.For instance, in the study by Hegarty, Canham, and Fabrikant [16], university students viewed weather maps and were tasked to determine wind direction.The researchers found no evidence to indicate that over the full trial period participants looked at the perceptually salient areas of the weather maps based on Itti's algorithm.However, the researchers did not limit their analysis to only the first 2 sec of viewing, when the effect of saliency driven bottom-up processes should be most pronounced.
Top-down information and the processes that act upon it are based on the viewer's prior knowledge, task goals, and expectancies.Top-down effects on attention tend to be mediated by higher brain areas and occur later in the time course of vision [17,18].Most importantly for the current study, it has been observed that experts in a domain attend to task-relevant portions of a diagram more than novices in that domain.Thus, the expertise of these individuals helps to guide their visual attention in the diagram.Jarodzka et al. [19] studied the visual attention of both novices and experts who viewed videos of unfamiliar fish swimming and classified the type of locomotion.The authors found that experts spent significantly more time fixating on relevant areas of the video than biology students, who had the necessary background knowledge for differentiating types of locomotion but little practice in this classification task.The authors also found that novices spent more time than experts fixating on areas irrelevant for determining locomotion.Similar studies have measured eye movements of experts when viewing art [20] and playing chess [21], and have shown that the increased domain knowledge in these fields affects where people fixate while performing domain-relevant visual tasks.Thus, important differences in the eye movements of experts, who possess the necessary domain knowledge, versus novices, who do not possess such knowledge, can be seen by tracking their eye movements while they are carrying out domain-relevant tasks [22][23][24].
Visual attention allocation in the discipline of physics may work somewhat differently than the previously discussed disciplines as our everyday interactions with the physical world may help us develop ideas about how it works without any formal instruction.Thus, novice reasoning may be influenced by top-down knowledge, which may be based on either correct or incorrect representations of the physical world.Physics education research has cataloged a pattern of consistently incorrect answers to many common physics questions.These patterns, called misconceptions [25,26], may be a result of stable mental entities built up through years of interaction with the physical world and through schooling.These consistently incorrect answer patterns have also been explained in terms of a misapplication of small chunks of information, referred to as resources [27], which students develop through their experience with the world.In a physics class, they may bring together groups of resources to answer questions, and may apply inappropriate resources to a given situation.Conversely, these consistently incorrect answer patterns may be the result of students categorizing scientific ideas into inappropriate ontological categories [28].However, while the precise cognitive processes that lead to these consistently incorrect answer patterns are still being debated, all the proposed explanations rely in some way on ''domain knowledge'' about how the world works.Thus, for the purposes of this paper, we will refer to the cognitive underpinnings of these consistently incorrect answer patterns in physics as novicelike misconceptions.Therefore, a key question addressed in the current study is whether both experts' scientifically correct domain knowledge and beginners' novicelike misconceptions exert topdown influences on visual attention when viewing physics problems.If novicelike conceptions do influence eye movements when answering physics problems, then participants who provide incorrect answers should spend more time fixating on irrelevant areas of a diagram than the relevant or perceptually salient areas of the diagram.
The interaction between perceptual salience and level of domain knowledge is also important to consider.A study by Lowe [29] found that the written responses of meteorology students who studied animated weather maps and recorded generalizations about them primarily contained information extracted from perceptually salient areas of the weather maps.However, a more recent study by Hegarty, Canham, and Fabrikant [16] showed an interesting interaction between bottom-up salience and top-down knowledge in guiding attention while looking at weather maps.The authors investigated this interaction by recording participants' eye movements as they viewed static weather maps in which the relative salience of task-relevant and task-irrelevant information had been manipulated.Before instruction, participants spent more time fixating on taskirrelevant areas when they were the most perceptually salient elements on the map.However, after instruction, there was no difference in the time spent fixating on taskirrelevant information regardless of which elements had been made most perceptually salient.Thus, while both of these studies show that novice learners are strongly influenced by areas of a diagram that are perceptually salient, the study by Hegarty, Canham, and Fabrikant shows that the effect of perceptual salience on attention decreases as domain knowledge increases.
Previous research has shown that there is competition for attention between bottom-up and top-down processes as people view visual stimuli.The key question addressed in the current study is how these processes interact when answering physics problems.We use eye-movement data to infer the extent to which bottom-up and top-down processes influence people's attention as they answer introductory conceptual physics questions containing diagrams.
We hypothesize that those with adequate domain knowledge to correctly answer a problem will spend more time fixating on thematically relevant areas of a diagram that provide the solution to the problem than on irrelevant areas of the diagram.Conversely, we predict that those who answer incorrectly will spend more time fixating elsewhere in the diagram.More specifically, based on previous research in physics education concerning novicelike misconceptions, which consistently lead to incorrect answers, we hypothesize that those answering the problem incorrectly will spend more time fixating on areas of the diagram consistent with a novicelike misconception.These participants will initially attend to perceptually salient areas of the diagram, but will quickly disengage their attention from these areas and instead attend to novicelike areas.Such effects would suggest a strong role for top-down factors in guiding attention while solving physics problems involving diagrams.
Alternatively, it has been shown that perceptual salience has a larger influence on novice learners' eye movements than those with more domain knowledge.Based on this finding, we could predict that the fixated locations of those who answer incorrectly are more likely to be influenced by perceptual salience than those who have adequate domain knowledge.Such effects would suggest a strong role for bottom-up factors in guiding attention during physics problem solving with diagrams.Thus, a key question is whether the attention of people who answer physics problems incorrectly is more influenced by the top-down factor of novicelike misconceptions or by the bottom-up factor of the perceptually salient areas of the diagram.
Specifically, we examine the following three-part research question: How does the correctness or incorrectness of one's answer to a physics problem involving a diagram relate to the time spent looking at those areas of the diagram that are (a) thematically relevant to the problem's solution?(b) consistent with novicelike misconceptions?Or (c) perceptually salient?

II. STUDY 1: INTERVIEWS TO DETERMINE NOVICELIKE AREAS OF INTEREST A. Study 1: Methodology
In order to define areas of a physics problem diagram that contain visual information related to a novicelike misconception, we conducted individual interviews with students enrolled in an introductory psychology course.We specifically looked at the interview segments where participants provided incorrect answers to the physics problems and observed the areas of the diagram that students identified and discussed while giving their verbal explanation.This information was used to define ''novicelike'' areas of interest (AOI), or specific areas of the diagram in which a participant who answered incorrectly would use to come to their answer.These areas of interest will be used in the analysis for study 2.

Participants
The participants were 13 students (eight females) enrolled in an introductory psychology course.All of the students had taken at least one physics course in high school, though some had taken an introductory physics course at the university level as well.They were given course credit for participation.

Materials
The materials consisted of 10 multiple-choice conceptual physics problems covering various topics in introductory physics including energy, kinematics, and graphing of motion (see the Appendix for a list of problems).Each problem contained a diagram that had a thematically relevant visual component that students needed to attend to in order to correctly answer the question.For example, in problem 4 (see the Appendix), to compare the speeds of ball A and ball B, one must attend to the distances between the balls at each time interval and ignore the point where the balls are aligned spatially.So, the distance between balls at 2 and 3 sec is the relevant area to attend to.These problems were chosen based on prior experience of the researchers which indicated that these problems could be answered using common naive conceptions documented in physics education literature [1,2,30].

Procedure
Each participant took part in an individual session which was between 20 and 40 min long.At the beginning of the session, participants were given a short explanation of the goal of the interview and the purpose of the research.Further, they were instructed to think aloud and explain their reasoning process as they answered each question.They were told they might be asked additional clarifying questions during their explanations.Participants were given one problem at a time, each printed on an 8 1=2 Â 11 sheet of paper.They were allowed to write or draw on the problems as they deemed necessary.If a participant's answer was not clear, the interviewer asked questions to clarify the meaning of the explanation.Participants' verbal explanations, gestures, and writing on the paper were recorded with a Flip video camera.

B. Study 1: Analysis
The purpose of these interviews was to determine which portion of each diagram was attended to by incorrect problem solvers.Therefore, only the interview segments where the participant gave a final incorrect answer were included in the analysis.A phenomenological approach was used to code the interviews [31].Table I contains the answers and reasoning provided by participants who answered the problems incorrectly.Four of the 10 problems used in the interviews showed no consistent answering patterns among incorrect solvers after a first pass analysis.These problems are not included here, as there were no identifiable novicelike areas to be utilized in study 2.

C. Study 1: Results and conclusion
The six problems included in this analysis (see the Appendix) showed consistent incorrect reasoning patterns.These answer patterns align well with previous findings in the literature.Student difficulties with distance versus time graphs were studied extensively by McDermott, Rosenquist, and van Zee [2] and Beichner [32].McDermott, Rosenquist, and van Zee interviewed students at all levels of introductory college physics as well as high school physics and physical science students.
They found when students responded to a problem very similar to problem 2 used in our study, they often selected the point where the graph crossed the x axis because ''the position was going from positive to negative,'' instead of correctly choosing the point on the graph where the slope was zero.In a similar study, Trowbridge and McDermott [33] found that a common student misconception is the idea that when two objects have reached the same spatial position they have the same speed.In their study, Trowbridge and McDermott used a problem very similar to problem 4 in our study, and found that a substantial number of students chose the instant when the balls passed each other as the time when they were moving at the same speed.In problem 4 in our study, this instant of the balls passing is at 1 sec, which is the most common incorrect answer we observed.Conflating position and speed is also observed in problems 3 and 7 in our study.In problem 7, we observed students incorrectly choosing the point where the graphs of two objects crossed as the point when the objects were moving at the same speed.This crossing point is the place where the objects have the same position, but not the same speed.In problem 3, we observed students choosing the points where the graph crosses the x axis as the place where the object's speed is zero.These crossing points are the places where the object has a zero position relative to the origin, but not a zero speed.So the incorrect answers we observed on problems 3 and 7 align well with this documented student difficulty.Viennot [1,34] also investigated student difficulties with force and motion.She surveyed about 2000 university and high school students in France, Belgium, and Britain and found that students often attempted to account for differences present in a diagram that may or may not be related to the problem solution.This is consistent with our findings in problems 1 and 10.In problem 1, tracks A and B are different, though one only needs to notice that the initial and final heights are the same, so the final speeds will be the same.Students who answered incorrectly in our study discussed the differences between the tracks to explain their answers.On problem 10, one needs to notice that the heights of each slope are the same.Those who answered incorrectly in our study primarily reasoned using the fact that the slopes were changing.
In sum, there was strong agreement between our interview findings and documented student difficulties in the literature.This gave us confidence that the definitions of novicelike areas of interest, for each physics problem, do indeed represent the most common novicelike answers of the larger population of introductory physics students.

III. STUDY 2: DETERMINING DIFFERENCES IN VISUAL SELECTIVE ATTENTION BASED ON CORRECTNESS OF PROBLEM SOLUTION A. Study 2: Methodology 1. Participants
There were 24 participants in the study (three females, two were graduate students and one was a psychology student) with two different levels of experience in physics.Ten participants were first-year through fifth-year Ph.D. students in physics who had either taught an introductory physics course or had been a teaching assistant for an introductory physics lab.One participant was a postdoctoral candidate in physics who had received his Ph.D. within the last two years and had teaching experience.Thirteen participants were enrolled in an introductory psychology course and had taken at least one physics course in high school, though some had also taken an introductory physics course at the university level.The Ph.D. students and postdoctoral candidate participated as volunteers and the psychology students received course credit for their participation.Because we were looking to compare those who answered the physics problems correctly to those who answered incorrectly, we selected participants with a broad range of experience.We expected that the Ph.D. students would answer correctly, while the psychology students might answer incorrectly, though we knew that this might not always be the case since there is a wide distribution of expertise among introductory physics students and physics graduate students [35].

Materials
The materials consisted of the six multiple-choice introductory physics problems analyzed in study 1 (see the Appendix).

Apparatus
Participants were presented with physics problems on a computer screen viewed at a distance of 24 in.using a chin and forehead rest to minimize participants' extraneous head movements.The resolution of the computer screen was set to 1024 Â 768 pixels with a refresh rate of 85 Hz.Each physics problem subtended 33:3 Â 25:5 of visual angle.Eye movements were recorded with an EyeLink 1000 desktop mounted eye-tracking system [36], which had an accuracy of less than 0.50 of visual angle.An eye movement was classified as a saccade (i.e., in motion) if the eye's acceleration exceeded 8500 =s 2 and the velocity exceeded 30 =s.Otherwise, the eye was considered to be in a fixation (i.e., stationary at a specific spatial location).A nine-point calibration and validation procedure was used at the beginning of the experiment.

Procedure
Each participant took part in an individual session lasting 20-40 min.At the beginning of the session, participants were given a short explanation of what to expect in the study.After calibrating the eye-tracking system, if the validation's mean error was 0:50 of visual angle, the experiment began, otherwise the calibration and validation was repeated until successful.Next, the participant was instructed to silently answer 10 multiple-choice questions while their eye movements were recorded.Participants indicated their answer to each question using number keys on the keyboard.Between questions, a calibration drift correction procedure was done to ensure proper calibration throughout the experiment.This procedure required the participant to fixate on a small white dot in the middle of a gray screen and press a key.Pressing the key caused the screen to advance to the next problem when the participant's fixation was within a predefined area around the white dot.Finally, each participant was asked to provide a cued verbal retrospective report [37] for which they were shown a replay of their eye movements on each problem and asked to explain their thought processes (either after watching the replay of their eye movements or concurrently while watching them).This method has been found to produce more in-depth explanations than without viewing one's eye movements.If a participant's explanation was unclear, they were asked follow-up questions.Participants were given unlimited time to answer the questions and provide retrospective verbal reports.Verbal explanations and gestures were recorded with a Flip video camcorder.

B. Study 2: Analysis
To analyze participants' eye fixations, we defined areas of interest (AOIs) for specified areas of each diagram.These AOIs were used to determine the total fixation time (i.e., the total amount of time the participant spent fixating on a given AOI).There were three different types of AOIs identified for each physics problem analyzed in study 1.These types were thematically relevant AOIs, perceptually salient AOIs, and novicelike AOIs.The definition for the thematically relevant AOI came from three independent raters, one physics professor and two Ph.D. students in physics, who indicated, on each of the problems, the area which contained visual information necessary to answer the problem.The definition for the perceptually salient AOI in each problem was determined using an implementation of the Itti, Koch, and Niebur saliency map algorithm in MATLAB [38].This MATLAB toolbox produced a heat map representation of relative saliency over the entire diagram for each problem (see Fig. 1).The area on the diagram with the highest rating of saliency was used to define the perceptually salient AOI.If there were several portions of the diagram with the highest level of perceptual salience, according to the salience map, then all of these areas were used when defining the perceptually salient AOI.
The novicelike AOI was defined based on the interviews described above in study 1. Figure 2 shows the thematically relevant, novicelike and perceptually salient areas of the problem whose heat map is shown in Fig. 1.
The areas of the diagram referred to by the majority of the interviewees from study 1 who answered the problem incorrectly were defined as the novicelike AOI for each of the problems.These areas are listed in Table II.
These thematically relevant, perceptually salient, and novicelike AOIs were applied to the problems analyzed in study 1.Additionally, an AOI containing the entire diagram was applied to each of the problems.The total amount of time each participant spent fixating on each AOI was determined (total fixation time), as well as the total time spent looking at the entire diagram.To account for differences in total viewing time on each problem, the percentage of time spent in each respective AOI was determined by dividing the total viewing time, for each participant, in a specified AOI by the total time spent viewing the entire diagram [39].The percentage of time spent in each type of interest area was compared between students who answered the problem correctly and those who answered incorrectly for the entire problem set.There were a few instances where the eye-movement data file were corrupted for a participant on a single problem.In this case, the participant's data were not included in the analysis.
We were also interested in determining if perceptual salience played a greater role in influencing eye  movements in the first two seconds of viewing the problem diagram.To do this, we determined the first time the participant's eye left the problem statement to look elsewhere.Applying the same AOIs described previously, we selected 2 sec of fixation data immediately following the transition from reading the problem statement to looking elsewhere in the problem.It should be noted that not all participants read the problem statement, viewed the diagram, and then the answer choices.Some participants looked from the problem statement to the diagram very briefly and then continued reading and some went from the problem statement to the answer choices.Thus, the first 2 sec of fixation data represents many different patterns of viewing.We then converted the fixation time from the first two seconds to a percentage and compared the percentage of time spent in each type of interest area between students who answered the problems correctly versus those who answered incorrectly.

C. Study 2: Results and discussion
Mixed factorial 2 Â 6 ANOVAs (analysis of variance) with proportion of time in each AOI type as the dependent variable and problem number and correctness of answer as independent variables were conducted for all three AOI types.Results for the full trial period are reported in Table III.Results for the first 2 sec of viewing the diagram are reported in Table V.

Full trial period
For the full trial period, we found a significant main effect for correctness of answer as well as for problem number for all three AOI types.We were looking for a main effect of correctness, as this would indicate there are differences in percentage of time spent in an AOI between those who answered correctly and those who answered incorrectly.The main effect of correctness addresses our research questions and will be further analyzed below.The  main effect of problem number indicates there is at least one difference in proportion of time in each AOI type between different problems.We were not interested in how the proportion of time spent fixating varies between problems, as this is not relevant to our research questions, so the effect of problem number will not be further analyzed.We found a significant interaction between problem number and correctness of answer in the perceptually salient AOI.This means the relationship between correctness and time spent in the perceptually salient area is different across problems.This interaction is not relevant to our research question and will not be further investigated.
The main effect of correctness was further analyzed for each of the six different problems using a one-way ANOVA with percentage of time for all three AOI types as the dependent variable and correctness of answer as the independent variable.Results of one-way ANOVAs for each type of AOI for the full trial period are reported in Table IV.Mean percentage of fixation time and standard error for the correct and incorrect responders for each question are also shown in Table IV.The footnote indicates a significant difference at the ¼ 0:05 level.
We found that on five out of six problems used in study 2, those who answered the problem correctly spent a higher percentage of total viewing time fixating on thematically relevant areas in the problem diagram (Table IV).Those who answered correctly likely had the domain knowledge needed to solve each problem, and therefore spent more time viewing the relevant areas in each diagram.This result is consistent with previous findings where those with high levels of domain knowledge in a discipline, such as identifying fish locomotion [19], art [20], and chess [21], spend more time looking at areas of diagrams and pictures relevant to a task.Our finding is evidence for top-down processes playing a key role in guiding visual attention when solving physics problems correctly.
We also found that on five out of six problems, those who answered the problem incorrectly spent a higher percentage of total viewing time looking at areas of the diagram consistent with a novicelike response (Table IV).
Furthermore, on the one problem that did not quite reach statistical significance (p ¼ 0:058) the effect was in the same direction as the other five problems.These novicelike AOIs were determined through individual interviews described in study 1, and were consistent with the physics education literature describing common student misconceptions.Importantly, the finding that incorrect solvers spent more time fixating on novicelike areas is evidence for their visual attention being guided by top-down processes.However, instead of attention being guided by scientifically correct domain knowledge, incorrect problem solvers' attention was guided by novicelike misconceptions.Thus, when solving physics problems, top-down processing plays a key role in guiding visual selective attention either to thematically relevant areas, or novicelike areas, depending upon the scientific correctness of a student's physics knowledge.
Concerning the effects of bottom-up processes in guiding attention during physics problem solving, we found that those who answered incorrectly spent more time in perceptually salient areas during the full problem period on only two of the six problems, namely, problems 3 and 7. Nevertheless, for five of the six problems the effect was in the predicted direction, such that incorrect problem solvers spent a higher percentage of total time fixating on the perceptually salient AOIs than the correct problem solvers.However, four of those effects were not statistically significant.A likely explanation for this result is that in problems 3 and 7, the perceptually salient AOI partially or completely overlapped with the novicelike AOI (Figs. 3  and 4), which was not the case for the other four problems.We have already shown that those who answered the problem incorrectly spent significantly more time fixating on the novicelike AOIs on problems 3 and 7 than those who answered the problem correctly.So the significant result for problems 3 and 7 for the perceptually salient AOI is likely due to this AOI overlapping with the novicelike AOI.This result also seems to indicate that attending to the perceptually salient area is not necessarily a good predictor of correctness.These results appear to be consistent with a study of change blindness that found that problem solvers seldom notice changes in color, even though color is most perceptually salient [40].Thus, when considering the full time period of problem solving, perceptual salience appears to have played a minimal role in guiding the attention of incorrect physics problem solvers.Nevertheless, previous vision research has suggested that the effects of bottom-up perceptual salience on eye movements are limited to the first 2 sec of viewing a stimulus [7].Thus, this seemingly null result could be argued to have resulted from diluting the effect of saliency by including eye-movement data from the entire duration of the trial, rather than only the first 2 sec.We therefore reanalyzed the data including only the first 2 sec that participants spent viewing the diagram.

First 2 sec after leaving problem statement
To reanalyze the data including only the first 2 sec of viewing a diagram, we completed a mixed factorial 2 Â 6 ANOVA with proportion of time in each AOI type as the dependent variable and problem number and correctness of answer as independent variables for all three AOI types for the first 2 sec of viewing the diagram.These results are reported in Table V.We were looking for a main effect of correctness, as this would indicate there are differences in percentage of time spent in an AOI between those who answered correctly and those who answered incorrectly.For the first 2 sec after leaving the problem statement, we found no main effect for correctness of answer for any of the AOI types.So, there are no significant differences in proportion of time spent fixating in the AOI types between those who answered correctly and those who answered incorrectly for any of the problems and no further analysis was conducted.
We did find a main effect for problem number for the novicelike and perceptually salient AOIs.This means for each of these AOIs, there is at least one difference in proportion of time between the different problems when considering the data for all participants.We were not interested in how the proportion of time spent fixating varies between problems, as this is not relevant to our research questions.We also found a significant interaction between problem number and correctness of answer in the thematically relevant AOI.This means the relationship between correctness and time spent in the thematically relevant area is different across problems.This interaction also does not address our research questions, and is not analyzed further.
The mean percentage of fixation time spent looking in thematically relevant, novicelike, and perceptually salient AOIs for participants who answered the question correctly and incorrectly for the first 2 sec of viewing the diagram is displayed in Table VI.As mentioned above, there are no significant differences between the percentage of fixation time for correct and incorrect solvers shown in this table.
The reanalysis of the data for the first 2 sec of viewing the diagram found no statistically significant differences between correct and incorrect solvers on any of the problems for the perceptually salient AOI.Indeed, there were no statistically significant differences between correct and incorrect solvers in time spent in the thematically relevant or novicelike AOIs.In sum, we found no support for the hypothesis that perceptual salience influences visual selective attention more for incorrect problem solvers during the first 2 sec of diagram viewing.This result is consistent with previous studies [e.g., [12,33]] that have shown that top-down influences on visual attention tend to dominate bottom-up influences when a viewer is given a specific goal or task.Nevertheless, such null results for the effects of bottom-up saliency on visual attention are consistent with our own results, which considered both the full problem solving time period and only the first 2 sec, and found little if any effects.
However, before completely rejecting the hypothesis that bottom-up saliency affects attentional selection during physics problem solving, we must consider two observations that provide partial support for it.First, it may be that the early effect of perceptual salience on eye movements was present; however, the data lacked sufficient statistical power to detect it.Some support for this explanation is shown by comparing the mean difference for the correct versus incorrect problem solvers for the perceptually salient AOIs for the first 2 sec of viewing the diagram (Table VI).Specifically, the percentage of time spent looking in the perceptually salient AOI is higher for incorrect solvers than correct problem solvers on five of the six problems, though not statistically significantly so.Thus, it is possible that a larger study with more observations might show this effect to be statistically significant.Secondly, the perceptual salience model proposed by Itti and Koch [6] predicted that early in scene viewing eye movements are more influenced by bottom-up perceptual information than top-down knowledge.Therefore, the saliency model would predict that early in viewing a physics problem, correct and incorrect problem solvers would not have had sufficient amount of time to apply their (correct or incorrect) top-down knowledge to guide their attention to thematically relevant or novicelike areas of the diagram.If so, during the first 2 sec of viewing the diagram, there should be no difference between correct and incorrect problem solvers' percentage of total fixation time in either the thematically relevant or novicelike AOIs.The data support this hypothesis, which shows that there is no significant difference in viewing time for thematically relevant AOIs between correct and incorrect problem solvers.In sum, the data showed essentially no influence by top-down domain knowledge during the first 2 sec of diagram viewing, though such effects were statistically significant later in time, when considering the full problem solving time period.Thus, based on the above two observations, we must withhold complete rejection of the hypothesis that bottom-up salience affects the visual selective attention of incorrect physics problem solvers.Even so, such an interpretation of the data should be made cautiously since it is based on null effects.Future studies will be required in order to explicitly test the effects of bottom-up and top-down information on early and late visual selective attention processes in eye movements.

IV. FUTURE DIRECTIONS AND LIMITATIONS
Overall, these findings motivate the use of visual cues to redirect individuals' attention to relevant portions of the diagrams and potentially influence the way they reason about these questions.The problems used in study 2 all contained AOIs consistent with novicelike misconceptions.Those who answered incorrectly spent more time looking at these novicelike AOIs.One way to help incorrect problem solvers pay attention to the relevant areas of a problem diagram is to overlay dynamic visual cues on it.These cues should have very high perceptual salience, perhaps using color or motion cues, in order to reliably attract the problem solver's attention.Visual cues have been found to facilitate comprehension in several contexts, such as insight problems [41] and educational animations [42].Grant and Spivey [41] studied an insight problem where one must determine how to use lasers to kill an inoperable tumor without harming the healthy tissue surrounding the tumor.To solve this problem, one must use several weak lasers at different spatial positions surrounding the tumor, so as not to damage the healthy tissue, but at the point at which the lasers converged, it would have a high enough intensity to kill the tumor cells.They found more participants correctly solved the problem when the task-relevant information in the diagram, namely, the healthy tissue, was made more perceptually salient by increasing and decreasing its width.Many studies using visual cues to focus viewers' attention on relevant information have been conducted using animations.In one of these studies, de Koning et.al. [43] used a spotlight cueing technique to focus learners' attention on the valves of the heart in an animation of the cardiovascular system.He found those who viewed the animation with the cues had higher comprehension and transfer scores on post-test questions about heart valves and the cardiovascular system.These examples and many others suggest that visual cues overlaid on physics problems such as those in the current study may help students to ignore the novicelike AOIs of diagrams and instead pay attention to the thematically relevant AOIs in order to reason in a scientifically correct manner about the problem.
This study describes only a limited number of introductory physics problems.To increase the generalizability of our conclusions, the study should be repeated with more problems from other areas of introductory physics and with students having a wider range of prior knowledge of physics.Additionally, the study could be improved by using a larger number of participants, which would increase the statistical power of the study and enable us to more thoroughly test the perceptual saliency hypothesis.Furthermore, the conclusions we have drawn about the influence of perceptual salience on visual attention must remain tentative as we only used one computational model of visual salience (albeit the most famous one) and in some of the problems the perceptually salient AOI overlapped the novicelike AOI.In future work several different models of saliency will be used and only problems where the perceptually salient AOI does not overlap any other AOI will be used.Thus, future research should include similar studies using the suggestions discussed above as well as studying the influence that visual cues overlaid on such problems have on students' visual attention and the correctness of their problem answers.

ACKNOWLEDGMENTS
We gratefully acknowledge the contributions of Dr. Elizabeth Gire for her participation in discussions of the design of the questions used in this study.

FIG. 2 (
FIG. 2 (color online).Thematically relevant AOI is the distance between balls at 2-3 sec.Novicelike AOI is when the balls are at the same position, at 1 sec.Perceptually salient AOI is oval around ball B at 3 and 4 sec.

FIG. 3 (
FIG.3(color online).Itti, Niebur, and Koch saliency map for problem 3. The perceptually salient AOI overlapped the novicelike AOI, which was at the origin of the graph.
Various problems used in studies 1 and 2 are shown in Figs.5-10.

TABLE I .
Number of students providing each answer and reasoning on conceptual physics questions with a diagram.

TABLE II .
Novicelike AOIs defined based on the most common incorrect student responses in study 1.
4 Point where balls A and B line up spatially 7 Point where graphs of two objects cross 10 Slopes A, B and C

TABLE III .
Results of mixed factorial ANOVA for all three AOI types for full problem period.* indicates an interaction of the effects of problem number and correctness of answer.

TABLE IV .
Mean percentage time spent (AE standard error) and results of one-way ANOVA during entire problem period for thematically relevant, novicelike, and perceptually salient AOIs for participants who answered the question correctly or incorrectly.

TABLE V .
Results of mixed factorial ANOVA for all three AOI types for first 2 sec of viewing the diagram.* indicates an interaction of the effects of problem number and correctness of answer.