Eye-gaze patterns as students study worked-out examples in mechanics

Adam D. Smith,1 Jose P. Mestre,1,2,3,* and Brian H. Ross4,3 1Department of Physics, University of Illinois, Urbana, Illinois 61801, USA 2Department of Educational Psychology, University of Illinois, Urbana, Illinois 61820, USA 3Beckman Institute for Advanced Science and Technology, University of Illinois, Urbana, Illinois 61801, USA 4Department of Psychology, University of Illinois, Urbana, Illinois 61820, USA Received 25 March 2010; published 7 October 2010


I. INTRODUCTION
Teaching in the physical sciences and engineering follows a fairly standard format: concepts and problem solving techniques are covered in class, students are given practice solving problems on their own for homework, and tests are used to assess their proficiency at solving problems.If asked, physics instructors would profess that they would like their students to acquire both a deep conceptual understanding of the material as well as the ability to solve problems.Yet, circumstances usually lead students to focus on problem solving at the expense of conceptual understanding.We tend to evaluate performance in terms of problem solving, likely because this is easier to do than designing measures of "deep conceptual understanding."Consequently, when physics instructors are asked about their students' knowledge, they state that, although the problem solving prowess of their students can range from tolerable to very good, the conceptual understanding of the major ideas in the course is often lacking-a view that is confirmed by conceptual measures such as the Force Concept Inventory ͓1,2͔.In discussion sections for introductory physics at the University of Illinois, where students work collaboratively on difficult, conceptrich problems, asking students to state the "big idea" used in a problem that they have just solved is typically met with blank stares or with answers that point to the equations used.The primary goals of the current paper are to analyze: ͑a͒ what students attend to during a typical science learning activity, namely, studying worked-out examples, in terms of the time they spend processing conceptual and mathematical information contained in the examples, and ͑b͒ whether in-structions given to students about the goal of the worked-out example study activity impact the relative attention paid to conceptual or mathematical information, as well as what they learn from the study activity.A secondary goal is to explore what conceptual and problem solving knowledge is learned from studying the examples.
Some programs have shown that, with appropriate structuring and coaxing, students will learn conceptual knowledge alongside problem solving skills.For example, Van Heuvelen ͓3͔ taught an introductory mechanics course with a two-pass approach, first conceptual and, only after students had gained a conceptual understanding of the material, a second pass through the same material using mathematical procedures.This approach, implemented using collaborative group learning during class, was more effective than conventional approaches as measured by conceptual and problem solving measures.Later, Gautreau and Novemsky ͓4͔ demonstrated that the approach was very effective with underprepared minority students.Another series of studies ͓5,6͔ found improvement if novices practiced with a simple menu-driven computer tool that constrained them to follow a conceptual analysis prior to solving mechanics problems ͑similar to that seen with experts ͓7,8͔ ͒.Other work ͓9͔ required students to write conceptual analyses including the major principle͑s͒, a justification for why the principle applied to the specific context, and a procedure for applying the principle.Students in an introductory class were better on later problem categorization and displayed better long-term retention months after the course was over as measured by ability to identify the most important physics ideas used to solve mechanics problems.These results are consistent with a variety of research in cognitive science showing that interventions that highlight the deep conceptual structure of a problem facilitate student learning and transfer including work on schema acquisition ͓10-14͔ and explanation ͓15-17͔.
Here we take a very different approach.Rather than trying to integrate various research on learning and problem solving into a new instructional technique, we begin an analysis of the current means of learning to investigate more carefully how students process information.In this paper, our focus is on how students process information during an activity that is both common and popular among students in learning science, namely studying worked-out examples.
Worked-out examples are an integral part of learning in problem solving domains.Class lectures often include some problem solutions, which are often similar to worked-out examples.Textbooks often teach new concepts through worked-out examples and these examples are referred to often when students are solving problems for homework.In addition, when studying for exams, students may go back over the text's worked-out examples, as well as ones from lecture or their homework, testing their understanding.
Although most instructors believe that having students actively solve problems is a great way to learn, there is evidence that it is not and that worked-out examples may be more effective for learning to solve problems and understanding the conceptual structure.Sweller, Mawer, and Ward ͓18͔ have conducted a careful analysis of how novices solve problems and claim that assigning well defined problems for students to solve on their own promotes means-ends analysis, whereby the problem solver attempts to reduce the difference between the current state in the solution and the goal state.They argue, using a variety of empirical data ͓19,20͔ and a theoretical model ͓21͔, that means-ends analysis as a solution strategy is a very memory-load intensive strategy that leaves few resources left for either learning how to solve problems of that type ͑reflecting on the solution͒ or for perceiving the problem's underlying structure.Problem solving also often leads solvers to spend significant time on unfruitful paths, which are then more likely to be tried in subsequent problem solving as well.
Worked-out examples appear to overcome some of the difficulties in learning from problem solving ͑also see Ref.
͓18͔ for other suggestions͒.They train students on the correct solution path, do not have them spending time in unfruitful approaches, and often provide some additional information about why these steps are chosen.In addition, students may self-explain the connections between steps, further promoting their understanding ͓15,22͔.Contrary to intuitions of some instructors ͑including the authors͒, worked-out examples promote better learning in less time than does problem solving ͓23͔.It may still be useful to have students practice actively solving problems, but research findings indicate that such practice is more effective following the presentation of some worked-out examples of the same type ͓23͔.
Despite the ubiquity and effectiveness of worked-out examples, we know little about what students focus on when they read them.It could be argued that students are expected to glean how conceptual knowledge is used to solve problems by following the textual information accompanying worked-out examples in textbooks.All worked-out examples in textbooks illustrate how conceptual knowledge is applied to solve problems by interleaving descriptions and explanations of concepts and procedures with equations and mathematical steps.On the other hand, students rarely read physics textbooks before attending lecture ͓24͔ and, when they do, it may be that they skim over the text and focus on the equations.Problem solving is the main way in which understanding is tested, leading students to focus on problem solving related information, which they often interpret as the equations.Other than some studies that have explored how students self-explain what they think is going on in workedout examples ͓15,22͔, we are not aware of a study that has explored what students focus on ͑or do not focus on͒ while processing a worked-out example.Do students just ignore the concept-laden text written around the equations and focus only on the equations, which likely results in processing solutions in terms of means-ends strategies in the minds of students?Alternatively, if students do not ignore conceptual information, how do they combine their processing of the text and equations?
The study reported here explores these issues.We provided science and engineering undergraduates with workedout examples of mechanics problems and, using an eyetracker, recorded eye-gaze patterns of students as they learned from reading the examples.The worked-out examples were presented in a two-column format, with one side containing the conceptual information ͑similar to the subgoal analysis of Catrambone ͓25͔͒ and the other side the mathematical equations.This format made it easy to tag the amount of time students spent on both conceptual and mathematical information, as well as the sequence of gazes through the examples.We further explored whether gaze patterns differed under two different conditions that students experience in real courses, trying to solve a particular problem and studying for a quiz.In the Homework condition students were initially shown a difficult target problem, told they would need to solve it eventually, and that the subsequent worked-out examples would help them solve the target problem.In the Quiz condition students were told that they would be asked to solve a later problem ͑not yet given͒ and that the worked-out examples they were about to study would help them solve it.Subjects in both conditions were given target problems to solve and a memory recall assessment to determine how well they remembered some conceptual information presented in the examples.

II. EXPERIMENT
The experimental design was driven by three research questions, the first two defined our primary goals and the third our secondary goal.Given that students rarely talk about the conceptual content of the problems they solve, to what extent ͑if any͒ do they attend to conceptual information when studying worked-out examples?Does the purpose for learning from the worked-out examples ͑i.e., solving a prespecified target problem in the Homework condition versus solving an unspecified problem in the Quiz condition͒ impact students' processing of conceptual information?In terms of recall and learning from the worked-out examples, are students able to solve a target problem following study of the examples, and are they able to recall conceptual information contained in the examples?
Addressing these questions requires measurements of where and for how long students look when studying worked-out examples.Ideally, we would measure where stu-dents are looking without their knowledge in a naturalistic setting.In practice, tracking the location of their eye gazes demands an initial calibration requiring the subjects' knowing participation.Given that the subjects are aware that their gaze pattern is being recorded, we seek to make the eyetracking experience as nonintrusive as possible in order to obtain a more natural gaze pattern.Using a Tobii 1750 eye tracker makes this possible.In this system, the subject sits at a computer monitor while infrared beams illuminate the eyes.An infrared camera records eye images that a desktop computer processes to generate gaze focus coordinates.Since the system can tolerate fairly large head movements, subjects are completely unrestrained and free to view the onscreen stimuli unencumbered by headgear or a chin rest.The subject's experience is essentially identical to viewing webpages on a computer screen.
Although commonly used in cognitive science to study reading ͓26͔ and other cognitive processes ͓27͔, eye-tracking studies in physics education research ͑PER͒ have been rare.Recently, a study by Rosengrant, Thomson, and Mzoughi ͓28͔ had a small number of novices and two experts answer questions about circuit diagrams which incorporated eyegaze patterns.In contrast to novices, the eye-gaze pattern of experts showed shifting back and forth between their written work and the circuit diagram, suggesting integration of the solution and diagram; experts' gaze patterns also focused on individual resistors that could be combined as well as the path of the current suggesting global processing of multiple facets of the circuit, whereas novices only looked at proximal resistors that could be combined using the series and parallel rules.Other eye-tracking studies have touched on problems and issues peripherally related to PER, such as troubleshooting of malfunctioning circuits ͓29͔, comprehending malfunctioning mechanical devices ͓30͔, and comprehending how mechanical systems work ͓31͔.One recent eye-tracking study ͓32͔ focused on differences among novices of high and low spatial ability in solving simple kinematics problems.However, only spatial ability was contrasted as opposed to conceptual or problem solving expertise in physics, so it is difficult to infer implications for physics education.

Method
Eye-tracking technology is an excellent tool to observe subject gaze patterns.However, merely recording where students look when studying an example would provide little insight into the underlying factors influencing their gaze patterns.The power of an eye-tracking experiment resides in coupling the comprehensive recording of eye positioning with one or more experimental manipulations to inform inferences about the underlying factors driving subject gaze patterns.While this methodology lacks the richness of a think-aloud protocol, it does allow the researcher to observe how subjects naturally look at a stimulus without the added cognitive load of explaining themselves verbally.
In order to determine whether the purpose of instructional activities changes the degree to which students look at conceptual information encoded as text, we framed the studying task in two ways.Over the course of an hour each subject repeated a single protocol twice, seeing a total of four worked-out examples and working two target problems.At the end of the session, subjects were given a memory recall assessment evaluating how well they remembered information from either the textual or mathematical columns.
We used two dependent measures to explore learning following the study task.The first was ability to solve the target problem; the second was a memory recall assessment evaluating how well they remembered some conceptual information contained in the textual column.
Formatting of the worked-out examples the subjects would study was an important consideration.In most worked-out example formats, explanations of what concepts are being used and why they are being applied are contained in prose interspersed between mathematical steps.We formatted the examples in two columns.One column presented textual explanations of the current step in the solution while the other displayed corresponding mathematical operations.This formatting makes a clear spatial contrast between reading purely mathematical information and reading the textual explanations containing conceptual information.Such spatial separation facilitates the tracking of eye-gaze transitions between reading textual and mathematical information.The solution was further divided into steps using dashed lines surrounding corresponding textual and mathematical information.The problem statement and an accompanying diagram were presented above the two columns of the solution.A sample worked-out example appears in Appendix A.
In an attempt to control for any processing bias associated with reading from left to right, we counterbalanced the order of the textual and mathematical columns.We also counterbalanced the ordering of the examples.Subjects were randomly assigned to receive examples with the textual information contained in the left or right columns.Subjects were also randomly assigned to a particular ordering of the examples presented.Each subject saw only one format and one ordering of the examples.As an additional precaution, we ensured that all subjects' self-reported native languages were read left to right.Of the 43 subjects, 32 ͑74%͒ were native English speakers with the remaining 11 having first learned a language read from left to right.One of the non-native English speakers had spoken English for three years; all others had spoken English for ten years or more.
Subjects were drawn from University of Illinois' calculusbased electricity and magnetism course, which has as a prerequisite a calculus-based mechanics course that subjects completed the previous semester.Subjects were compensated $10 for participating.Each of the 43 subjects was assigned to either the Quiz or the Homework experimental condition.To assure that the groups were equivalent with respect to performance in the previous mechanics course, we used a matched random assignment scheme.This process involved selecting the subjects with the highest two scores in the previous calculus-based mechanics course and randomly assigning one to each of the two conditions.This procedure was repeated with successive pairs of subjects ordered by course performance until all were assigned to a condition.

III. RESULTS
The results are organized around the three research questions driving this study.Do students attend to the conceptual information encoded in the text of worked-out examples?To what extent does the stated purpose of the studying affect the way students process this conceptual information?Did students learn problem solving and conceptual knowledge from studying these worked-out examples?

A. Do students attend to conceptual information?
Our experience when discussing problem solving with students in our physics classrooms is that they rarely refer to or talk about the conceptual information in the problems they solve or study.Our expectation from this experience was that subjects would spend little time looking at the textual portions of the examples.Our initial goal was therefore to determine whether students incorporate conceptual information encoded as text into their studying of worked-out examples.
To answer this global question, we aggregated subjects from both the Quiz and Homework conditions into a common pool for analysis.
We use two measures to quantify the contribution of text in a subject's overall gaze pattern.The first is the fraction of gaze fixation time spent looking at textual columns.This provides an indication of whether students are even attempting to process the textual information.A second set of measures quantify the pattern of transitions between textual and mathematical information.These transition frequencies indicate to what extent students are attempting to integrate information from both sources together.
The examples' two-column format allowed textual and mathematical explanations to be divided into separate regions of interest.Figure 2 depicts the regions of interest schematically.Each example contained a problem statement ͑P͒, a diagram ͑D͒, and between four and five solution steps containing textual ͑T1-T4͒ and mathematical ͑M1-M4͒ information ͑see Appendix B͒.The individual textual and mathematical blocks may be aggregated into single textual ͑T͒ and mathematical ͑M͒ columns to obtain the global impact of each type of information on a subject's processing pattern.
Figure 3 shows the mean percent of the total text-math fixation time spent in the text and mathematical regions of the examples.Since the text percentage frequency distributions were essentially normally distributed, the error bars in the figure are the standard error of the mean.While subjects spend the majority of their fixation time looking at the mathematical sections of the examples, they still spend 40Ϯ 3% of their fixation time in the textual explanations.
Contrary to our expectations, subjects clearly spend a significant portion of their time reading the textual explanations.They are therefore processing two sources of information, the textual and the mathematical explanations of the pre-  I.
While all transitions are shown for completeness, the relevant comparison for the present discussion is the relative frequency of transitions between the textual and mathematical columns ͑bold entries in Table I͒ compared to those within textual and mathematical columns ͑italic entries in Table I͒.Combined text to mathematics and mathematics to text transitions occur nearly as often as the combined transitions within text and mathematics.
Since subjects appear to look frequently between the text and mathematics, we wish to determine whether these transitions are primarily between corresponding text and symbol blocks or part of a more global scanning pattern.We use the fine-grained regions of Fig. 2͑a͒ and label horizontal transitions ͑e.g., T1 ↔ M1͒ as corresponding transitions.That is, these transitions represent looking between textual and mathematical information describing the same step in the solution.Additionally, we label as sequential those transitions that end at the right side of one line and move to the start of the next, i.e., a "carriage return" for reading left to right.All other transitions are labeled other.
Figure 4 presents the mean number of text-mathematics transitions per subject per slide.The vast majority of transi-tions occur between related text and symbol sections, i.e., sections describing the same step in either words or mathematics.For comparison, a similar analysis of transitions among mathematical regions ͑e.g., M1 ↔ M2͒ revealed 80% of transitions between adjacent regions with 20% among nonadjacent regions.Subjects seem to be processing the textual and mathematical information together, as opposed to treating them as two unrelated sources of information.

B. How does the stated goal of studying impact conceptual text processing?
Given that subjects do integrate text explanations containing conceptual information into their processing of workedout examples, we now determine the extent to which the purpose of the studying affects the way in which students process text.This section explores the effects of the Homework and Quiz studying conditions.
Our findings indicate no significant differences between conditions in the allocation of time between text and mathematics.Figure 5͑a͒ presents the percentage fixation time in text for each condition.Both groups spent a majority of their time looking at the mathematics.While the Homework group appears to spend a larger fraction of time on text compared to the Quiz group, the difference is not statistically significant ͑t = 1.56, df = 39.0, p = 0.13, 2-tailed Welch test͒.Figure 5͑b͒ shows the same data grouped by example.For every example, the Homework group had larger text time fractions than the Quiz group.While the individual differences are not  statistically significant, the consistently larger text time fractions suggest that a larger sample size might show greater emphasis on the text in the Homework condition.

C. Performance on measures of learning
We also analyzed subjects' ability to solve the target problem presented after studying each pair of worked-out examples.Each problem required two principles outlined in the worked-out examples and was scored for the correct application of those principles.Since we are interested primarily in how well subjects can transfer the concepts from the examples to the target problem, two points were given for each correct principle used with an additional point for correctly expressing it mathematically.This makes six points the maximum for each problem.
Figure 6 summarizes the problem solving performance for both conditions.Homework condition subjects saw the target problem before studying and performed significantly better on Target Problem B, which followed worked-out examples 3-4 ͑t = 2.17, df = 43.8,p = 0.04, 2-tailed Welch test, effect size 2.2͒.This is not surprising, as they were presumably thinking about the target problem while studying the examples.
Target Problem A, which followed worked-out examples 1-2, showed no performance difference between experimental conditions ͑t = 0.43, df = 41.1, p = 0.67, 2-tailed Welch test͒.Scores were near 50% because all but one subject applied only energy conservation when momentum conservation was also necessary.While this point was embedded in one of the worked-out examples studied, it did not transfer to this very difficult target problem.
In addition to the target problem solution task, we also administered three memory recall questions to each subject at the end of the experiment asking about conceptual information contained in the textual description of the worked-out examples.The goal was to explore any relationship between time spent reading text and subjects' ability to recall the conceptual information contained therein.A sample assessment question appears in Fig. 7.
Recall assessments were scored using a binary correct or incorrect scheme.Responses that referenced the ideas or specific language contained in the relevant example text explanations were scored as correct.Table II shows that subjects in both the Homework and Quiz conditions performed equally poorly on the recall assessment.Overall, there was no correlation between recall assessment score and either percent text fixation time ͑Pearson's r 2 = 0.024͒ or total fixation time ͑r 2 = 0.021͒.It is possible that those few subjects who did answer the questions correctly knew how to apply the concepts before reading any of the examples.While poor conceptual recall performance is consistent with our expectations from classroom interactions, this result is surprising given that students spend a large fraction of their time reading the text, which contains explicit answers to the conceptual questions.

IV. DISCUSSION
The current study was motivated by our classroom experiences of student problem solving behavior.Students rarely discuss with one another, or with the instructors, the conceptual underpinnings of problem solutions.First, we sought to determine the extent to which students pay attention to the  Findings indicate that subjects spent a significant fraction of their time looking at the textual explanations ͑containing conceptual information about the solution͒ of the worked-out examples.Additionally, the strong majority of their gaze transitions were between blocks of mathematics and text that explain the same solution step.These behaviors were the same across both the Homework and Quiz conditions.The implication is that students do incorporate the textual explanations as an integral part of their approach to studying worked-out examples.This is in direct contrast to our expectations from classroom experience.When asked about conceptual information applied in their problem solutions, students usually discuss the equations they used and not the concepts underlying them, a finding that is consistent with other studies ͓33͔.Thus, we had expected that students would largely ignore the text in processing worked-out examples, but findings from this experiment indicate otherwise.
Study condition did not significantly influence the time spent on textual information.Whether students were told that  How did we know that momentum could be conserved along this direction?
they would be given worked-out examples to study to prepare them for a problem they would solve afterwards ͑Quiz condition͒, or shown a target problem first and told that they would be shown worked-out examples that would help them solve the target ͑Homework condition͒, did not influence the relative time they spent on textual and mathematical information.There was a significant effect of ability to solve one of the two target problems, with the Homework group performing significantly better than the Quiz group.Despite students spending significant time processing the textual information containing a discussion of conceptual knowledge in generating problem solutions, performance on the conceptual questions suggests that students did not retain the conceptual information contained in the worked-out examples.This could perhaps be attributed to the result of their previous instruction; if conceptual content is not evaluated as a part of instruction, then it is likely that students will not attend to it.Alternatively, perhaps students need to reach a certain level of experience in a domain before they begin to understand the role of conceptual knowledge in problem solving.Another possibility is that students are using the text for a task other than gleaning conceptual insight into the steps of the solution.For instance, they may be trying to link particular mathematical statements with technical physics terms contained in the text.
If subjects had spent little time reading the textual information, then their poor recall of conceptual information would be easily explained.The surprising result is that they do not remember conceptual information despite spending significant time reading the text containing it.Subjects did not simply ignore the text; text regions were fully coordinated into their reading and processing of the examples as evidenced by their transition patterns and fixation times.Their poor conceptual recall suggests they either did not learn the conceptual information in the first place or did not retain it after reading the example.

V. CONCLUSION
In contrast to our expectation from the classroom observations, we find that students do incorporate textual informa-tion as an integral part of their studying of worked-out examples.Time allocation to textual or math information was not affected by the framing of the studying purpose to model either a quiz or a homework condition.Despite spending significant time reading the textual information and combining it with the corresponding mathematical statements, students still had very poor recall of the high level conceptual information the text contained.The extent to which this inability to recall conceptual information is due to framing of the task, lack of requisite expertise needed to process and retain high level conceptual knowledge, or simply using text for lower level processing ͑e.g., English or Physicsmathematics translation͒ remains to be determined.

Figure 1
Figure 1 summarizes the two experimental conditions.The Quiz condition was inspired by a quiz scenario where students study worked-out examples to prepare to solve an unknown problem that is presumably related to the presented examples.These subjects were told that studying the examples should help them solve a problem later in the session.The Homework condition was inspired by a homework assignment where students may study examples in a textbook with the goal of solving a problem they currently have in mind.These subjects were presented with a target problem to read before studying the worked-out examples.Just as in the Quiz condition, these subjects were told that the examples should help them solve the target problem.We note that the designations Homework and Quiz are simply for convenience.The tasks themselves are administered in a controlled, clinical setting.The environment is therefore significantly different from a quiz or homework assignment given in the context of a standard physics class.Over the course of an hour each subject repeated a single protocol twice, seeing a total of four worked-out examples and working two target problems.At the end of the session, subjects were given a memory recall assessment evaluating how well they remembered information from either the textual or mathematical columns.We used two dependent measures to explore learning following the study task.The first was ability to solve the target

FIG. 2 .
FIG. 2. ͑Color͒ Worked-out example regions of interest.Textual ͑T1-T4͒ and mathematical ͑M1-M4͒ information in the solution were spatially separated beneath the problem statement ͑p͒ and diagram ͑d͒.Two granularities were used, treating each step either individually ͑a͒ or aggregated into textual and mathematical columns ͑b͒.

FIG. 3 .
FIG. 3. ͑Color͒ Mean percentage fixation time by region using aggregated columns of Fig. 2͑b͒.Error bars show the standard error of the mean.

FIG. 5 .
FIG. 5. ͑Color͒ Mean percentage fixation time in textual explanations.The group means ͑a͒ are presented for the Homework ͑hatched͒ and Quiz ͑solid͒ conditions.Means are additionally grouped by each individual example ͑b͒.

FIG. 7 .
FIG. 7. Sample conceptual recall question.The example problem statement is shown to the subject, but the two-column solution is removed.Note that the answer to this question is contained in the second textual statement on the left column of the worked-out example shown in Appendix B.

FIG. 8 .
FIG. 8. Sample worked-out example presented to subjects.The text and mathematics are arranged into two columns and grouped into steps with dashed lines.

TABLE I .
Mean number of inter-region transitions per subject per example.The regions, pictured in Fig. 2͑b͒, are the problem statement ͑P͒, diagram ͑D͒, text column ͑T͒, and mathematics column ͑M͒.FIG. 4. Mean transitions grouped by type.Categories are between Corresponding textual and mathematical information, Sequential information, and all Others.

TABLE II .
Percent of subjects with correct memory recall responses pertaining to conceptual information encoded in text.Each of the three recall questions referred to a different worked-out example.The question text is included to make clear what is meant by a conceptual question.