Changes in students’ problem-solving strategies in a course that includes context-rich, multifaceted problems

Most students struggle when faced with complex and open-ended tasks because the strategies taught in schools and universities simply require finding and applying the correct formulae or strategy to answer wellstructured, algorithmic problems. For students to develop their ability to solve ill-structured problems, they must first believe that standardized procedural approaches will not always be sufficient for solving engineering and scientific challenges. In this paper we document the range of beliefs university students have about problem solving. Students enrolled in a physics course submitted a written reflection both at the start and the end of the course on how they solve problems. We coded approximately 500 of these reflections for the presence of different problem-solving approaches. At the start of the semester over 50% of the students mention in written reflections that they use Rolodex equation matching, i.e., they solve problems by searching for equations that have the same variables as the knowns and unknowns. We then describe the extent to which students’ beliefs about physics problem solving change by the end of a semester-long course that emphasized problem solving via context-rich, multifaceted problems. The frequency of strategies such as the Rolodex method reduces only slightly by the end of the semester. However, there is an increase in students describing more expansive strategies within their reflections. In particular there is a large increase in describing the use of diagrams, and thinking about concepts first. Disciplines Physics | Scholarship of Teaching and Learning Comments This article is from Physical Review Physics Education Research 5 (2009): 020102, doi:10.1103/ PhysRevSTPER.5.020102. Posted with permission. Rights This article is available under the terms of the Creative Commons Attribution 3.0 License. Further distribution of this work must maintain attribution to the author(s) and the published article’s title, journal citation, and DOI. This article is available at Iowa State University Digital Repository: http://lib.dr.iastate.edu/physastro_pubs/217 Changes in students’ problem-solving strategies in a course that includes context-rich, multifaceted problems C. A. Ogilvie Department of Physics and Astronomy, Iowa State University, Ames, Iowa 50011, USA Received 5 September 2008; published 28 August 2009 Most students struggle when faced with complex and open-ended tasks because the strategies taught in schools and universities simply require finding and applying the correct formulae or strategy to answer wellstructured, algorithmic problems. For students to develop their ability to solve ill-structured problems, they must first believe that standardized procedural approaches will not always be sufficient for solving engineering and scientific challenges. In this paper we document the range of beliefs university students have about problem solving. Students enrolled in a physics course submitted a written reflection both at the start and the end of the course on how they solve problems. We coded approximately 500 of these reflections for the presence of different problem-solving approaches. At the start of the semester over 50% of the students mention in written reflections that they use Rolodex equation matching, i.e., they solve problems by searching for equations that have the same variables as the knowns and unknowns. We then describe the extent to which students’ beliefs about physics problem solving change by the end of a semester-long course that emphasized problem solving via context-rich, multifaceted problems. The frequency of strategies such as the Rolodex method reduces only slightly by the end of the semester. However, there is an increase in students describing more expansive strategies within their reflections. In particular there is a large increase in describing the use of diagrams, and thinking about concepts first. DOI: 10.1103/PhysRevSTPER.5.020102 PACS number s : 01.40. d


I. INTRODUCTION
2][3] The challenges facing our workforce are often ill-structured [4][5][6][7] that might contain unclear goals, insufficient constraints, multiple alternative options, and different criteria for evaluating proposed solutions.Most students struggle when faced with these complex and unstructured problems because the problem-solving strategies taught in schools and universities simply require finding and applying the correct formulae or strategy to answer well-structured, algorithmic problems.Solving complex real-world problems requires deep, organized conceptual understanding, relevant procedural knowledge, and metacognitive strategies that allow one to formulate potential solution strategies, implement a course of action, and reflect on the viability of their solution from multiple perspectives.As educators, it is vital that we help students develop and practice these stronger approaches to more complex problems.
Insight into how to make progress on this educational challenge comes from research on how experts and novices approach complex tasks. 8Experts have strong organized conceptual knowledge in the domain, 8,9 so they can first qualitatively analyze problems to quickly determine the main essence of the task. 10,11This avoids distraction due to surface features or fine details of the problem that will not be needed until later in the solution.Experts also have stronger metacognitive skills, 10 including monitoring the progress of their solution to check whether their chosen path is still potentially fruitful, as well as evaluation skills such as testing the solution against assumptions made, and using extreme conditions to check the solution's validity.Strong problem solvers also incorporate the experience gained from each problem into their ever-deepening knowledge structure that can be drawn upon when confronted with new ill-structured problems. 12n contrast, many students believe that problem solving is being able to apply set procedures or algorithms to tasks 13 and that their job as students is to master an ever-increasing list of procedures.This gap between students' beliefs and the broader, deeper approaches of experts is a significant barrier to preparing students to succeed in their future careers.For students to develop their ability to solve ill-structured problems, they must first believe that standardized procedural approaches will not always be sufficient for solving illstructured engineering and scientific challenges, and that it is these complex tasks that they need to prepare for during their time at university.
Can an introductory STEM course impact students' beliefs about problem solving and hence set them on a path of ever-increasing skill development?This is the main research question for this paper: to assess the extent that students' beliefs about problem solving change after participating in a course in which they work on ill-structured problems.

II. BACKGROUND RESEARCH ON EPISTEMOLOGY
Student epistemology is defined as the implicit assumptions and beliefs held by students about the nature of knowledge and learning ͑see reviews by Hofer, 14 Hofer and Pintrich, 15 Muis, 16 and Schraw 17 ͒.Multiple categories of student beliefs have been distilled from many studies, interviews, surveys and three of these are summarized by Schommer 18,19 as ͑a͒ certainty of knowledge: where students range from a belief that knowledge is fixed to a belief that knowledge can develop and improve, ͑b͒ source of knowledge: where students believe that knowledge comes from authority figures or that knowledge is developed by personal and collective effort, and ͑c͒ the structure of knowledge: where beliefs range from knowledge being isolated fragments to interrelated ideas.There are reported differences within these categories between students majoring in different disciplines, and a given individual may simultaneously hold different beliefs about knowledge as they think and work in different disciplines. 16,20,21As examples, validated survey instruments have been developed to assess students' beliefs toward the nature of science, VNOS, 22 as well as beliefs about learning physics: CLASS, 23 MPEX, 24 and in chemistry. 137][28][29][30] For example, Paulsen and Feldman 31 have established that student beliefs help determine the specific strategies they use to learn.Students who believe that the ability to learn is fixed and hard to change are less likely to regulate their time.These students disagree with statements such as "even when course materials are dull and uninteresting, I manage to keep working until I finish."They are also unlikely to optimize their environment for learning, and are less likely to work with peers or to seek help. 31Beliefs about knowledge are also well correlated with student achievement [32][33][34] though a causal relationship is difficult to establish.As an example, Mason 34 demonstrated that students' belief that they can solve time-consuming problems was strongly correlated with their academic performance.
Within student epistemology research there have been many studies about students' attitudes toward problem solving.In mathematics, Schoenfeld 10 videotaped students solving geometry problems and found that students do very little planning, they apply memorized procedures without considering their appropriateness to the situation, and they rarely monitor their progress as the solution becomes more complex, i.e., they carry on with work even if their work has diverged from the original goal.He conjectured that the root cause of this behavior was students' beliefs about mathematics and mathematical problem solving.In a follow-up study Schoenfeld 35 found that high school students believed that understanding mathematics meant being able to solve problems in 5 min or less, and that one succeeded in mathematics by performing tasks exactly as instructed by the teacher.This prevailing belief that problem solving is following a set of prescribed rules has also been documented by Brown et al. 36 and Garofolo 37 in middle school and high school.Such beliefs lead students to search the textbook for procedures without applying reasoning as to which methods were most appropriate. 37Encouragingly, in a recent study of middle school students Schommer-Aikins et al. 38 showed that the less students believe in quick or fixed learning and the more they believed that math was useful, then the better they performed on mathematical problems.
In physics, the problem-solving challenge for students is to use their conceptual understanding of both physics and mathematics to solve quantitative problems.Instead of using this understanding-based approach, Larkin 39 has documented that physics students expect to solve problems by searching for an equation that simply contains the same variables in the problem statement.Students with a prevailing belief that problem solving is being able to apply procedures are also more likely to set a short, maximum time limit to a task and will stop working in case of difficulties. 40In addition, physics students who believe that knowledge is a series of isolated facts spend their study time memorizing facts, 40 rather than building an organized structure of concepts they could draw on as they attempt to solve novel problems. 8There is however some evidence from university genetics courses that more successful novices recognize that they should develop a solution that is both internally consistent and externally valid with respect to the rest of their domain knowledge. 41imilar results were found by May and Etkina 42 who reported that students who focused on constructing a coherent structure of physics concepts showed a larger gain on conceptual tests pre-to postinstruction than students who approached their learning tasks in a rote fashion.The lack of coherence in student knowledge has been documented by Lising and Elby 43 who report that students believe that formal and everyday understanding operate in different aspects of their lives and that there is no need to reconcile them if they conflict.This belief in separate spheres of applicability is a strong barrier to improving students' conceptual understanding.For example, in topics such as force and motion, the non-Newtonian misconception that a force is required to maintain motion is derived from everyday experience and needs to be replaced and reconciled with the correct and more expansive knowledge that forces cause a change in motion. 44he origin of these beliefs toward problem solving may be rooted in class-room instruction.Schoenfeld 45 reported that the problems students are asked to solve in K-12 classrooms are rarely open-ended challenges, but exercises in familiar tasks, with an emphasis on completing these tasks quickly and efficiently.Similar observations have been made by Doyle 46 who analyzed the tasks addressed inside mathematics classrooms: teachers predominantly asked students to solve familiar work rather than novel challenges.

A. Prior research on how to change students' beliefs
A key challenge is how to change the beliefs of students so they can develop the skills and approaches needed to solve open-ended, ill-structured problems.Elby 47 and Hammer 48 have proposed infusing explicit discussions about epistemological beliefs throughout a course.These proposals were made in response to the observations made by Redish et al. 13 who documented that students' beliefs in the structure of physics knowledge often worsen as the result taking physics courses, even reformed courses that improve student conceptual understanding.However it is not clear that having students discuss epistemology in their science courses will be sufficient.Bendixen 49 has suggested that conceptual change research could be used to guide instructors on how to impact student beliefs about knowledge and learning.Bendixen argues that students must realize that something in their beliefs is impacting their ability to make progress, i.e., they must acknowledge there is a need to change.Then the new strategies or beliefs must make sense to the students and they must see how to apply these ideas, and finally the students need to experience that the new beliefs produce some success.Similar proposals for achieving epistemological change have been proposed by Baxter Magolda, 50 Kitchener et al., 51 Kloss, 52 Chin and Brewer, 53 De Corte, 54 Kuhn, 55 and Felder and Brent. 56nterventions based on creating this sense of "epistemic doubt 16 " in students have helped to change student beliefs about problem solving.Higgins 57 studied the beliefs of middle school students after they were challenged with openended mathematical problems.Key to creating doubt is the lack of success that a student experiences if s/he tries to find an existing algorithmic procedure for these ill-structured problems.This lack of success, coupled with support for the new strategies, has the potential for students acknowledging a need to change their beliefs and experiencing some success with their new fledgling strategies.In Higgins' study 57 students worked on complex problems over the course of a week while receiving support from the teacher and engaging in classroom discussions on different problem-solving strategies.Compared to a control group, students at the end of the year reported stronger beliefs that mathematics was more than memorizing facts and procedures, and that they could develop knowledge and understanding through their efforts rather than relying on an authority figure.Similar success in improving students' beliefs by using nonroutine, complex problems was reported with elementary and middle school children by Verschaffel, 58 Mason and Scrivani, 59 and Liu. 60n addition, research shows that internships 20 and openended project-based 27,61 experiences taken by college students have impacted students' beliefs about problem solving.The authors in these papers conjecture that the ambiguous context of the tasks caused the students to think about their STEM knowledge in more complex ways.

B. Creating epistemic doubt in large introductory university courses
For large introductory science courses it is difficult to implement project-based experiences 27,61 due to large staffing and time demands.An alternative that may impact students' beliefs on problem solving and still fit within a course structure of lectures and recitations is the use of multifaceted problems. 62Multifaceted problems lie somewhere between well-structured problems found in textbooks and large, illdefined, open-ended challenges in the degree-of-difficulty these pose to students.Multifaceted problems require students to integrate multiple concepts in building a solution.Typically these problems also place the student in the middle of a challenge, for example, "you are a design engineer for a company that has been asked to build a ski ramp."However, the main characteristic is that the problems involve more than one concept, hence students cannot readily use a direct algorithmic approach as in a classical textbook exercise.An example of a multifaceted thermodynamic problem is: You are in charge of drinks at a picnic that will start at 3pm.You place ice inside a cooler at 6am, when the temperature outside is 10 °C.The day is forecast to warm up steadily to reach 30 °C by 3pm.Estimate how much ice you will need.
At least two concepts are involved: heat transfer through a wall, and the amount of heat required to melt ice.It is also moderately ill structured, in that the problem statement does not specify the wall thickness of the cooler, or the material used.Students must identify that they need these quantities for a final solution and then find or estimate that information.Although the problem is not mathematically complex, it does require that students ͑if working in a group͒ need to discuss the problem, identify the main concepts that are involved, qualitatively analyze the problem, find or estimate the required information, and from there build a solution.
Multifaceted problems have been advocated by several groups across many disciplines for both school and university use.Within physics this pedagogy has been developed by the physics education research groups at University of Minnesota 62 and Ohio State University. 63Similar pedagogy has been used in chemistry, 11 industrial engineering, [64][65][66][67] and in several disciplines and school levels by the Cognition and Technology Group at Vanderbilt 68 and the IMMEX project at UCLA. [69][70][71] In this paper we describe the extent to which students' beliefs about physics problem solving change after participating in a course that emphasizes problem solving and includes several multifaceted problems.The beliefs students have at the start of the course not only provide our baseline, but also provide insight into the beliefs about problem solving that students have early in their university careers.Our primary research question is whether a course that includes multifaceted problems can create sufficient epistemic doubt in students to change their beliefs about problem solving, or to increase their awareness of the limited usefulness of weaker strategies.

III. EDUCATION CONTEXT
The data presented in this study come from the Spring 2006 semester of a sophomore, calculus-based physics course at Iowa State University.330 students took the course that was taught by the author of this paper.The course met for three lectures each week, one recitation and one laboratory.The active-learning format of the lecture was approximately 10 min of minilecture about an idea, followed by a conceptual question ͑referred to as a concepTest 72 ͒ that the students answered via infrared clickers.The students first answered individually, followed by a group discussion, and then provided an answer as a group.Problem solving was discussed in lectures, mainly by modeling the parts of solving multifaceted problems that students have most difficulty with.Over the semester the lecturer modeled several times how to qualitatively analyze a multifaceted problem, what aspects of a drawing a diagram at the start helped the brainstorming process, and sample questions to ask within your group in order to monitor progress through a multifaceted problem.Where feasible, clicker questions were included in these examples to increase student participation.The recitations used a mixture of Physics Tutorials 73 as the vehicle to develop conceptual understanding of the material and context-rich, multifaceted problem solving 62 designed to increase problem-solving skills.
Each topic in the course followed approximately the same sequence of two to three lectures, an introductory tutorial during recitation to address the main concepts, a laboratory, and two problem sets.The first problem set was due early in this sequence and contained mainly conceptual questions.
The second problem set focused on standard end-of-chapter problems designed to reinforce procedural knowledge in the topic area, i.e., the basic how-to knowledge of solving wellstructured problems, e.g., setting up free-body problems, being consistent in sign conventions for thermodynamic problems, and consistency in current directions for circuit problems etc.Solutions to the problem sets were made available to the students electronically.The solutions often contained a qualitative analysis of the problem and a schematic diagram.The laboratories have been in place for many years and are largely step-by-step procedures modulated by questions that ask for predictions from students and for explanations after specific measurements.
The context-rich multifaceted problems served as a capstone event for each topic area.The problems use both recent material from the course but also ideas from the prior semester ͑statics, dynamics, and torques͒ as well as concepts from earlier in the course, e.g., needing to also use electric potential while solving a problem that primarily focused on magnetic forces.Groups of two to three students worked on these multifaceted problems during their recitation session where approximately 20 students meet with their teaching assistant ͑TA͒.The TA's role was to provide guided instruction on problem-solving strategies, e.g., how to qualitatively analyze the problems, how to work from concepts and diagrams to build a solution.We trained the TA's how to use leading prompts in all their discussions with students.At the start of the semester, this was a challenge for the TAs because their inclination is to provide more direct help, for example, to make a suggestion of an approach or to identify the key constraint in a problem.We trained the TAs how to scaffold: to ask prompts 74 that support the students in the early stages of the semester; such as What information is missing?How are … related to each other?What do you think are the primary factors of this problem?Why is it …?Please explain.
During the semester groups of students worked on six multifaceted problems; two in thermodynamics, one in waves, two in magnetism and magnetic induction, and one on optics.The problem statements for these six problems are given in Appendix A. Twice during the semester student groups also solved a multifaceted problem during a group exam.Their exam work was evaluated based on the problemsolving approaches that they used, e.g., performing a qualitative analysis, rerepresenting the problem with a diagram, their description of ongoing monitoring of the progress of the solution, and final checks of the solution.The two group exams contributed a total of 7.5% toward each student's final grade.

IV. STUDENT SELF-REFLECTIONS
During both the first and last week of the semester students were asked to write a short reflection on their approach to problem solving.The text of the prompt to the students was the same for both reflections:

Please reflect on how you approach physics problems. What methods do you use? What mistakes do you have to watch for? And are these approaches similar to skills that you will need in your future studies or career?
The students received 0.2% of extra credit for each of the two reflections they could submit.Students' self-reports may not match what they believe and that these self-reflections will likely over-report those beliefs that the students know have been a goal of the course.In addition, there is a broad range of students' sophistication in expressing their problemsolving behaviors which makes it a challenge to extract precise meaning from student reflections.Given that we wanted to explore the impact of problem-solving experiences on a large number of students ͑ϳ300͒ it was not viable to perform one-on-one interviews.We also chose to have the students write open-ended responses to the prompt on problem solving, rather than answering survey questions on a Likert scale 6,13 because there are potential educational benefits to students in the act of self-reflection.For example, May and Etkina 42 asked students to write weekly reports reflecting on what and how they learned and found that intentional selfreflection improved students' conceptual understanding.MacGregor 75 has also reported success in improving students' attitudes toward learning by having them intentionally reflect on their learning experiences.
At the start of semester 292 students submitted their selfreflection out of the 331 students originally enrolled in the course.At the end of the semester 224 students submitted a self-reflection, a decrease due in part due to students not completing this optional assignment and in part due to some students dropping the course.216 students submitted both reflections, i.e., one at the beginning and one at the end of the course.These reflections were analyzed coded for key beliefs described by the students.

A. Typology of student reflections
The student reflections contained one or more beliefs about problem solving.Each student reflection was coded by two people, the author of this paper and an experienced physics instructor.The list of student responses was randomly ordered before coding with no information visible to the coders on whether the student reflection was written at the start or at the end of the semester-apart from the occasional tell-tale student comment such as "As we reach the end of the year…."The average inter-rater consistency for the coding was = 0.90.
The student reflections were examined for strategies that they use as they start and work their way through problems.Some student reflections mentioned strategies they use at the end of problems, e.g., reviewing their work, but these were infrequent and are not the focus of this paper.Four categories of "limiting" ideas or problem-solving strategies were identified in student reflections: the phrase "limiting" 76 is chosen because these strategies may work well for well-structured, end-of-chapter exercises, but they begin to fail as the problems become more complex, and will not work for more ill-structured or open-ended problems.From the student reflections we also identified four categories of "expansive" ideas.The term "expansive" was chosen because these approaches to problems can be readily applied to more illstructured challenges, and these strategies have also been identified as characteristic of expert problem-solving approaches.The limiting and expansive categories are listed in Table I and described more fully via representative examples of student reflections are given below ͑without spelling or grammar corrections͒.
Each student reflection may contain one or more ideas about problem solving.For example a reflection may mention both a Rolodex equation matching technique as well as drawing diagrams.Such a reflection was coded as containing both ideas.
Rolodex equation matching.In this strategy students select equations largely because the equations have the same variables as the list of knowns and unknowns.Some illustrative quotes from student reflections are listed below.
"I read through the problem noting the information given.Then I look for a formula that involves these variables.When I approach a physics problem, I first find what the problem is asking for.I then write down all the values that I am given in the problem.After making a list of all the givens, I then find all the equations that have these givens and the answer I'm looking for in them." "At this point I use a fairly systematic process.First, I write down the known values in variable form.͑example B = 1.3 T, i =2A, etc.͒ and then draw a simple diagram involving the known values and the desired value.Then, I write down applicable equations relating the known values and the desired value.If there is a direct connection between the known values and the desired output, I then "plug n chug."If not, I think about how to manipulate equations or to find intermediate values that could lead to the desired value." Listing known quantities.This strategy is categorized as limiting because as problems become more complex the amount of information rapidly expands and not all known information is relevant.For well-structured problems a listing of knowns might be useful since this can focus one's attention on the given information.It could help a student to rapidly access the given information without searching through a lot of text.Also it is possible that some students filter out some information after a first analysis of the problem, yet still describe in their reflection this as "writing down all the knowns."Hence it is difficult to distinguish between students who make a filtered list as an aid to rerepresent the problem and those who write down all numerical values or variables that are provided in the problem.When coding we examined student responses for the presence of an unguided and complete listing of quantities.Also note that there is a moderate correlation between the listing of knowns and the mention of a rolodex method which is an unguided strategy.The correlation coefficient for student reflections in these two categories is 0. We did not include into this category reflections that indicated some discrimination in which quantities were listed."When first approaching a physics problem I admit that one if the first things I do is try to compile or narrow down the information listed." Listing unknowns.This was coded if the student makes a list of unknown variables with little to no focus on what was the goal for the problem.Fewer than 10% of students described this strategy in their reflections.Some illustrative examples are listed below: "I should write down all the known and unknown variables, while figuring out their relationships or interactions.I found that writing out all the information I was given-then used it to find all the new information I could find using equations ͑or if I could see far enough ahead, only what I knew I would need͒." Prior examples in text or lecture.In this strategy students search for similar examples in a text or other resource.From the reflections it is not always possible to identify how a given students uses prior examples, whether they look for problems with similar surface features, or problems that employ the same principles.We coded for this strategy if the reflection described the process of copying or minimally adapting the worked examples to the problem at hand.We note that students whose reflections were coded as searching for prior examples rarely contained mention of more expansive strategies ͑see later in this section͒ such as trying to understand the concepts or qualitatively analyzing the problem.The correlation coefficient between "prior examples" and "concepts" is 0.02 indicating that these two codings are independent, while the correlation coefficient between "prior examples" and "qualitative analysis" is −0.1 indicating that if students mention prior examples they are less likely to describe a qualitative analysis in their reflections.
The strategy of prior examples was classified as limiting because students who employ such a strategy may struggle on a novel problem for which there is no working model, or for problems that require a combination of approaches.Some illustrative quotes are listed below: "For any given problem I always approach it from a "zero" knowledge point.I do not really have any idea of how to do it yet.In other words there is not much prestudy time for me.I see what the problem entails, and then I search the text book for a given example that resembles that particular question." "I look at examples of similar problems and learn how to work my own from that.I search for example problems in the text.Although the medium may change from paper to websites I think as an engineer I will be constantly adapting an existing solution to solve my problem.I find that I am doing this often with physics problems." The last example indicates a dilemma in labeling this a limiting strategy.The student has realized the benefit of adapting prior work and it is possible that many of his/her future challenges will entail both small and large adaptations.
Diagram.Experts often represent problems in more ways than novices do. 8,77Exploring multiple views of complex problems can lead the problem solver toward building a solution.Some types of rerepresentations are applicable over a wide range of subjects, e.g., drawing a schematic diagram, or noting specific stages and key moments in a problem and using these to anchor a qualitative analysis.Other rerepresentations are more domain specific, e.g., changing the view on mechanics problems in physics from a view where forces are applied, to a view where energy is transformed from kinetic to potential energy.Novices rarely attempt these rerepresentations though explicit instruction and support on qualitative diagrams has been successful. 78isted below are some illustrative quotes from student reflections about drawing diagrams: "Currently, my first response to a physics problem is to draw a diagram.I believe the visual image provides a very good idea of where to go next with the problem.""To start with I figured out that I need to visualize what is happening.This may include drawing multiple diagrams for complex problems ͑like the ones that I get in my thermodynamics class͒ or it could be as simple as a mental image of electrons moving in circles or visualizing the flux changing through a loop.Then based on this I will try to figure out what else will change with respect to the given information." From the reflections it is not always possible to ascertain how students actually use the diagrams, i.e., whether they list this strategy purely because teachers have told them to, or because they use it as an alternative representation to help them apply physics principles.
Concepts first.In this strategy students first think about the ideas and concepts involved in the problem and from there start an analysis of the problem.As problems become more ill structured such an approach is often used by experts, and requires strong organized domain knowledge. 8Listed below are some illustrative quotes from student reflections: "I now think deeply into overall concepts of a problem, the big picture and physics principles if you will, before I dig deeper.Then I use the main ideas to see how the problem will flow, putting in some equations but keeping them in variable form and making sure they represent the overall concepts of the problem.From there I identify missing pieces and from the overall concepts and substitute in the detailed equations of the problem.I then simplify and much as possible, while checking units, and substitute numbers in at the end." "The way I like to approach problems is to be able to have a good understanding of the concepts behind the parameters you are working with.That way you not only can crunch the numbers but you know exactly why and how.I also believe that truly knowing the concepts can help you approach a problem with an open mind so that you know how to solve different problems of the same nature." Qualitative analysis.In this strategy students first identify key moments, constraints or locations in the problem before starting the quantitative work. 79 Subproblems.In this strategy students divide the larger challenge into a series of subproblems that they know how to solve.Such a "divide-and-conquer" strategy requires planning and analysis so that time is not wasted on solving for subgoals that are not needed for the final solution.Some typical quotes from students are listed below: "I like to break physic's questions apart and solve each part and then plug it into another part and I eventually end up with the final answer.Some times I have trouble breaking a problem into smaller parts and I just get stuck on the problem." "I try to break problems down in to smaller components.I used to just try and solve the problem as a whole, but I found that method ineffectual for this class." "I try to analyze every aspect of the problem and break it down so that each step is simple and it also makes it a lot easier to figure it out.With splitting up problems into smaller pieces it makes any project or problem easier to accomplish and figure out the best solution."

B. Frequency distributions of student beliefs
All the student reflections were coded by two reviewers as containing one or more limiting or expansive beliefs about problem solving.The distribution of these ideas both at the start and at the end of the semester can be represented by frequency = ͑No. of responses containing a category͒/͑No. of responses͒.
The average coding of the two raters is used for this normalized frequency.Figure 1 contains the frequency for each category at the start of the semester as a solid histogram ͑pre͒ and at the end of the semester as a dashed histogram ͑post͒.For example, a frequency of near 0.4 for the strategy "diagrams" postcourse indicates that almost 40% of the reflections contained the idea of drawing diagrams to rerepresent the problem.Some observations from this data are: ͑i͒ The four limiting strategies ͑shown on the left͒ dominate at the start of the semester, with over 50% of students mentioning Rolodex equation matching, i.e., they solve problems by searching for equations that have the same variables as the knowns and unknowns.
͑ii͒ The frequency of these limiting strategies does not greatly decrease by the end of the semester.
͑iii͒ There is an increase in students describing more expansive strategies within their reflections.In particular there is a large increase in describing the use of diagrams and thinking about concepts first.
͑iv͒ Working from examples or creating subproblems are not commonly mentioned as strategies, neither is the directionless listing of unknowns.
There are at least two possible hypotheses to explain the resilience of the limiting strategies over the course of the semester: ͑1͒ that these are old habits and hence hard to shake or ͑2͒ that for many exercises the students face, these strategies are still successful.Exploring which explanation is correct is a task for future research, while the phenomenon of students describing both expansive and limiting strategies in their reflections is explored in the next section.

C. Characterizing each student
A complementary view of the same data is to examine individual students, categorizing each student as being either predominantly limited or expansive in their preferred strategies.On a student-by-student basis we calculate the number of codified strategies that they listed in their reflection into two groups: expansive = ͕diagram, concepts, qualitative anal, subproblems͖, limiting = ͕Rolodex equation, list known, list unknown, examples͖.
For example, if a student listed both drawing diagrams and subproblems in their reflection then we would assign expansive= 2 for that student.Likewise a student could mention a certain number of limiting strategies, e.g., if they mentioned the rolodex method and listing knowns then they would be assigned a value of limiting= 2. This allows us to define a scale for a student's problemsolving preference pscale = ͑expansive − limiting͒/͑expansive + limiting͒ which has two limits, pscale= −1 corresponds to a student who solely describes limiting strategies, and pscale= +1 corresponds to a student who solely describes expansive strategies.The average coding between the two raters is used to calculate pscale.
The distribution of pscale for reflections written at the start of the semester is shown in Fig. 2 in the filled histograms.Only the 216 students who have written a reflection at both the beginning and end of the semester are included in this plot and the following averages.At the start of the semester many of students can be classified as using limited strategies ͑pscale= −1͒ and the class average is ͗pscale͘ pre = −0.35Ϯ0.05.
For the reflections that were written at the end of the semester ͑striped histogram in Fig. 2͒, the distribution of pscale changes to an even balance between limiting and expansive strategies.There are three observations comparing postsemester with presemester: there is a nearly a factor of 2 decrease in the number of students who describe a purely limited strategy ͑pscale= −1͒, the number of students who describe a mixed strategy ͑pscaleϳ 0͒ slightly increases, and there is an increase in the number of students who describe a purely expansive strategy ͑pscale= +1͒.The class average is ͗pscale͘ post = 0.00Ϯ 0.05 reflecting the balance of student beliefs reached by the end of the semester.
Another way of quantifying the change in beliefs is the level of improvement from pre-to postsemester for each student.When averaged over the whole class we find: Ͻ⌬pscale Ͼ = ͗pscale post − pscale pre ͘ = 0.35 Ϯ 0.06.
Statistically the increase in pscale is approximately 5 to 6, i.e., the improvement is a statistically significant effect.This is confirmed using a Wilcoxon Signed-Rank Test to estimate that the two distributions of pscale ͑pre and post͒ have significantly different means with a p-valueϽ 0.0001.This nonparametric test was chosen since it makes no assumptions on the shapes of the pscale distributions.
While pscale is a useful integrative measure of each student's problem-solving preferences, it mixes presence of both expansive and limiting strategies.It is therefore also useful to examine the change within each category of strategies, i.e., to define for each student ⌬͑expansive͒ = expansive post − expansive pre , ⌬͑limiting͒ = limiting post − limiting pre .
Averaging over the class reveals a growth in the number of expansive strategies that students describe in their reflections Ͻ⌬͑expansive͒ Ͼ = 0.49 Ϯ 0.06, i.e., a statistically significant average increase of half an expansive strategy per student.The Wilcoxon Signed-Rank Test for the hypothesis that the pre-and postexpansive distributions have the same mean has a p-valueϽ 0.0001.The effect-size for this increase in expansive strategies is 0.6.
The number of limiting strategies decreased over the semester Ͻ⌬͑limiting͒ Ͼ = − 0.27 Ϯ 0.08, i.e., an average decrease of quarter a strategy per student.The Wilcoxon Signed-Rank Test for the hypothesis that the pre and post limiting distributions have the same mean has a p-value= 0.001.The effect size for this decrease in limiting strategies is −0.3.Comparing the two changes in expansive and limiting strategies, there was less a change in the limiting strategies: consistent with the earlier observation that these limiting approaches are resilient.

D. Correlations with student performance
By correlating student reflections with how the students performed during the course we can address whether those students with more expansive problem-solving strategies do well in mastering the physics content.We correlate each student's pscale with both their exam score ͑midterm exams and the final͒ as well as with their final grade ͑sum of scores in weekly problem sets, lab scores, two midterm exams, two group exams, a group design project, and a final exam͒.In Fig. 3 we show the correlation between precourse pscale and exam score for each student.
There is no observed correlation between pscale pre and the exam scores: the correlation coefficient= −0.017 and the nonparametric Spearman-coefficient= −0.002.A similar lack of correlation is found between pscale pre and the final grade: the correlation coefficient= −0.003 and Spearman-coefficient= −0.004.Hence a student's initial preference for expansive or limiting strategies does not predict their exam score or final course grade.
Figure 4 shows the correlation between postcourse pscale and exam score for each student.There is a very weak correlation between pscale post and the exam scores: the correlation coefficient= 0.17 and the nonparametric Spearman-coefficient= 0.14.A similar weak correlation is found between pscale pre and the final grade: the correlation coefficient= 0.14 and Spearman-coefficient= 0.11.
It is important to note that these are correlations and not necessarily causations.From these data alone it is not possible to conclude that if students develop stronger problemsolving approaches there is a slightly increased chance they will do well in the course, or whether those who do better in the exams and course are slightly more likely to develop stronger problem-solving skills.

E. Examples of changes in reflections
Further insight into the change of students' problemsolving strategies can be obtained by looking at reflections from the same student at the beginning and at the end of the semester.Some example reflections that showed an increase in expansive strategies are listed in Table II ͑without grammar editing͒.Table III, on the other hand, contains some example reflections from students who showed resiliency in using a limited strategy at both the beginning and end of the semester.

F. Other messages conveyed by students during their reflections
One of the side-benefits of asking students to reflect on their learning is the insight an instructor can gain into student lives.As an indication of the depth and breadth of perspectives that students raise, here are some further quotes from the problem-solving reflections.
First a quote that typifies the benefit to a student to reflect on their learning: 75 "I have found ͓snip͔ that being the middle of the semester had midterms and other things going on, that I was crunched for time.Instead of learning the material and then doing the assignments, I would just try to search for equations in the text that would solve the problem, and if I couldn't figure it out, just guess.I know that this is almost an immature re- know into words, rather than just an equation, I have been more successful.I think that this method of trying to visualize the problems and think them through before just plugging them into equations will help me out with future problem solving where there may not be a cut and dry equation or answer that I am looking for.I always approached the problems knowing that it was possible to solve them and simply tried to connect the dots.I found that writing out all the information I was given-then used it to find all the new information I could find using equations ͑or if I could see far enough ahead, only what I knew I'd need͒.Now instead of looking directly for equations ͑which I do still do sometimes͒ I'll usually try and identify concepts at different parts and qualitatively analyze it before jumping into the math.Once I've figured it out conceptually ͑usually w/a diagram/picture/etc͒ I'll see which equations might help and then I just do the math.I approach physics problems by trying to dissect them.I try to be very methodical and organized.After reading the problem, I list the given information, what I'm solving for and any known relationships or equations.
I approach physics problems in a very qualitative respect.After reading a problem statement and identify what is to be solved for, I put down on paper the relative concepts for the problem.Usually diagrams with necessary parts labeled.Being able to draw such a "complete" diagram insures I have knowledge of the necessary information to solve the problem.When I am trying to answer a physics question I first like to understand what the question is asking.Once I understand the question I try to find any equations that are helpful in solving the question.
I have begun to at least mentally, and sometimes physically map out the problem.This is a new thing, and I have found that it really helps me to keep terms and ideas straight as I work thought the problem.sponse to being busy.I am concentrating now on learning the material, not just regurgitating it.I know that in a career in the future, that being busy doesn't mean you just slack or get by.I will concentrate on pushing myself toward learning, in all my classes." In contrast, here are examples of students who do not fully understand the wider range of skills they will need in future careers: "I use a very solution oriented method of solving physics problems so far.I tend to use the numbers right away instead of deriving the equation and then plugging in the numbers.Since I use this method I definitely need to watch for mathematical errors in my solutions.I think that this method and the skills I get from it are perfect for my future in the construction engineering industry.Do not spend a lot of time on theory just solve the problem presented." "The way I approach physics problems is by looking for a formula to follow.I will start a problem, look in my notes for a formula, look on the discussion board for a formula, look on the formula sheet for a formula, and lastly, look in the book for a formula.This is probably exactly I will approach some problems in Computer Science.If I need to implement a function, I will need to find the syntax for the function and a little excerpt saying what inputs it has and what outputs it has." And finally, here is an example of a student's candid admission that reminds instructors how hard these introductory university courses can be: "I feel lost in the questions that are asked and most of the time do not know where to start on the subject.I have a tutor but it seems like it is almost to late to relearn everything over the semester.I am not saying I am going to quit trying but I do feel lost."

V. DISCUSSION
Key to students developing their problem-solving skills and strengthening their abilities to tackle ill-structured and open-ended challenges is for students to believe that problem solving is more than applying an ever-expanding list of procedures to tasks.Experts utilize a broad and rich range of approaches to challenges: they rerepresent complex problems to gain insight, use their organized understanding to qualitatively analyze and plan possible solutions, monitor their progress so they stay on track, and finally evaluate and justify their solution when the criteria for judging the solution are not clear cut.As educators, we must provide opportunities for our students to develop these skills in introductory courses.If we wait until only later courses to challenge students with rich, complex tasks, then they will have little time to gain confidence and experience with these approaches.
From research on student epistemology, a prerequisite for students to be motivated to develop these skills is for students to realize that something in their beliefs is impacting their ability to make progress, i.e., there must be some epistemic doubt or acknowledgment that there is a need to change.Then the new strategies or beliefs must make sense to the students, and they must see how to apply these ideas in their situations, and finally the students need to experience that the new beliefs produce some success.In this paper, I have described one method of creating epistemic doubt in an introductory physics course by having students work on context-rich, multifaceted problems.These problems require groups of students to analyze complex problems and build a solution.Plug-and-chug strategies of searching for an existing equation do not work.The students are supported in developing stronger problem-solving skills by explicit classroom discussion of these methods, support from the TAs, practice throughout the semester, and accountability via group exams.
Students were asked to reflect on their problem solving at the beginning and at the end of the semester.These reflections were coded as containing one or more problem-solving ideas.Four of these ideas were "limiting" strategies, approaches that fail as problems become more ill structured, and four were more "expansive" strategies.The four limiting ideas are: Rolodex equation matching, listing known quantities, listing unknowns, and searching for prior examples in text or lecture.The four expansive strategies are: drawing a diagram, examining concepts first, performing a qualitative analysis, and dividing the problem into subproblems.
At the beginning of the semester the four limiting strategies dominate student thinking, with over 50% of students mentioning they use Rolodex equation matching, i.e., they solve problems by searching for equations that have the same variables as the knowns and unknowns.The frequency of these limiting strategies reduces slightly by the end of the semester.The average decrease is a quarter of a limiting strategy per student.There is also an increase in students describing more expansive strategies within their reflections, with an average increase of a half an expansive strategy per student.In particular there is a large increase in describing the use of diagrams, and thinking about concepts first.Our numerical results are statistically significant at a level of several .
From the beginning to the end of the semester the number of students who describe a purely limited strategy decreases by nearly a factor of 2, and there are slight increases in both the number of students who describe a mixed strategy and in the number of students who describe a purely expansive strategy.The data indicate the presence of an intriguing balance: at the conclusion of the course the students are almost equally split among three groups: solely limited, solely expansive, and mixed strategies.A similar interplay between the development of expansive beliefs and the resilience of limiting beliefs has been observed by others.Kuhn 80 describes metacognitive development as "rather than constituting a single transition from one way of being to another, entails a shifting distribution in the frequencies with which more or less adequate strategies are applied, with the inhibition of inferior strategies as important an achievement as the acquisition of superior ones."Kuhn, 80 p. 179.
Overall the correlation between student's reflections and their performance on the course was very weak, indicating that a broad range of students hold similar beliefs about physics problem solving.A student's initial reflection and whether it contained expansive or limiting strategies had no predictive power on how well the student would do in the course.There was however a very slight correlation between student's postcourse reflection and overall course score.
This study did not include a control group, i.e., students taking the same course but who did not work on context-rich problems.Hence it is not possible to say whether these problems contributed to the change, or whether the results are due to the maturation of the students, or the strategies discussed in lectures, the guidance of the TAs, etc.The strongest conclusion that can be currently drawn is that the students' beliefs about problem solving did change by the end of a course that emphasized problem solving by having students work on multifaceted context rich problems.These problems were the vehicle that the lecturer used to discuss problem solving in lecture, and that the TAs used to guide students during recitation.
There is considerable future work to be done.More detail can be obtained from interviews of students who report using mixed strategies, e.g., does their choice of strategy depends on the context of the problem or on the discipline.Also what happens to these students after they leave this course?Do their problem-solving beliefs and skills continue to grow, or are the changes reported here not rooted enough to be stable after the students leave a course that is heavily focused on problem solving.We are also designing a study to ascertain the fidelity between the problem-solving strategies students report and what they actually use in solving complex problems.Finally, it will be interesting to correlate student beliefs about problem solving with other measures of their intellectual development. 23,24,81edagogically it is also important to find ways to make a larger impact on students' beliefs, especially the group of students whose beliefs do not change over the course on a semester.One option may be to code the presemester reflections very quickly and form groups of students with a broad range of problem-solving beliefs.In a heterogeneous group, students who approach problems with expansive strategies may be able to help develop stronger strategies in their fellow group members who may initially only use limiting strategies.
buy enough ice to put in the box at 6 am so that the temperature stays at 0 °C until the picnic starts at 6 pm.You do not want to buy too much ice because that means that you will have less money to spend on food and other picnic items.How much ice will you need?
͑2͒ You are an engineer designing a nuclear power plant.The core of the reactor is designed to operate at a temperature of T H and the cooling water is at a temperature of T C .Your group has found that you may be able to reduce the construction cost of the plant considerably by using two smaller engines to convert the heat from the reactor into work in the form of electrical energy.The design concept you have developed is to use a tank of liquid lithium as a buffer to be held at temperature T between T H and T C and then use one generator to operate between the core and the lithium tank and another to operate between the tank and cooling water.
You are to present your idea to the engineering committee which is to decide whether a full scale engineering study of this design is to be undertaken.You need to develop the case for this design, including what temperature T for the lithium tank produces the greatest efficiency for this two-engine solution and how does this efficiency compare to the standard design.
͑3͒ Your friend, an artist, has been thinking about an interesting way to display a new wind sculpture she has just created.In order to create an aural as well as visual effect, she would like to use the wires to hang the sculpture as sort of a string instrument.Her basic design involves vertically hanging two pieces of wire from two eye hooks on the ceiling that are approximately then hanging the heavy sculpture from a horizontal bar from some point along the bar.The aural effect that she would like to achieve is that when the wind blows across two vertical strings, they play a perfect fifth, i.e., the ratio of the frequencies of the two sounds is 3:2.Your friend tells you that she has been successful in hanging the sculpture but not in choosing the point along the bar to hang the sculpture giving the desired sound.Desperate for success, she knows you are taking physics and asks you for help.
What is your advice concerning the design of the sculpture.What notes will the two strings play with your design?
͑4͒ You have a summer internship at a company that makes medical instruments.During medical surgeries, there is a need to measure the amount of blood flow through arteries that have been exposed by the surgery, but otherwise have not been cut.That is blood is still flowing through these arteries.You know from your studies of biochemistry that blood contains a reasonable amount of both positive and negative ions.If you place a small magnetic field across the artery, then these moving ions would experience a magnetic force.Your company also manufactures a range of devices that can measure the electrostatic potential between two points.Later this morning you are due to meet with your boss.You need to sketch out a device that could provide the blood flow based on the measurement of the electrostatic potential across two points on the artery.Based on the model you develop, what electrostatic potential would you expect to observe?For the device to be practicable it needs to respond relatively quickly, so you should also estimate the order of magnitude of time it takes for the electrostatic potential to develop across two points on the artery.
͑5͒ You are helping out during the summer at a relative's farm.In one corner of the farm are some high-tension power lines.Having aced Phys 222, you know that each power line will be surrounded by a magnetic field that changes with time.You wonder whether you could use this to induce an emf in a coil, and use the induced emf to drive some of the farm equipment.To test this idea you construct multiple loops of wire and connect it to an ac voltmeter.Later this morning you are due to show your relative your loop, your measurements, and an explanation of this works.What induced emf will you measure?Your relative will also want to know whether this is really power for free, or how could the utility company detect the theft of this power.Note neither the Physics Department at ISU, nor your instructor endorses this method of obtaining power.
͑6͒ Your PhD project in Limnology ͑the study of lakes͒ consists of studying the movement patterns of fresh water copopods, small ͓O͑1 ϳ mm͔͒ crustaceans near the base of the food chain.You decide to track your specimens in an aquarium with a computer driven optical device.A chargedcoupled device ͑CCD͒ is mounted in a camera assembly above the aquarium, above the water in the air, which can be moved horizontally by a computer controlled positioning platform.The camera assembly is a vertical tube with the CCD at the top and a lens ͑focal length in air= f͒ at the bottom.The lens is attached to threaded rods, also controlled by the computer, which can raise and lower the lens and thereby project an image from a specific depth onto the CCD.Once the computer finds the image of a copopod, it adjusts the horizontal and vertical position of the device so that a focused image is at the center of the CCD.The computer cycles through these horizontal and vertical adjustments at a rate of ϳ100 Hz and records the position and height of the lens as a function of time.From this data the motion of the copopod can be reconstructed and studied as a function of temperature, pH, food concentration, oxygen levels etc. Develop the expressions which will allow you to determine the position of the copopod from the lens position.
1 Committee on Prospering in the Global Economy of the 21st Century: An Agenda for American Science and Technology, National Academy of Sciences, National Academy of Engineering, Institute of Medicine "Rising Above The Gathering Storm: Energizing and Employing America for a Brighter Economic Future," National Academies Press ͑2007͒.
FIG. 1. Frequency distribution of how often a problem-solving method was mentioned in a student reflection.The reflections written at the start of the course ͑"pre"͒ are shown in solid black and the reflections written at the end of the course ͑"post"͒ are shown as a striped histogram.The "limiting" strategies are shown on the left and the "expansive" strategies are shown on the right.

FIG. 3 .
FIG. 2. Distribution of students who have a given problemsolving preference, pscale, where pscale= ͑expansive − limiting͒ / ͑expansive+ limiting͒.The reflections written at the start of the course ͑"pre"͒ are shown in solid black and the reflections written at the end of the course ͑"post"͒ are shown as a striped histogram.
3. Some illustrative quotes from reflections that were coded as listing knowns are quoted below: "In approaching physics problems the first thing I always do is write down what I know.I find writing this down helps to show what I am working with and what I am looking for.""I write down the known facts and what I need to find.I assign variables to each fact-known or unknown alike.My biggest problem is finding information that is not needed in the problem and, therefore, a waste of time."

TABLE I .
Categories used to code student reflections.values for inter-rater reliability are listed below each category.
Such a strategy helps ensure that the main aspects of the problem are correctly dealt with before working out the final details of the solution.Listed below are some representative student quotes: "The first thing I always try to do is get a mental picture of what is going on.If I can get a physical picture it will make the problem easier.Then I analyze the problem qualitatively and try to figure out what is going to happen.Then I move on and analyze the problem quantitatively and double check with my qualitative analysis to see if they agree.""If the problem is confusing, I go through my head at what would make logical sense in solving the problem.Sometimes for it to make sense, I have to imagine myself in the situation and think about what would occur in this situation."

TABLE II .
Examples of comments from the same student that showed large growth from the beginning to the end of the semester.

TABLE III .
Examples of comments from the same student at the beginning and end of the semester.These comments showed resiliency to maintaining a limiting strategy looking at what variables are given.Then, I figure out what term the problem is asking me to solve for.I usually then look over the equation sheet and see if any of the equations look like they could be of use in the problem.I first look to see what I need to find in the problem.Next, I look at what variables are given in the problem.Then I look at my lecture notes for an equation that could be used.If I cannot find a useful equation, then I search the book for problems or examples.