Conceptual problem solving in high school physics

Jennifer L. Docktor, Natalie E. Strand, José P. Mestre, and Brian H. Ross Department of Physics, University of Wisconsin–La Crosse, La Crosse, Wisconsin 54601, USA Department of Physics, University of Illinois, Urbana, Illinois 61801, USA Beckman Institute for Advanced Science and Technology, University of Illinois, Urbana, Illinois 61801, USA Department of Educational Psychology, University of Illinois, Champaign, Illinois 61820, USA Department of Psychology, University of Illinois, Champaign, Illinois 61820, USA (Received 30 April 2015; published 1 September 2015)


I. INTRODUCTION
Physics teaching in both high school and college places an emphasis on problem solving [1][2][3][4][5][6][7][8], and although students demonstrate reasonable competence in traditional assessments of problem solving skills, there is evidence that understanding of fairly fundamental concepts is weak or lacking following completion of introductory courses [9][10][11][12][13][14]. Students in introductory physics courses solve problems largely using a process termed means-ends analysis, whereby they search for equations containing the quantities in a problem and try to reduce the "distance" between the goal state and their current state in the solution process [5,8,15,16].Students are not taught to solve problems simply by manipulating equations since instructors typically mention the concepts and principles that they are applying, but students rightly perceive the equations as being central to obtaining quantitative answers and tend to ignore conceptual information.This approach can be effective at getting answers, but falls short in understanding the conceptual underpinnings of the solution process.It is, therefore, not surprising that students learn or retain little conceptual knowledge following introductory physics courses.
Although physics instructors at all levels would agree that integrating conceptual knowledge with problem solving is a desirable goal in physics instruction, traditional materials tend to promote, albeit inadvertently, equation manipulation at the expense of conceptual understanding.Standard physics textbooks present equations in terms of general symbols and elaborate upon what those symbols stand for; however, there is little guidance for students regarding when it is useful to apply a particular relation to a problem [15].Understanding the "conditions of applicability" for a principle and the procedures for determining whether the necessary conditions have been met are essential for proficient problem solving, and these conceptual aspects need to be made explicit during instruction [6,14,17].
The equation-centered approach favored by beginners is in contrast to the more strategic approach favored by skilled problem solvers.Skilled problem solvers organize their solution strategies around major principle(s) or concept(s) [2,5,18].For skilled problem solvers, principles or concepts in memory are also bundled with contexts or conditions in which they can be applied and with procedures for applying them [2,15,19].This type of integration of major ideas, contexts, and procedures provides skilled solvers with a hierarchically structured, well-integrated knowledge base that guides their problem solving [17,18,20].Since building such a knowledge base takes considerable time and effort, an important consideration is whether there is benefit in attempting to help students in introductory courses develop the type of knowledge needed for skilled problem solving.Several studies suggest an affirmative answer.
One intervention in introductory mechanics with undergraduates began with a conceptual overview of the material in qualitative form [21]. Students learned to reason about phenomena qualitatively, using various representations (e.g., free-body diagrams and energy bar charts) for the concepts learned.After students learned qualitative ways of describing and reasoning about physical phenomena, they then experienced a second pass at the same content but this time with mathematical descriptions.They also worked on multistep problems that required application of knowledge from different parts of the course, thus integrating knowledge across course components.Additional activities included problem categorization, where students identified the major principle(s) or concept(s) needed to solve problems without actually solving them.Students experiencing this intervention outperformed students in a traditional course in conceptual and problem solving assessments, and were also better able to retain conceptual knowledge long after the course was over.The same results were replicated with at-risk students at a different university [22].
Another attempt to integrate conceptual knowledge within problem solving used a computer-based tool that allowed users to analyze problems in terms of the principles or concepts needed to solve them [23,24].To analyze problems using the tool, students made selections from a series of menus, with each subsequent menu becoming increasingly specific and building on choices made in previous menus.By selecting appropriate principle(s) or concept(s) (e.g., work-energy theorem or conservation of energy), and specifying initial conditions and problem context (e.g., whether or not frictional forces were present, whether a body possessed kinetic and/or potential energies in some initial state), conceptual qualitative analyses of problems could be performed, which in turn resulted in one or more equations that the tool generated to fit the specific analysis performed.Compared to students who experienced a more traditional control treatment, students who used the analysis tool were better able to categorize problems according to the underlying principle [25] and to solve problems.
Yet another study [26] conducted in a large introductory mechanics college course asked that students write conceptual, qualitative "strategies" [descriptions of the principle(s) needed to solve a problem, a justification for why the principle(s) applied to the problem, and a procedure for applying the principle(s)] for problems in homework and exams prior to generating a solution.Course instructors modeled writing quality strategies in lecture and discussion sections, and posted solutions for the weekly homework assignments that displayed both strategies and solutions for all problems.When compared with students undergoing traditional instruction, students who practiced strategy writing displayed benefits in conceptual measures, including better ability to categorize problems according to principles, and better ability to identify the major ideas covered in the course several months later.
At the high school level, studies of the impact of integrating conceptual knowledge in problem solving on students' conceptual understanding are rare, although there is a curriculum [27] that targets the development of physics conceptual knowledge, and a program [28] specifically aimed at helping students overcome misconceptions.However, both are designed in a context outside of problem solving and with no mathematics.One high school study [29] did compare the impact of "explicit problem solving instruction" to traditional problem solving instruction on various conceptual and problem solving measures.Both approaches included five steps: • ) check the answer.The step of "focus the problem" guides students to state a general approach in terms of concepts and principles, however, it does not emphasize the justification for the appropriateness of those principles in the manner that the Conceptual Problem Solving method does [30,31].Although there were some advantages for the explicit problem solving group, the two groups performed equivalently in tasks requiring conceptual knowledge, such as planning solutions and conceptual questions.
The current study investigated a question with dual goals: Can pedagogical approaches that have been shown to be effective at promoting CPS among college students be adapted for use in a HS setting in ways that allow flexible adoption by teachers, and that promote measurable positive outcomes in students?The need for flexibility in adoption among high school teachers is important.The college studies reviewed above that promote conceptual development within problem solving were both developed and implemented by the same faculty member or researcher, thus creating investment or ownership as well as insuring fidelity of implementation.On the other hand, it is unrealistic to have high school instructors adopt wholesale an approach handed to them by university researchers.High school teachers face constraints and challenges in teaching their physics courses that may place limits on the time they devote to implementing the new approach, as well as to the style in which they implement it; for example, the time that can be devoted to implement a new approach as well as to administer assessments to measure outcomes was limited.It was impossible, therefore, to design a classroom-based high school study in which the teachers adopted or delivered the approach in carefully controlled ways that one could achieve in the researchers' laboratory.
The intervention was designed to provide teachers with ways of highlighting the role of conceptual knowledge in problem solving while keeping the implementation of the intervention flexible, thereby allowing teachers ways of promoting conceptual problem solving in ways to fit their curricular demands.Materials were developed to encourage students to perform qualitative analyses of problems (similar to the strategies described in Ref. [26]), and then to use those strategies in formulating quantitative solutions to problems.The intervention attempted to steer students' tendencies away from equation hunting and toward following a general framework for solving problems that began with identifying principles to apply, justifying why they could be applied to the problem, and then generating a solution plan to generate an answer.
The second goal was equally important, namely, measuring whether the approach had an impact on student outcomes.Assessments needed to be developed that attempted to measure conceptual understanding as well as problem solving.The assessments developed took more time to administer than teachers typically had available to devote to noncourse-related assessments.Teachers had the flexibility to decide how much class time they would devote to administering study assessments subject to their curricular constraints, as well as to which particular assessments would be administered.
There were additional challenges facing the study.Whereas almost all the studies cited above evaluating interventions to promote conceptual problem solving were done with college students who had declared an interest in pursuing a STEM major, the high school students in our study had much more diverse interests and mathematical preparations.In addition, unlike the college studies cited earlier where those designing and implementing the intervention were physicists, the high school teachers in this study reflected the background of teachers nationwidesome were teaching out of field while others had degrees in physics.Finally, the number of students participating in each class in our study was small given physics enrollments at the participating high schools and the high schools were different enough that we felt it best to analyze each separately, placing constraints on the interpretation of the findings.Even though this limitation requires a focus on trends across schools, we believe the trends are fairly strong.In summary, the study reported here was challenging, looking for what were likely to be somewhat subtle changes in problem solving and conceptual understanding resulting from an intervention that was implemented in different ways over different durations by teachers with varying degrees of physics competence to students with varying degrees of physics interest and mathematical preparation.
The next section describes the materials, participants, and experiment design in more detail.It is then followed by the study results and a general discussion of the findings.

II. METHOD
A. Materials

Learning materials
Conceptual Problem Solving (CPS) is not a curriculum, but rather is a framework for solving physics problems that can be easily adapted to existing course materials.The features of CPS are adapted from Ref. [26] and modified Problem: At point A on a roller-coaster, a 150 kg car is traveling at 13 m/s and is 3 m above the ground.Calculate the speed of the car at point B, when it is 5 m above the ground.

Principle:
Conservation of energy: the total mechanical energy of an isolated system (sum of kinetic and potential energies) is the same in the initial and final states.

Justification:
There are no non-conservative forces doing work on the roller coaster car as it travels along the track (we neglect air drag and friction).Therefore no energy is gained or lost and the energy of the car at point A is equal to the energy of the car at point B.

Plan:
1. Draw a picture and assign symbols for quantities in the problem.Choose a coordinate system.to be more suitable to a high school student population.The strategies used in Ref. [26] for the study with university STEM majors were intended to be prose descriptions of the principle(s), justification(s), and procedure(s) needed to solve problems, but no specific guidance was provided for how to write strategies other than telling them that a good strategy had to be a logical discussion of how to solve a problem that contained the three pieces above.For the high school student population we broke down this process into more specific stages to provide more guidance to students on what was expected of them.Further, because writing qualitative descriptions for solving problems is a high level, difficult task, we deemed it important to provide as much structure as possible to help students succeed in writing strategies, while at the same time not being overly burdensome in our requirements.The CPS approach that we finally decided upon contained three separate parts: Principle (the principle or concept applicable to the problem), justification (explanation of why the concept or principle is appropriate), and plan (numbered steps that provide the "recipe" for solving the problem and the equations that do with each step in the plan.Students carry out their plan for solving the problem by formatting their solution as two columns: A left column describing the plan step, and a right column consisting of equations or mathematics that go along with the step [32].When the method is first introduced, scaffolding is provided in the form of headings or blank worksheet templates that prompt students to complete each part.A sample strategy is presented in Fig. 1 and its accompanying two-column solution is presented in Fig. 2.
It is important to note that students were not required to write the principle, justification, and plan before carrying out the quantitative solution on the right-hand side of the two-column solution, although most students did, as did instructors when illustrating the approach to students.Here, as in Ref. [26], the aim was to highlight the role of conceptual knowledge in problem solving, and this could be accomplished by writing the principle, justification, and plan either before or after generating a solution with equations.
A sample set of problems, strategies, two-column solutions, and blank worksheet templates were written by the authors prior to classroom implementation of

FIG. 2. Sample two-column solution.
CPS. Typically 10 or more problems were developed for each physics topic, exhibiting a range in problem complexity and difficulty.The topics included motion in one and two dimensions, Newton's laws, momentum (Impulse-Momentum Theorem and Conservation of Momentum), and energy (Work-Energy Theorem and Conservation of Energy).Upon request from participating teachers, a few problems were also written for minor topics including vectors and torque.The files for these problems and solutions were provided to teachers participating in the study, who were given the option to use these problems as written, modify these problems, or to use their own problems but format them to look similar to the samples.

Assessments
Students from the traditionally taught classes and the CPS classes completed a series of written assessments to evaluate their understanding of physics concepts and their problem solving skills (see Supplemental Material [33] for sample items from the assessments).There were five different tests administered to assess different aspects of the students' understanding, which we briefly explain here.Three of the tests (problem solving, conceptual questions, and categorization) examined important aspects of what might be learned from CPS, another test examined whether there was any influence on a different complex skill (finding errors), and the final test examined students' understanding of the variables in the final equation (equation instantiation).Examples of each of the tests are provided in the Supplemental Material [33] and should be examined when reading these explanations.We return to these assessments again in the results section.
Problem solving.-Items in this test are standard free response problems, similar to traditional problem solving measures used in physics courses, that students were asked to solve.Most physics courses use problem solving as their primary assessment, so it was deemed critical to examine problem solving.CPS is meant to help the students to proceed from a problem to a principle, so should improve problem solving.Total time: 15 min for 3 items.
Conceptual questions.-Items in this test prompt students to write free-response explanations for a realistic physical situation (school A) or answer conceptual questions in a multiple choice format (schools B-C).This test evaluated whether students could link physical situations to principles and concepts-a basic goal of CPS.Total time: 15 min for 6 items.
Problem categorization.-Items in this test are similar to the 3-problem categorization tasks used in Ref. [25] but modified to be grade appropriate.Students must select one of two problems that would be solved most like a third "model" problem.Problems are specifically designed to vary their match on superficial features (objects or context) or match on concepts and principles used to solve them.This test evaluated ability to determine if problems were solved with similar principles and whether students were distracted by the presence of similar surface attributes in problems.Total time: 25 min for 9 items.
Finding errors.-Items in this test contain a problem statement and worked-out solution that includes a conceptual physics error.Students are asked to identify and describe the error in writing, and are specifically informed that there are no mathematical errors in the solutions.This test measured if students could detect solutions that were lacking the application of a needed principle or solutions that applied concepts incorrectly.Total time: 15 min for 3 items.
Equation instantiation.-Items in this test include a problem and worked solution in symbolic form.Students are asked to assign or match the appropriate values for each quantity in the final expression.Because the final expression from the principle is given, it does not require any decision about the correct principle or justification for it.Instead, the test requires students to be able to match particular quantities in the problem to variables in the final expression.This is important knowledge for getting the correct answer, but it is not what would be learned by CPS, and we did not expect this would lead to much difference between the groups.It is possible that there will be some small effect because a better understanding of the principles could also lead to better understanding of what the principle variables map onto in a problem.Total time: 15 min for 5 items.

B. Participants
Physics teachers from four high schools agreed to participate in the study, however due to unforeseen circumstances that affected the implementation and assessment procedures, one of the schools was excluded from the data analysis. 1The remaining schools and CPS teachers will be referred to as A, B, and C.These teachers were selected in part because they were teaching nonadvanced placement classes at their school and had some flexibility in designing classroom activities.
The particular experiment design at each school was determined by situational constraints, such as the number of physics teachers and number of classes offered there.At every school there was at least one CPS class and one class taught using traditional, equation-focused problem solving methods.At school A there was one teacher who taught the same course over multiple years, at school B there was a single teacher teaching two sections of the same class using different methods, and at school C there were two different teachers.Because of such vast differences in the design, the student populations at each school, and the teachers' knowledge and experience teaching physics, each school will be treated separately in the data analysis.
A brief profile for each of the four schools is provided in the Table I  School A is a small suburban school with an experienced physics teacher and an affluent, high-achieving student population.This was the teacher's second year participating in the project, with the first year used as a control class and taught without CPS.All students were in their senior year.School B is a rural high school with a student population that is primarily low income.The teacher taught two sections of the same course, one using the CPS approach and one using his usual way of teaching problem solving that emphasizes equations and mathematical procedures.The teacher's area of expertise is chemistry, and this was only his second time teaching physics.Students enrolled in this physics course at school B are in their junior year.School C is a high school in a small city with a diverse student population.There were two teachers both teaching different sections of the same course (one used CPS and one taught traditionally).Although both teachers have a strong background in physics the CPS section teacher typically teaches math courses and the control teacher has far more experience in terms of years teaching physics.
All participants whose data are reported signed consent forms allowing their results to be used (students under 18 years of age signed assent forms and their parents signed consent forms).
C. Procedure

Training procedure
Teachers met for a half day with researchers during the summer prior to using the CPS approach to receive minimal training and guidelines for implementing Conceptual Problem Solving.The teachers at each of the three schools were given guidance on key elements of the CPS approach, such as the goal being to focus on the role of concepts in solving problems.They were shown specific examples of several problems solved using the CPS approach (see Figs. 1 and 2).Teachers were also told that they would be given detailed CPS solutions to problems (as in Figs. 1 and 2) during the semester as they covered new material to use in their teaching.We pointed out to them that they had substantial freedom to adapt the CPS approach to their particular class and student population.This was intentional since no curricular intervention will be implemented unless teachers have the flexibility to use it in ways that fill their instructional needs.Prior to this short training session, the teachers were not familiar with the approach, other than previously agreeing to participate in the study prior to the training.Our observations suggested that the teachers bought into the CPS approach to the extent that they were presented with previous research evidence favoring conceptual approaches to problem solving (as reviewed in the introduction), and to the extent that they found the approach reasonable and useful for classroom implementation.

Observations and debriefing meetings
Each CPS teacher's classroom was observed at least once for each major physics topic (four times) during the implementation.Each teacher met with the researchers for a debriefing session at the conclusion of the study; these semistructured interviews were audio recorded, which provided self-reported descriptions of how they implemented CPS, what they thought about teaching this way, how frequently they used materials, and teachers' perceptions of students' reactions to the approach.

Implementation of the intervention
The conceptual problem solving approach was designed to be implemented during classroom activities that pertained to problem solving instruction.Since the emphasis placed on problem solving varied for each course and student population, the time spent on CPS also varied.Other course activities such as lecturing, demonstrations, and laboratories remained unchanged.
There was substantial flexibility in how teachers could implement the approach, and as a result the fidelity of implementation varied across teachers.There were variations across teachers in how much time they spent on the approach during class, how classroom activities were structured, and how they assessed student understanding of the approach.Despite these differences, there were some common elements across all of the schools.Each teacher devoted at least two class periods to the approach for each 2-3 week long topic unit studied.They were all accustomed to having students work together in cooperative groups of between two and four students during class, and CPS was no exception to this procedure.All of the teachers chose to grade students on their use of the approach in some fashion, and typically this occurred on an exam.How teachers implemented the approach was in fact one of the factors of interest and so we report below on our observations regarding implementation as well as teachers' descriptions of their own implementation of the CPS approach.
Implementation at school A. Teacher A was an experienced physics teacher who preferred to adapt the approach to his own established set of homework problems.He reported implementing the conceptual framework primarily for more advanced problems, which represented between one-third and one-fourth of students' homework (problems with multiple steps, multiple principles, or more advanced mathematical procedures).Students were not graded on homework, however, at least one problem on each exam required students to use the approach.He was observed to model the strategy-writing aspects for students when doing example problems on the board during class (principlejustification-plan), but only used the two-column solution in an informal way.
For example, late in the semester teacher A was observed to use the approach for a complex problem in which a block slides down a frictionless curved ramp, collides with a second block, and the blocks fall off of a table onto the floor below.He prompted students to decompose the problem into three segments and identify a relevant principle for each part (conservation of energy, conservation of momentum, and motion with constant acceleration), which he wrote on the board.The Justifications for these principles were brief statements: "frictionless ramp," "elastic collision," and "constant acceleration."He then guided students to construct a plan of 3 steps and wrote them out in words on the board.Then he took each step individually and (with verbal prompts from the students) completed the mathematical solution for each step sequentially.Although the solution was not presented side-by-side in the two-column format, all of the components were present.
Teacher A also devoted some class time to "synthesize" the big ideas that had been covered during the fall semester.He did this by showing a diagram that listed the major theorems and conservation laws in boxes, and arrows that connected them to corresponding equations.He emphasized to students that if they are uncertain how to begin solving a problem, they should refer back to the "big ideas" from the class and select one of these concept(s) or principle(s).
Implementation at school B. In contrast, teacher B primarily teaches chemistry and physical science courses, and was teaching this physics course out of field for only the second time.As a result, he described his instructional plans as somewhat fluid and subject to change on a weekly basis.Teacher B taught one section of the course using the conceptual problem solving approach, and taught a second (control) section using a more equation-centered approach (a framework called GUESS-write what you are Given, the Unknown quantity, an Equation, Substitute values, and Solve.)He almost exclusively used sample problems provided by the researchers for both of his classes, but formatted them in a different way for the control class.When introducing the conceptual approach for a new topic, he was observed to write the strategy on the board word for word from the researchers' examples and refer to the example on paper as he did so.Students in the CPS section were given blank worksheets that prompted them for each step in the strategy and to format their solutions as two columns, and they often worked in cooperative groups.Students' homework was graded and at least one problem on each exam (for the CPS class) required them to complete a strategy and two-column solution.Teacher B described his classroom implementation of CPS in the following way: "Initially it was more my guidance, and then while they were still learning it, then they were in pairs or maybe even three to help each other out getting through it while I'd be going around.And then toward the end it was more individual.It'd be, 'Ok, start on it individually, and then now let's compare and see where we are.'And then go big group and I'd have somebody talk about what they thought" (teacher B).
In addition, teacher B later adjusted the way worksheet templates were used in response to student feedback.When some students complained about writing their plan steps and then recopying them into the left column of the solution template, he modified the approach to have students write their plan steps directly into the left column of the twocolumn solution template rather than writing it as part of the strategy.
Implementation at school C. Teacher C has a strong background in physics, but was hired primarily to teach math classes at his school and only taught physics classes on an "as needed" basis.He used many of the problems provided by researchers but also incorporated his own set of problems.He admitted that he did not use the approach very much for beginning topics like one-dimensional motion, but increased his use of strategy writing substantially for later topics like energy and momentum (approximately six times in a 3-4 week period).Teacher C felt that the approach was more relevant later in the semester: "I didn't focus a lot on it during the first half of the semester because the entire first half is constant acceleration…I was kind of holding back on it until we had a few more things in our toolbox before we could start to use it to differentiate between the different concepts.So as we started to get into momentum and energy and torque and things like that I started to use it more frequently" (teacher C).
At the introduction of a new topic, he would write a sample strategy and the initial plan steps on the board with verbal guidance from students.The plan step usually involved a general framework of drawing a picture, listing what you know, writing equations (separating the x and y directions when appropriate), and solving for the unknown quantity.The solve step was not very specific; he told students that as they got started the details would become more obvious.Students followed the two-column solution format on homework, but were not required to do so on exams.
Teacher C made extensive use of cooperative group problem solving and categorization with the conceptual approach.During a group problem solving session, he handed out worksheet templates for four to six problems that prompted students to complete only the first steps of principle and justification.Then, each student group was required to write out a plan and solution to just one or two of those problems, which they presented to the class.

Assessment procedures
As described previously, there were five different written assessments (see sample questions provided in the Appendix).However, due to time constraints not every school gave every test.The researchers allocated time per test and all the tests were gauged to take 85 min (25 min for the categorization test and 15 min for each of the four remaining tests).However, teachers had different limitations on how much class time they could give.The total time allotted for the assessments ranged from 45 to 85 min (administered in either one session or split over two class periods).Teachers did not have access to the assessments ahead of time.The assessments completed by participating students were scored by researchers and did not contribute to students' grades.
Questions that required written explanations or problem solutions from students were scored independently by two researchers using agreed-upon rubrics (this pertained to the free-response conceptual questions used as school A, and problem solving questions used at all schools).These scores were further discussed by the researchers to reach a single consensus score.The problem solving questions were scored according to a rubric modeled after Ref. [35], which scores solutions on the categories of useful description, selecting a relevant physics concept or principle, applying the concepts to the specific conditions in the problem, executing mathematical procedures, and the overall communication of a logical reasoning pattern.The finding errors test was scored for both identifying a mistake in the sample solution and giving an appropriate explanation for the mistake.

III. RESULTS AND DISCUSSION
A. Assessment scores

Description of assessment scores
The average scores on each assessment are provided in Table II.Schools that did not administer a particular test are designated with "not applicable" (NA).In general, the problem solving and conceptual measures administered to students showed a consistent advantage to the Conceptual Problem Solving sections over the classes taught more traditionally.
The problem solving test scores were significantly different at school C, differing by 16% [tð28Þ ¼ 3.17, p < 0.005], with the CPS class having stronger skills at both selecting and applying an appropriate principle.The differences in problem solving scores were large but did not reach significance at the other two schools; school A (10%) and school B (15%).Each problem on the test was scored according to 6-7 subparts as described in the assessment procedures section (useful description or diagram, physics principle, specific application of physics, mathematical procedures, final units, and communicating reasoning.)Each part received a score of 0, 0.25, 0.5, 0.75, or 1.The interrater reliability for the two scorers was a Cohen's weighted kappa (quadratic squares) value of 0.72 AE 0.02 indicating substantial agreement [36].Each subpart was weighted to obtain a total score out of five points (with more weight placed on the physics principle and application).Scores reported in Table II represent consensus scores reached by the two scorers.The conceptual questions test showed a significant difference at two of the schools, a 10% difference at school A [tð35Þ ¼ 2.54, p < 0.05] and a substantial 20% difference at school B [tð15Þ ¼ 3.53, p < 0.005].Although not significant, school C also showed a difference of 11% favoring the CPS class.For the free-response version of the conceptual questions test administered at school A, each of the six questions was scored according to 2 or 3 subparts as 0, 0.5, or 1 point.The interrater reliability for two scorers was a Cohen's weighted kappa (quadratic squares) value of 0.48 AE 0.06 indicating moderate agreement.
The categorization test, equation instantiation test, and finding errors tests did not show any significant differences between groups.However, there was an 11% difference in categorization at one of the two schools that administered the test (2% at the other school), and a 12% difference in equation instantiation at one school (with 1% and 5% at the other schools), all differences favoring CPS.The finding errors test was extremely difficult for all sections at both schools that administered it, with almost all students receiving no credit for their answers.

Interpretation of assessment scores
Overall, the CPS classes performed better, but the differences for tests and schools varied.The assessment scores require a bit of interpretation, but we stress two points in evaluating the results.First, there were not large numbers of students in these classes, so the standard errors are sometimes large.Given the large differences in schools and implementation, we decided that the individual school data are more informative than some way of combining it across the schools.Second, there was great variation in the fidelity of the implementation.Finally, the CPS part of the course was quite short (less than 7 h during a semester).We begin by addressing each test.
The traditional measure in physics classes, problem solving, showed a clear benefit for CPS across the schools but with much variation between students as well.The mean differences were 10%, 15%, and 16%, though only the last reach conventional levels of significance.Given the short amount of time used for CPS, these are consistent and important increases.
The conceptual questions also showed consistent and important increases, with CPS advantages of 10%, 20%, and 11%, with the first two having a statistically significant difference at the 0.05 level.Note that between these two tests, all schools showed one statistically significant difference and one sizable, if not quite significant, difference.The results of the problem solving and conceptual questions provide strong evidence for the effectiveness of the CPS intervention.
The categorization test was administered at two schools and did not show consistent or significant differences (2% and 11%).We expected that this test would be sensitive to the CPS intervention and were surprised the overall effects were not larger.We mention two issues that may have contributed to smaller than expected effects.First, these are difficult tests.Remember, the student was given a choice of two problems to match to the model, one of which was very similar superficially but was solved by a different principle and one of which was very dissimilar superficially but was solved by the same principle (see Appendix).As Table II shows, all groups chose the superficially similar one slightly more often than the similar principle one (all average scores are below 50%).Second, after we conducted this study, we have done additional work on problem categorization [37].That study showed that even once people appreciated that principles should be used to categorize, they found it very difficult to determine the underlying principles.This test required students to determine at least two underlying principles (the model and one other) and to overcome a strong tendency for novices to choose superficially similar problems.
The finding errors test was simply too difficult to find differences-almost all students received no credit for their answers.We knew that such a test requires rather different skills than the CPS focus, but thought it would be interesting to examine if there was transfer.However, given the low performance, it is difficult to know whether the problem is the low transfer or difficulty of the particular problems.
The equation instantiation test showed small nonsignificant advantages for the CPS intervention (12%, 1%, and 5%).We expected any effect of CPS here would be small, since it tests a more specific understanding (objects matched to variables) than the CPS focus, although the trend favors the CPS classes.It is possible that a better understanding of why a concept is applicable could also help to understand how to instantiate the particular variables in the equation of that principle.
Overall, there was a clear advantage for CPS, particularly, for the problem solving and conceptual questions, despite the little time used for the intervention and the small number of students in each comparison.We do recognize that there were unexpected difficulties with the categorization and finding error tests, but we believe there is a clear pattern of performance that favors CPS classes over traditional classes.

Additional assessment issues
We also wish to address two issues brought up in discussions and reviews.First, although this covered a full semester of a course, it would be helpful to know if CPS understanding is not just specific to the assessments used in the study.One school (school C) gives all students districtwide tests in various subjects and the teacher was able to give us the summaries from this CPS year and the previous year for his class for each problem.For the 12 conceptual questions asked in the test, his class for the year using the CPS performed 9% better across the problems [marginally significant, tð11Þ ¼ 1.84, p < 0.10; the CPS class also did better on 9 of the 12 problems, also p < 0.10 by a sign test].Thus, even after the course was over, the CPS intervention led to some gain in district-wide tests developed independently of this study.
Second, is there any evidence that the CPS and traditional classes were of equal ability at the beginning of the study?This was the students' first physics course so we have no way to examine prior physics ability.We chose not to include access to various test scores as part of the assent or consent forms to insure greater participation, so we cannot look at those (and are not sure of the correlation with physics performance).We think ability differences are very unlikely to explain the CPS advantage.Recall that each school had different arrangements for choosing the class that was CPS and the one that was traditional.In school A, with only one physics class per year, one year was traditional and the following year was CPS (this was with a very experienced teacher so the extra year was not likely to improve teaching performance).In school B, the teacher had two classes and randomly chose which would be CPS and which traditional.In school C, the teacher had one class and used the class of another teacher (with far more experience) as the traditional class.Thus, it seems very unlikely that the CPS advantages were due to differences in prior ability.

B. Debriefing meetings
Each teacher met with researchers at the conclusion of the study for a semistructured interview referred to as a debriefing meeting.They were asked to describe how they implemented conceptual problem solving in their class and reflect on what they thought about teaching in this way.

Comments from teacher A
Teacher A said the approach is good in that it forces students to think about what they are doing and why, but said that students at this level struggle with the math and it is difficult to introduce high-level concepts right from the start.He also felt that students had trouble developing a plan before actually trying to solve the problem because they might need to do a little trial and error before knowing what principle to use.His students expressed some resistance to the additional writing required for a strategy, but appreciated that it helped them make sense of the equations.

Comments from teacher B
Teacher B observed that students had the most difficulty with the justification component of strategy writing.
Identifying a relevant principle was not as difficult for these students because they recognized what chapter or unit was currently being covered, and they were never tested on multiple concepts at the same time.For more complicated problems that included multiple steps, some students also struggled with planning the solution.
Teacher B stated that early in the semester he found that it took longer to go through problems with strategy writing and he had some concerns that this might hinder students in the treatment section if they didn't see as many problems during class.Despite his initial difficulties to manage classroom time, he later came to the decision that the additional depth of coverage for the CPS section was merited: "I think a lot of the style I use is more equation based.So I think to me it probably is a pretty good thing to slow down a little bit and concentrate more on the concepts.Because it seems like it merits that, that they would get more out of that and maybe even doing less of the problems and focus a little bit more on the conceptual.I think also getting them to produce more of the conceptual too rather than me talking to them about it, that they're going through and producing it on their own" (teacher B).

Comments from teacher C
Teacher C indicated that prior to using this conceptual approach some students would just solve problems by writing down some "random stuff," because they weren't sure how to start, and with this framework he did not see this happening.He elaborated on this when describing how students were assessed on their use of the approach for several problems on a unit exam: "They had to write out the principle and justify their answer as part A, and then part B was solve it.So it was sort of adapting the method.Because we had done enough practice on it they almost all got that.And once they knew the principle, solving it was easy.And that was one of the themes that, at least I took away from it, was that the actual algebra was really easy, it's just kind of knowing, looking at this problem, how do we even start it.Having them focus on the principle first gave them the confidence to know how to solve it" (teacher C).
He observed that students were producing a higher quality of work and engaging in richer discussions than in the past, and they performed better on a district-wide physics assessment than in past years.Teacher C felt encouraged by improved student performance in the course, especially on a unit test near the end of the semester: "So I thought, well, best way to try this out is to give them momentum, energy, and torque all in one test and I was really really happy.I think it was one of the best results I've ever seen with those concepts all mixed together, all jumbled up…That's something that I've observed in virtually every math class, is that the students can always do any individual skill, but when they see all the skills all at once on like the final, or on an AP test, they just freeze." Teacher C stated that he agrees with the philosophy of the approach, because concepts and principles are essential for learning math and science.He was observed to make this goal explicit to his class through statements such as, "I want to change your thinking from 'Which equation should I use?' to 'What is the concept?'"In the near future he hopes to apply the conceptual approach to classes he teaches in mathematics.

C. Summary of debriefings
Each teacher had a slightly different interpretation of the best use for CPS in their classes.Teacher A found it helpful for students to use this framework to organize their thinking on challenging, multistep problems after they had some practice with simpler (traditional) problems.Despite this partial implementation his students saw modest gains on the assessments, particularly, the free-response conceptual questions.Teacher B used a full-fledged implementation of CPS during all problem solving activities in his class.He exclusively utilized the sample materials designed by the researchers, including the worksheet templates for students.This resulted in widespread improvement over traditional methods; his students showed differences of 10% or higher on three of the five assessments: categorization, conceptual questions, and problem solving.Last, teacher C viewed CPS as a tool to help students distinguish among multiple principles that could be applied to solve problems (e.g., energy and momentum), and as a "starting point" for problem solving.Once students could identify the principles needed to solve problems teacher C felt that problem solving became more straightforward for students.His students made significant gains in problem solving, and performed higher on a district-wide assessment.
All teachers acknowledged that it takes more time to go through a conceptual problem in depth, and, consequently, implemented CPS in ways to suit their own instructional needs as well as perceived pressures (e.g., to cover more content).Teacher A only used the full approach for one fourth to one third of all problems, teacher B ended up assigning more problems as homework for the CPS class or going through fewer problems, and teacher C had students just practice the first two steps (principle and justification) for several problems and only solve one or two of them completely.The teachers also indicated student resistance to having to write more, which prompted them to grade their use of the approach on homework or exams.Teacher B also modified the two-column solutions to reduce the amount of writing.

IV. GENERAL DISCUSSION
The motivation behind the Conceptual Problem Solving approach is to shift students away from their focus on equations and mathematical procedures and toward problem solving approaches based on physics concepts and principles.We found positive effects of CPS even after relatively short exposures to the approach (on average, less than 7 h of CPS classroom instruction over a 4-month period).Except for one of our measures (finding errors) where students from all schools performed near floor, gains favoring the CPS-taught classes in a battery of tests that included both conceptual and problem solving measures consistently emerged.Gains varied across schools and measures, likely due to differing student populations and different implementation of the approach by the three teachers.
CPS is an approach that is flexible, because it does not require major restructuring of the way instruction is carried out in high schools and because it allows teachers to implement it in different ways.This was confirmed by observing the ways in which each teacher adapted CPS to align with their personal teaching style and the perceived needs of their students.In general, the teachers who were more experienced (teachers A and C) used the approach for a subset of problems and placed greater emphasis on choosing among multiple principles or combining principles for problems, whereas inexperienced teacher B relied upon researchers' materials for all problem solving activities and as a result implemented the approach with a high level of fidelity.More rigorous training beforehand and/or during the semester may have led to a closer alignment with researchers' intentions for the materials.
The realities and constraints of working in real classrooms with real teachers created implementation challenges.For example, the participating teachers' physics expertise varied, from two teachers who had physics undergraduate degrees to one teacher trained in chemistry who was teaching physics out of field for only his second time.Pressure to cover content and time constraints also limited the amount of time spent on CPS and on assessing its impact.Finally, students tended to resist the approach since it required them to do significantly more prose writing about physics and to carry out more conceptual discussions, which were both difficult for students to do and time consuming.
In light of these difficulties, our experiences suggest possible improvements to the approach.Implementing CPS using technology (e.g., via a web-based program that presents problems and students perform analyses by answering questions posed by the program and receiving conceptual feedback) would create several advantages.For example, students could use CPS both for in-class problem solving and for homework, thereby spending more time thinking about the application of concepts in problem solving.Designing the technology-based program in ways where students are queried about what concepts they could apply and allowing them to make selections from choices provided would streamline the analysis and reduce student frustration during times where they were stuck with the paper-and-pencil version.This would also add flexibility to teachers' implementation of the approach.Additional improvements to the approach include greater emphasis on categorization practice, as was done at school C (identifying appropriate concepts and principles for problems without solving them) and prompting students to synthesize across all topics, as was done at school A.
Looking toward the future, there is a need for curricular materials that help students achieve better conceptual understanding of physics in high school, and indications are that CPS is one possible way of achieving this goal.
Further study of Conceptual Problem Solving is warranted, in order to determine which specific elements of a teacher's implementation are linked to student performance.The findings from this study support the goal of emphasizing concepts used to solve problems, so that students do not treat physics problem solving as a hunt for equations that are forgotten soon after a course is over.
Explicit problem solving: (a) focus the problem; (b) describe the physics; (c) plan the solution; (d) execute the plan; and (e) evaluate the solution.• Traditional problem solving: (a) Draw a sketch; (b) define known and unknown quantities; (c) select equations; (d) solve equations; and (e

1 .
Draw a picture and assign symbols for quantities in the problem.Choose a coordinate system.m = 150 kg mass of car v i = 13 m/s initial speed of car (at A) h i = 3 m initial height of car h f = 5 m final height of car (at B) v f = ?final speed of car (at B) 2. Write an equation for conservation of mechanical energy.Expand this equation to include the initial and final kinetic and potential energy terms.
for the final speed of the roller coaster car.Substitute values and calculate a numerical answer.
below.Information regarding the proportion of students who are classified as low income, and the proportion of students who meet federally mandated standards for AYP (adequate yearly progress) in reading and mathematics are taken from public reports by the Illinois State Board of Education [34].

TABLE I .
Profile for each high school participating in the study.Proportion of students at the school classified as low income.b Proportion of students who meet or exceed federally mandated standards for adequate yearly progress (AYP) in reading and mathematics. a

TABLE II .
Performance on each assessment by condition and school.The Conceptual Problem Solving treatment group typically scored higher than the Equation Problem Solving control group.Values represent a percent score AE the standard error of the mean and stars indicate significant differences p < 0.05.
a At school A the conceptual questions were free response explanations and at schools B and C they were multiple choice questions.