Developing the Learning Physical Science Curriculum: Adapting a Small Enrollment, Laboratory and Discussion Based Physical Science Course for Large Enrollments

We report on the adaptation of the small enrollment, lab and discussion based physical science course, Physical Science and Everyday Thinking (PSET), for a large-enrollment, lecture-style setting. Like PSET, the new Learning Physical Science (LEPS) curriculum was designed around specific principles based on research on learning to meet the needs of nonscience students, especially prospective and practicing elementary and middle school teachers. We describe the structure of the two curricula and the adaptation process, including a detailed comparison of similar activities from the two curricula and a case study of a LEPS classroom implementation. In LEPS, short instructor-guided lessons replace lengthier small group activities, and movies, rather than hands-on investigations, provide the evidence used to support and test ideas. LEPS promotes student peer interaction as an important part of sense making via ''clicker'' questions, rather than small group and whole class discussions typical of PSET. Examples of student dialog indicate that this format is capable of generating substantive student discussion and successfully enacting the design principles. Field-test data show similar student content learning gains with the two curricula. Nevertheless, because of classroom constraints, some important practices of science that were an integral part of PSET were not included in LEPS.


I. INTRODUCTION
During the past 25 years a number of inquiry-based physics and physical science curricula have been developed for nonscience majors, especially prospective and practicing elementary and middle school teachers.Such curricula include Physics by Inquiry [1], Powerful Ideas in Physical Science [2], Activity-based Physical Science [3], Operation Primary Physical Science [4], Physics and Everyday Thinking (PET) [5], Physical Science and Everyday Thinking (PSET) [6], and Physical Science for Future Elementary Teachers [7].In all these curricula, students work in small groups and are actively engaged in many of the practices of science: for example, posing questions, developing and using models, carrying out investigations, analyzing and interpreting data, constructing explanations and engaging in arguments from evidence [8].In addition to promoting teacher learning of science through inquiry, the National Science Education Standards [9] also calls for courses that provide opportunities for teachers to reflect both on the nature of science and the nature of learning.Of the above named courses, only Physics and Everyday Thinking and Physical Science and Everyday Thinking provide these explicit opportunities [10].
All of the courses mentioned above were explicitly designed for small enrollments, typically 30 students or fewer, where laboratory work is integrated with small group and whole class discussion.Such courses tend to be taught in classes of 2-3 hours in duration, meeting 2 or 3 times a week.They were not designed for large, lecturehall environments, where classes tend to meet 2 or 3 times a week for 50-75 minutes, and where the use of hands-on materials and small group pacing (as opposed to instructor pacing) is not feasible.Nevertheless, some curricula intended for smaller enrollments have been adapted for larger enrollments [11].Because of logistic and budget constraints, many universities cannot offer small enrollment courses for nonscience majors, but instead attempt to address the needs of this audience through larger, lecturestyle courses.While excellent physics curricula exist that support inquiry-based practices in large-enrollment courses [12] or complement lecture-based instruction with pedagogical strategies that promote active learning [13][14][15], none of these address the range of content traditionally associated with a course in physical science, nor do they include an explicit focus on both the nature of science and nature of learning.
In this paper we describe how we adapted Physical Science and Everyday Thinking to address the need for an inquiry-based curriculum that focuses on the fundamental concepts of physical science, includes an explicit focus on the nature of science and the nature of learning, and is designed to be taught in a typical large-enrollment lecturestyle setting (with fixed seats).We call the new course Learning Physical Science (LEPS) [16].Like PSET, LEPS is a one-semester guided-inquiry course designed to enable students to develop a deep understanding of the conceptual themes of energy, forces, and the atomic-molecular theory of matter.LEPS (again, like PSET) is also intended to enable students to develop an understanding of important aspects of scientific thinking and the nature of science, and to enhance their ability to monitor and reflect on their own learning.The initial version of LEPS was piloted at two institutions by developers and revised based on these experiences.Later drafts were field tested by 8 instructors who attended a 2-day orientation workshop.LEPS was again revised according to field tester feedback.
Section II describes the development of LEPS and its adaptation from PSET, including a set of guiding design principles.Section III provides a brief description of the structure of the LEPS curriculum.To better illustrate how LEPS was adapted from PSET, Secs.IV and V provide a contrastive case study comparison of a PSET activity and homework and the corresponding LEPS set of lessons and homework, including samples of student discourse as they work through the LEPS lessons.Section VI discusses aspects of the course evaluation related to students' content learning, and Sec.VII compares LEPS to PSET.Finally, Sec.VIII provides a conclusion.

II. ADAPTATION: FROM PSET TO LEPS
A. Design objectives and assumptions LEPS and PSET share nearly identical curricular goals, and their developments were guided by a common set of design principles.Section II B describes these design principles, which are based on research on learning and include ideas such as ''learning builds on prior knowledge,'' ''learning is a complex process,'' etc.Despite these commonalities, the differences in the intended learning environments require major differences in the curricula.This section describes the constraints, assumptions, and objectives that guided the adaptation of LEPS from PSET, and the approaches taken to meet those objectives.Subsequent sections describe LEPS in more detail and present an example classroom implementation.
The PSET curriculum design assumes small classes in which students can readily engage in hands-on experimentation, small group work and discussions, and whole class discussion.PSET also assumes $75 hours of class meeting time.Thus, PSET is not suitable for courses with large enrollments, courses taught in typical lecture-style classrooms [17], or courses with only $45 hours of class time (typical of courses without a lab component).By contrast, LEPS is intended to be compatible with such settings, using technology to assist collaboration and interactive engagement both inside and outside the classroom.Specifically, LEPS can be used in standard lecture halls without a reconfiguration of the classroom space.
When developing LEPS, we assumed several constraints associated with large-enrollment courses and lecture room settings: students would be unable to engage substantially in the practices of science (e.g., developing, testing and revising models, designing and carrying out experiments, argumentation) [18]; participation in whole class discussions would be limited; uniform pacing would be required (students cannot work at their own speed); and total class time available for instruction would be about 37 hours.We assumed instructors would have access to an in-class computer, projector, and an electronic polling system (''clickers'').Students were assumed to have access to computers with Internet access outside of class.
Considering these assumptions and constraints, we developed the following objectives to guide the development of the LEPS pedagogical structure: Provide opportunities for students to learn content, the nature of science, and to reflect on their own learning.Follow the five design principles of PSET (described in Sec.II B).Use existing, proven instructional techniques for large-enrollment classes (such as peer instruction) when appropriate.Develop a standard structure for class activities and homework.
Provide sufficient flexibility for use in different institutional contexts.Provide instructors with tools to guide their classroom implementation.The resulting curricular structure represents one approach to meeting these objectives.LEPS employs alternatives to those PSET features that rely on a small, discussion or laboratory format: instructor-led classes with interactive elements substitute for small group guided-inquiry activities; peer discussions and electronic class polling substitute for small group and whole class discussions; movies of experiments take the place of hands-on experimentation; online homework tutorials compensate for reduced class time; and units consist of a greater number of shorter, more focused lessons.Table I presents a comparison of the course features in the LEPS and PSET curricula.

B. Design principles
As mentioned above, the original development of PSET and the subsequent adaptation of LEPS from PSET were guided by five design principles based on research on learning.Below we provide a brief description of these principles (see Table II).A more detailed description of the design principles and references to supporting research are provided elsewhere [19].Sections V B, V C, and V D illustrate how these principles influenced the LEPS curriculum design and its enactment in a LEPS classroom.
Design principle no.1: Learning builds on prior knowledge.-Priorknowledge includes students' everyday experiences and intuitions as well as ideas that they learned in the current or previous courses.This knowledge strongly influences how students interpret situations and guides their predictions about what will happen [20,21].Their prior knowledge also becomes a resource on which they can draw when developing new ideas and understandings [22].In both PSET and LEPS, specific questions are designed to elicit this prior knowledge and to help students build on that knowledge.
Design principle no.2: Learning is a complex process.-Ifwe believe that learning builds on prior knowledge (design principle no.1), we must also assume that students change their ideas as they gather new information, and that this process is complex and takes time.Many ideas in physics are difficult to learn (e.g., Newton's second law), and in the PSET and LEPS curricula such ideas are usually broken down into a developmentally appropriate sequence of smaller grain-sized ideas that become the focus of individual activities or lessons, with later ideas building on those learned earlier.Major curriculum-wide ideas, such as describing interactions in terms of energy, are introduced with simple situations (e.g., colliding objects) early in the curriculum, while more complicated situations (e.g., interactions between magnets) are examined later.
Design principle no.3: Learning is facilitated through interaction with tools.-Toolsfacilitate interactions with others and the environment, and hence learning [23,24].Major pedagogical tools within the PSET and LEPS curricula include laboratory experiments and computer simulations, or movies of them, and various types of representations.Students use data from laboratory experiments to provide evidence in support (or refutation) of the ideas they develop, and computer simulations provide  opportunities for students to test their ideas against scientists' models.The simulations include representational tools such as graphs, speed and force arrows, and energy bar charts, requiring students to make sense of these representations and make connections between them and the simulated (as well as the observed) phenomena.Finally, the student workbook, clickers, and instructor presentation slides are tools that structure the students' experience of the curricula and their interactions with each other and the instructor.Design principle no.4: Learning is facilitated through interactions with others.-Whenstudents verbalize their ideas to other students in the class, they clarify their thinking and organize their ideas [25,26].As such, when students answer questions within the PSET and LEPS curricula, they are encouraged to discuss their thinking with other students (either other group members in PSET or classroom neighbors in LEPS).PSET includes specifically designed opportunities (usually at the beginning and end of each activity) for students to share and defend their ideas with the whole class.In LEPS, the instructor can call on students to share their responses to specific questions.
Design principle no.5: Learning is facilitated through establishment of certain specific behavioral practices and expectations.-Inthe research literature these classroom behaviors and expectations are known as norms [27], and in PSET and LEPS the norms are intended to model the normal practices of a scientific community.For example, in both PSET and LEPS, students are expected to use experimental evidence to support their ideas.Students are expected to reach a consensus on ideas, helping them recognize that scientific ideas are built on consensus.They are expected to assume that scientific ideas make sense, and if they are confused by specific ideas they need to take an active role in figuring things out.They also are expected to contribute ideas to group and class discussions, and to constructively critique others' ideas.In PSET and LEPS these norms are established and reinforced by comments made by the instructor, by specific prompts and comments in the curriculum, and by students' own comments when sharing answers with the whole class.

III. DESCRIPTION OF LEPS
A. Content structure LEPS contains in-class lessons and computer-based homework (with online quizzing) addressing physical science content, the nature of science, and nature of learning.The content of LEPS focuses on the themes of interactions, energy, forces, atomic-molecular theory, conservation of matter, and gas behaviors.The content learning objectives address many of the physical science-related benchmarks and standards in the AAAS Benchmarks for Scientific Literacy [28] and National Science Education Standards [9].Table III lists the six units in the LEPS curriculum, each of which covers the same topics as the corresponding chapter of the PSET curriculum.The tables of contents of the PSET and LEPS curricula are included in the Appendix.

B. Structure of a unit
Each unit was designed to address one or more of the benchmarks or standards.Because a benchmark typically addresses a large idea, each was broken down into a subset of ideas that could be addressed at the level of one or two 25-minute lessons.Each unit consists of 8-13 lessons, the last of which focuses on evaluating explanations of phenomena for accuracy, using the ideas developed in that unit.In addition, three units include lessons focused on the nature of learning (NOL) and the nature of science (NOS).

C. Structure of a lesson
Large-enrollment university courses typically comprise either two 75-minute periods or three 50-minute periods per week.We divided the LEPS material into 25-minute lessons during which the instructor guides the entire class using PowerPoint TM slides.Class time is spent on responding to ''clicker questions'' (which students answer with electronic response devices, often following a discussion with nearest neighbors [13]), observing videos of experiments and simulations, and answering ''making sense'' and ''summarizing'' questions.As the instructor navigates through the slides, the students fill in data tables and answer corresponding questions in printed lesson sheets that guide their work.We expect 2-3 such lessons to be completed during each class meeting, which is consistent with field testers' experiences.
Each LEPS lesson consists of three sections.First a brief Purpose and Key Question(s) section gives the rationale for the lesson and provides the lesson's focus.The second and major portion is the Predictions, Observations, and Making Sense (POM) section.The questions in this section guide students through predictions, observations, and inferences to help them answer the key question(s).The choice of questions was informed by the extensive literature on students' understanding of physical science.The POM section often begins with a clicker question about a scenario to elicit students '   Certain lessons focus on the NOL or the NOS and have a structure identical to that of a conceptual lesson.However, instead of showing videos of experiments, the instructor shows classroom video of either elementary-aged children or students from previous LEPS classes talking about science questions.LEPS students are expected to select excerpts from provided transcripts to support claims about the students' ideas.Because of length limitations, we will not discuss these NOL and NOS lessons further in this paper [29].
A PSET activity typically has four major sections: Purpose and Key Question, Initial Ideas, Collecting and Interpreting Evidence, and Summarizing Questions.Thus, the structure of a LEPS lesson is similar to an activity in the PSET curriculum, with one exception: PSET includes a specific section that addresses students' prior knowledge, while LEPS does not.Instead, in LEPS, questions eliciting initial ideas are incorporated into the POM section.Pedagogically, there are three major differences in how the lessons or activities are implemented: (1) in PSET, students work at their own pace throughout the main sections of the activity, whereas in LEPS the instructor guides the whole class through all the questions; (2) in PSET, students perform their own experiments, whereas in LEPS the instructor shows movies; and (3) in PSET, specific questions are intended to support significant intragroup and whole class discussions, whereas in LEPS these often appear as clicker questions.An example of a PSET activity is described in Sec.IVA.

D. Homework
A homework assignment is associated with almost every 25-minute in-class lesson.This means that students are typically responsible for 2 or 3 homework assignments per class meeting.The homework is computer based and consists of a series of narrative text and links to videos of demonstrations, experiments, or simulations [30].Students certainly could run the actual simulations, but to ensure that students focus on specific aspects of the simulations, we have made movies of simulation arrangements for students to view, followed by questions with feedback.The focus of each assignment is either to practice using the ideas introduced in the associated lesson or explore new, but related, ideas.The homework is an Adobe Flash multimedia module that students access either online (as part of the learning management system) or on the student resources DVD [31], along with associated materials (such as experiment or simulation videos).Questions within the homework provide feedback to students but are not graded.A graded quiz at the end of each homework activity consists of multiple-choice questions.Students access the quiz online through a learning management system such as Blackboard or Moodle.The learning management system automatically scores the responses and records them.In this way, students get instant feedback on their quiz scores and the instructor does not need to grade the homework (important in a large class).Examples of homework from LEPS are provided in Sec.V E.
Whereas a LEPS homework is highly structured and provides immediate feedback, a typical PSET homework is much more open-ended.It includes guiding questions, but any feedback students receive is usually delayed until graded papers are returned.Section IV B describes an example of a PSET homework.

E. Instructor materials
The instructor manages LEPS as a large guided-inquiry class.The curricular materials include Microsoft PowerPoint TM slides for the instructor for each lesson.The slides include all clicker questions and making sense questions (that students answer on their printed lesson sheets), embedded movies of demonstrations, experiments or simulations, and summaries of key definitions and ideas.The final slide for each lesson briefly describes the associated homework.Each slide also includes implementation notes for the instructor (in the ''Comment'' panel of PowerPoint TM ), with answers to the questions.
Instructors are also provided with instructor versions of the student workbook lessons.These lesson keys include implementation notes that indicate where each slide fits into the lesson.Though the implementation notes are similar to the comments in the PowerPoint presentations, the notes in the keys are more in-depth.The keys also contain solutions to the homework quizzes.
In contrast, PSET instructors use an online teacher's guide for notes on implementing the curriculum.The PSET teacher's guide Web site offers a description of the pedagogy, a detailed list of target ideas, and material specific to a chapter or activity.For an individual activity, these materials include detailed notes on each section of the activity and an activity key with additional notes.We chose not to include an online teacher guide for LEPS because the different presentation style allows implementation notes to be integrated directly into the instructor's materials.

IV. EXAMPLE OF A PSET ACTIVITY AND HOMEWORK
This section describes a typical PSET activity and homework assignment, and Sec.V describes the corresponding lessons and homework assignments in LEPS.This will facilitate direct comparison of the treatment of similar material and illustrate how differences in course format influenced the two curricula.Throughout, we will highlight the similarities and differences that resulted from the adaptation process.Section V also describes how the design principles (Table II) are instantiated in LEPS.We follow Ball and Cohen, who distinguish between the curriculum materials per se and the ''enacted curriculum,'' which they describe as ''jointly constructed by teachers, students, and materials in particular contexts'' [32].We will specifically consider how the curriculum materials embody the design principles and how instructor and students using the curriculum enact the design principles.Since we will not be discussing further how the design principles were instantiated in PSET, we encourage the reader to refer to Goldberg, et al. [19] for a discussion relevant to the design principles in the Physics and Everyday Thinking curriculum [5], which are similar to those in PSET.
To provide a concrete example we will focus on Chap. 2 in PSET (and Unit 2 in LEPS), Interactions and Forces.Our design of this chapter (or unit) was informed by the research literature describing the common ways that students think about forces and motion based on interpretations of their everyday experiences [33].For example, students may think that giving an object a quick push transfers force to it that is then carried by the object until it eventually wears out.They also may think that if an object is moving, there must be a force in the direction of motion causing it to move.The first activity in Chap. 2 of PSET begins to deal explicitly with these issues.However, students are not expected to resolve all the relevant issues until they have worked through the entire chapter.
As mentioned previously, each PSET activity consists of several sections: Purpose and Key Question, Initial Ideas, Collecting and Interpreting Evidence, and Summarizing Questions.Students work through these sections in small groups, filling out questions on the activity sheets, conducting experiments with hands-on materials and computer simulations, and at appropriate times sharing their ideas in whole class discussions.Guiding questions in the activity are intended to engage students in substantive discussion.The class norms encourage students to share their thinking and critique the thinking of other students, not only in small groups but also during whole class discussions.

A. Description of PSET activity: Chapter 2 Activity 1-Interactions and Force
Activity 1 in Chap. 2 (C2A1) of PSET is called Interactions and Force [34].Feedback from field testers indicates that the activity takes about 120 minutes to complete.Much of this PSET activity is very similar to the first activity in Chap. 2 of the Physics and Everyday Thinking curriculum, and we provide only a brief description of the PSET activity below.The PET activity is described in detail elsewhere [19], including a case study that gives the reader a sense of the issues that emerge among a small group of students as they work through the activity, collecting experimental evidence and answering the questions.In the following paragraphs we will refer to that case study as the PET case study [19].
The Purpose section reminds PSET students that, in Chap. 1, they described contact push-pull interactions in terms of energy and that, in Chap.2, they will describe similar interactions in terms of pushes or pulls, which are called forces.The activity is guided by the key question, ''How does an object move when a force is acting on it?'' The Initial Ideas section asks PSET students to think about a soccer player kicking a soccer ball.After the kick, the ball rolls across the grass and gradually comes to a stop.Students sketch a speed-time graph for the motion of the ball from the moment the player's foot first comes in contact with the ball until it comes to a halt again.They then label on their graph where the foot was in contact with the ball and where there was a force pushing the ball forward.Finally, they draw pictures of the ball and show what forces (if any) they think are acting on the ball during the kick and also after the foot has lost contact with it, but before it stops.Student groups answer these questions, sketch graphs and drawings on large white boards, and then share their thinking with the whole class.As illustrated in the PET case study [19], it is not uncommon for students to draw an arrow on the after-the-kick picture representing the force from the kick.
The Collecting and Interpreting Evidence section consists of two experiments using carts, tracks, and motion probes, an exploration using a computer simulator, and several questions to answer.For the first experiment, PSET students give a low-friction cart three quick pushes as it moves along a track and use a motion sensor connected to their computer to display a speed-time graph.The subsequent observation and making sense questions ask students what happens on the graph during those periods when their hand is in contact with the cart, what happens during the periods when their hand is not in contact, and what evidence from the graph would suggest there is a force acting on the cart pushing it forward.To focus the students' attention on the relationship between the motion of the cart and whether or not there is a force acting on it, the activity provides the following hypothetical conversation between three fictional students.Samantha: The force of the hand is transferred to the cart and is carried with it.That's why the cart keeps moving after the push.
Victor: The force of the hand stops when contact is lost, but some other force must take over to keep the cart moving.
Amara: After contact is lost there are no longer any forces acting on the cart.That's why it moves differently.
PSET students then discuss which of the three fictional students they agree with and why.This question usually elicits substantive discussion, since the three fictional students express commonly held views about the relation between force and motion (see the PET case study [19]).
For the second experiment, students consider what happens when a cart is given a continuous push.They first draw what they think the speed-time graph would look like.Then they try to give the cart a constant continuous push with their finger.Generally, they find this very hard to do because the cart ''runs away'' from their finger as it accelerates.The curriculum then suggests using a cart with a fan mounted on it rather than pushing with a finger.After letting the fan cart push against their finger, and listening to the sound of the fan's motor, students conclude that the running fan seems to exert a constant continuous push.Finally, students release the fan cart starting from rest.As it moves along the track, they observe a linear speed-time graph with positive slope and associate this motion with a continuous constant force being exerted on the cart.
During the next part of the Collecting and Interpreting Evidence section, PSET students work with a computer simulator.First, the activity sheets show a simulator-drawn speed-time graph representing the motion of a cart subject to a series of intermittent short-duration, constant forces (see Fig. 1) and asks students to sketch what they think the corresponding force-time graph would look like.Students then run the simulation, controlling the simulated push with the space bar.The simulator generates a force-time graph very similar to the one shown in Fig. 2. As illustrated in the PET case study [19], this sequence can engage the students in extensive discussions regarding their prediction for the force-time graph and their interpretation of the simulation results.
After working with the simulator, a series of making sense questions asks PSET students whether there were forces on the simulated cart during and between the pushes.These questions are intended to help students conclude that the speed increases only when the force acts, and, conversely, that when the force does not act, the speed does not change.(See the PET case study [19] for an example of the kinds of issues that emerge when students answer these questions.) The students next simulate a constant strength force acting continuously on the cart.The resulting speed-time graph is linear with positive slope; the force-time graph starts from zero but increases almost instantaneously to a value that remains constant thereafter.Students can thus conclude that when a force continues to act on an object in the direction of its motion, the object continuously speeds up [35].
The last part of the Collecting and Interpreting Evidence section introduces PSET students to drawing force diagrams and comparing force and energy descriptions of interactions.The energy diagrams in Fig. 3 show three time periods for a cart being pushed by a hand: before the interaction, during the interaction (when the hand is in contact with the cart), and after the interaction (after the hand is no longer in contact).Students sketch the force diagrams corresponding to instances during these three time periods.Figure 4 shows an appropriate answer for the force diagrams (but the student materials do not provide these answers).
The Summarizing Questions section consists of six questions.The first two questions ask what happens to the motion of a cart when a force acts on it-either starting from rest or while it is already in motion.The third question asks why it was difficult to exert a continuous constant force on a cart with a finger.The fourth question considers a fan cart speeding up along a track and asks what would happen to its speed if the fan suddenly stopped turning.The fifth question asks at what point the force stops acting on the cart when you give it a quick shove.The last question asks what is transferred from a source to a receiver during an interaction: energy, force, both, or neither.This last question tends to generate much discussion since it requires students to tease apart the separate concepts of force and energy.The PET paper [19] discusses both small group and whole class responses to this question.For each summarizing question, students are expected to explain their reasoning and, where appropriate, to cite evidence from the activity to support their answer.
Following the Summarizing Questions section, there is a discussion of how to write scientific explanations using force ideas and a comparison of explanations using both energy and force ideas.Finally, PSET students construct their own explanation for why a fan cart speeds up after being released, using the idea(s) developed in the activity.A copy of PSET C2A1 is included in the Appendix.

B. Description of PSET C2A1 homework
PSET homework assignments are paper-and-pencil based with a structure similar to the in-class activities, including the same four sections.As described in the Purpose section, the C2A1 homework provides students with practice applying the ideas developed in C2A1.The Initial Ideas section asks students to imagine applying a constant forward force to a friend currently coasting on a skateboard (ignoring friction).See Fig. 5.
The student sketches speed-time and force-time graphs that include periods prior to, during, and after the push, and draws force diagrams for instants during each of these three periods.In the Collecting and Interpreting Evidence section, the students work with a computer simulation analogous to the ''pushing a skateboarder'' situation described in the Initial Ideas section.They generate speed-time and force-time graphs and compare the simulation's graphs with their predicted graphs and force diagrams.The Summarizing Questions section includes questions about the observed graphs and the simulation, and then asks the students to write an explanation for a similar situation.
The PSET homework does not provide feedback to students about their answers to the questions (although when running the simulator, they can compare their predictions to the simulation).The instructor is responsible for discussing the homework during class, posting answers, grading the homework, and providing feedback.A copy of the PSET C2A1 homework is included in the Appendix.

V. EXAMPLE OF LEPS LESSONS AND HOMEWORK
In this section we describe the first three lessons in Unit 2 of the LEPS curriculum.The purpose of this description is to illustrate how specific curricular elements (such as activities, experiments, and questions) from PSET were incorporated into LEPS.In particular, we highlight differences in form while noting similarities in pedagogical purpose (namely, the design principles discussed in Sec.II B).These lessons were adapted from the single PSET activity, C2A1, described in Sec.IVA.The individual LEPS lessons are significantly shorter than those of PSET, and the equivalent content of one lesson in PSET is spread over several lessons and homework activities in LEPS.For example, in the lessons compared here, the single PSET C2A1 takes 120 minutes, while the same content is covered in three LEPS lessons (Unit 2 Lessons 1-3) for a total of about 85 minutes.As described in Sec.II A, this basic structural feature was a response to the shorter meeting times associated with lecture-format courses (typically 50 or 75 minute periods).Copies of the three LEPS lessons are included in the Appendix [36].
The key question for PSET activity C2A1 was, How does an object move when a force is acting on it?This single question was separated into three separate and more specific questions that are the key questions for the corresponding three lessons in LEPS: (U2L1) When does the force of a quick push stop acting on an object?(U2L2) When an object is moving, does this mean there must be a force pushing it in the direction of its motion?(U2L3) How does an object move when a force of constant strength continuously pushes it forward?
A. Following a group of students In addition to describing the lessons below, we include comments made by students in a LEPS class.During the fall semester 2009 LEPS was offered for the first time in a large class setting at the institution of one of the authors (F.G.).There were 85 students enrolled in the class.All the students were undergraduate liberal studies majors who intended to become elementary school teachers.They were predominantly seniors, with a few juniors.The three lessons described here took a total of nearly 85 minutes in this implementation.Students sat in movable chairs in a large nontiered room and there was sufficient space between adjacent rows of chairs for the instructor to walk amongst the students when he chose to do so.Students were recruited (voluntarily) to sit in chairs near the front of the room and be videotaped as they discussed their ideas with each other.At the beginning of Unit 2 a new group of four students was chosen to be videotaped.The students were selected on the basis of their willingness to be videotaped, not based on any performance criteria.We will refer to the students by the pseudonyms Brad, Mary, Laurie, and Maia.Based on course grades, these students represent the midrange of academic performance in the class.All the other students in the class had consented to be videotaped during whole class discussions.Some of their comments are included below, and again we identify them by pseudonyms.
We have two purposes for providing examples of student discourse.First, we wish to provide the reader with a sense of the kinds of discussions and ideas that emerge when students respond to the clicker questions and making sense questions in the LEPS materials.The reader can compare the substance of these responses with those made by students working through the much more self-guided PET curriculum as reported in the PET case study [19].The second purpose is to show how the instructor and students using the curriculum enact the five design principles described in Sec.II B and in Table II in an actual classroom implementation of LEPS.

B. Unit 2 Lesson 1: Interactions and Force
Unit 2 Lesson 1 (U2L01) introduces students to the force description of interactions.The key question is, When does the force of a quick push stop acting on an object?The pedagogical purpose is to immediately begin addressing the common idea that ''force'' is something that an object carries along with it and is transferred between objects when they contact each other [33,[37][38][39][40]. Compared to physicists, students often ascribe a different meaning to the word force and often talk about the ''force of an object'' as if it were a property of the object.This lesson provides students with evidence that a force is an interaction between two objects, not something carried by an object or something associated with an individual object.
At the beginning of the POM section, students consider the clicker question shown in Fig. 6 [41].This question follows from design principle 1 (learning builds on prior knowledge), and is similar to the question in the Initial Ideas section of PSET C2A1.However, in contrast to PSET, LEPS students are not asked to draw the speedtime graph themselves.Furthermore, in determining when the force stops acting, PSET students are given the open-ended prompt ''indicate on your graph,'' while LEPS students are asked to choose between a specified set of points.However, the choices in LEPS clicker question (CQ) 1-1 were based on students' common responses to similar questions in the PSET and PET curricula, which suggested reasoning such as: the force is applied for only a very short time, after which the puck speeds up before starting to move with constant speed (choice A); the puck must speed up ''just a little'' after the ''hit'' is finished (choice B); the force stops as soon as contact is lost (choice C); as long as the puck is in motion (even after contact is lost) it is influenced by the force of the stick, or the force is carried along as the puck moves (choice D).
Although the clickers provide quantitative information on students' responses, the instructor has limited access to students' reasoning and instead must interpret the results based on previous experience or knowledge of the literature on students' ideas on force and motion.
In the fall 2009 field test, the instructor presented the question and asked students to respond individually.The distribution of responses was A ¼ 4%, B ¼ 17%, C ¼ 70%, D ¼ 9%.The large number of students choosing C was surprising, considering that the physics education research literature suggests that many students often think a force continues to act on an object after contact is lost.However, students may have been recalling what they had learned previously in Unit 1 Lesson 2, which explored a similar situation in terms of interactions instead of forces [42].
Next, the instructor showed a movie of a low-friction cart being given three quick pushes as it moved along a track, with the speed-time graph displayed.In the PSET activity described in Sec.IVA, the students generate the speed-time graph themselves.LEPS students do not conduct the experiment themselves, but including the movie supports the norm that arguments in science should be based on evidence (design principle 5-learning is facilitated through establishment of norms).CQ 1-2 asks the students to imagine what force-time graph would correspond to this speed-time graph, and choose from three possible graphs (see Fig. 7).Again, the distractors are drawn from the developers' experience with the PSET curriculum and represent typical responses from PSET students.These include the idea that the force is transferred to the cart during each push, but that the force slowly runs out after the push (choice C), and the idea that whenever there is motion, there must be force, and that more speed means more force (choice B).
The class responses were A ¼ 65%, B ¼ 24%, C ¼ 11%.The instructor then showed a movie where the cart was given three successive pushes, while both speed-time and force-time graphs were generated from motion and force sensors.The class agreed that the evidence supported graph A from CQ 1-2.The students then talked with their neighbors and answered several making sense questions to help them interpret and draw conclusions from the movie.The last making sense question focuses on the issue of transfer: Does the force of the hand continue acting on the cart after it has lost contact?How do you know?After students discussed the questions with their nearest neighbors, the instructor called on students in the class to share their answers and reasoning with the whole class.
This sequence illustrates how LEPS reflects the design principles.The curriculum elicits students' prior knowledge through the first clicker question (supporting design principle 1) and provides opportunities for students to engage in discussions (supporting design principle 4).LEPS also promotes norms (design principle 5) that ideas should be supported by experimental evidence, that scientific ideas should make sense, that students need to take an active role in learning, and that students are expected to contribute ideas to group and class discussions.
We now consider whether, in the fall 2009 field-test implementation, the curriculum functioned as intended, helping the instructor and students create a learning environment consistent with the design principles.For the last making sense question the instructor called on students in the videotaped group; part of that discussion is included below.The use of [???] indicates that a comment is unintelligible (mainly due to background noise from other students) and (. ..) indicates discourse that was not included for brevity.Descriptive comments are included in brackets.The numbers are included for easy reference to specific statements.4. Maia: Well, I said yes.I think I was the only one who said 'yes,' which I'm probably wrong.But, I said 'yes' because, um, uh, it remains in motion and if it didn't, if there was no force continue acting on it, it would immediately stop after the hand stopped the interaction.So, because it remains moving there is a force acting on it after the hand released it.
5. Instructor: Okay, that makes sense.What's the counter argument?6. Mary: Well then, I said no, just because it may be gaining momentum from the force but it doesn't mean the force is still pushing on it.Like, it just gained the speed from the force, if that makes sense.
7. Instructor: But there is no force on it?8. Mary: Right.The force is gone.
The above excerpt illustrates how the instructor (statement 5) and Mary (statement 6) maintained the class norm that ideas are supposed to make sense (design principle 5).Also, in statement 4 Maia drew on her beliefs about forces and motion (prior knowledge) to answer the question, consistent with design principle 1.
The instructor then spoke about the importance of using evidence to support their answer, helping to maintain the evidence norm from design principle 5.A student raised her hand to support her ''no'' answer.In her response she talked about how she interpreted the speed-time and forcetime graphs (Fig. 7) to support her answer (design principle 3-learning is facilitated through interaction with tools): 9. Elizabeth: On the force [time graph], the force when it's going down, and then you correspond that with the speed-time graph, the speed is staying constant.And then when there's an increase, it's speeding up.So, in the question it says 'after the hand has lost contact.'So, after the hand [it] is just coasting, you look at the speed-time, uh, the force graph, there is no force being shown.

FIG. 7 (color online). Second clicker question in U2L01.
This careful attention to the comparative features of the speed-time and force-time graphs as a means to guide her reasoning was also a common practice in the more self-guided PET and PSET curricula (see the PET case study [19]).
The Summarizing Questions section begins with a question specifically asking about transfer: Do you think the force of the hand was transferred from the hand to the cart during the push, and then continued to act on it after contact was lost?What evidence supports your thinking?The members of the videotaped group discussed the question.22. Mary: I think because there is no friction.
In the above excerpt Maia seemed willing to go along with the group's thinking because of the evidence (statement 18), consistent with design principle 3, but not necessarily because the answer made sense to her (in statement 12 she admitted that the question was confusing to her), consistent with design principle 5.The last comment by Mary (statement 22) suggests that she may have realized that the reason the cart continued to move without slowing down after the push was over was because there was no friction; presumably, if there were friction, the cart would slow down.Finally, Laurie asked questions to the group, expecting that her fellow group members could help her figure it out (statements 19 and 21), consistent with design principle 4.
The class worked on this and two other summarizing questions and then the instructor asked some students to share their thinking with the rest of the class.One student claimed there is no transfer, and described how on the force-time graph the force drops quickly to zero, and when it is zero force the speed-time graph shows a horizontal line.The instructor then asked if anyone else in the class had something to say about that, and called on a student.
23 Nicole I think the force was transferred to the cart only during the push, and then that it did not continue to act on it after contact was lost.Because the graphs show that during the time that the force was showing on the forcetime graph that was the time the speed was increasing for the cart so it shows that force was transferred from the hand to the cart.But then after the force-time graph goes down to zero, the cart continues.It like, it levels out, so that means the force is no longer acting on it.
Nicole may have thought that force is a quantity that is possessed by an object [33,37], and that during the interaction this quantity is transferred from the hand to the cart, but after contact is lost, the cart no longer has it.However, the instructor did not pursue that conjecture with Nicole.
The last summarizing question is the same as CQ 1-1 (see Fig. 6).The results were A ¼ 11%, B ¼ 5%, C ¼ 79% (the correct answer), D ¼ 5%.Although there was a slight increase in the percentage of students who recognized that the force stopped acting when the speed stopped increasing (that there was no transfer of force), about one-fifth of the class at the end of the lesson seemed guided by alternative ideas.The lesson provided the opportunity for students to think about what is transferred and what is contact dependent, and the above transcript excerpts suggest that at least some of the students were engaged with the issue.Nevertheless, because the issue is complex, some students need additional time before changing their ideas (consistent with design principle 2).The lesson took 23 minutes.The instructor then moved immediately to the next lesson.

C. U2L02: Motion and force
The key question for U2L02 is, When an object is moving, does this mean there must be a force pushing it in the direction of its motion?This lesson has two pedagogical purposes.One is to address the common idea that if an object is moving there must be a force on it in the direction of its motion [33,37].The lesson provides evidence to help students see a force as an interaction between two objects, and, given that definition, to conclude that motion alone is not evidence that a force is acting; only changing speed provides that evidence.(Later in the unit the term ''net force'' replaces the term force.)The second purpose is to provide additional opportunities for students to consider what, if anything (force, energy, something else), is transferred during or after an interaction.
The Predictions, Observations and Making Sense section begins with students thinking about a low-friction cart rolling along a track after it was given a quick push.To encourage students to consider and examine some possible relationships between motion and force, a clicker question presents three different ways of thinking about this relationship expressed by three fictional students, Samantha, Victor, and Amara (see Figs. 8 and 9).Samantha expresses the view that force is transferred to an object during the interaction and the object then carries the force with it, which accounts for why the object keeps moving after the push.Victor's view is that force is not transferred during the interaction, but the fact that the object keeps moving implies that there must be (some other) force that keeps acting on the object after contact is lost.Amara's view, consistent with the physicists' view, is that force is not transferred and that because there is no force acting on the object in its direction of motion, the object stops speeding up [43].This question is very similar to a question in Chap. 2 Activity 1 of PSET (and also in PET), described above in Sec.IVA.However, whereas in LEPS students respond to a clicker question and tend to talk to their neighbors for only about a minute or two prior to voting, in the PSET and PET contexts a group of students working together could spend several minutes discussing and writing down their ideas (see PET case study [19]).
In the fall 2009 field test, students discussed this question with their neighbors and then voted with their clickers.
The results were A ¼ 11%, B ¼ 27%, C ¼ 62%, D ¼ 0%, indicating that the majority of the class thought Amara's ideas made the most sense, but about a quarter of the students thought Victor's idea made the most sense.These students seemed to be thinking that if an object was moving, there must be a force acting on it in its direction of motion.
Next, students see a speed-time graph taken from a computer simulation of a cart moving along a track while it is given three successive quick pushes.CQ 2-2 asks what the corresponding force-time graph would look like.See Fig. 10.(This question is similar to that asked in U2L01 CQ 1-2, described in Sec.V B, but here the graphs are generated by a simulation, rather than by the motion and force probes.) In the fall 2009 field test the class discussed this question and then voted A ¼ 65%, B ¼ 6%, C ¼ 29%.When comparing the results from this clicker question with CQ1-2 (see Fig. 7), about the same number of people chose graph A (but not necessarily the exact same students), but with the remaining students there was a shift away from graph B to graph C. Graph C also can be interpreted as supporting Victor's ideas from CQ 2-1 (see Fig. 8) since it shows there exists a force after the push that seems distinct from the force of the push itself [44].As with CQ 1-2, LEPS students choose from a set of graphs, while in the PSET activity described in Sec.IVA, students draw a force-time graph prediction.
The instructor then played the simulator movie, which showed both speed-time and force-time graphs being generated as three successive pushes were provided to the simulated cart.(In the corresponding PSET activity, students run the simulation themselves.)The class agreed that the simulator evidence supported choice A from CQ 2-2.They then answered some making sense questions that followed, intended to help students make connections between the forces shown on the force-time graph, the actual pushes of the cart, and the speed-time graph, thus explicitly addressing the key question for the lesson.Again we see how the design principles are realized in LEPS: the use of simulators (design principle 3, interaction with tools), exploring a concept in a variety of contexts (design principle 2, learning is a complex process), opportunities to answer questions, discuss with others, and receive (general) feedback from the instructor (design principle 4, interactions with others).The Summarizing Questions section helps students address issues raised in both U2L01 and U2L02.The second question explicitly deals with the issue of transfer and the differentiation between energy and force: During a contact push-pull interaction what do you think is transferred from the source to the receiver: energy, force, both, or neither?Explain your reasoning.(The question is identical to one of the summarizing questions in the PSET activity.)In response to this question, the group of students in the fall 2009 field test discussed the distinction between force and energy and came to an agreement that energy, not force, is transferred: 24.Mary: There is where I think they bring the idea in.I said, force got it moving, but then energy keeps it going.Because I think it said it has kinetic energy.But I'm not sure, I'm not sure.(. ..) [Group is silent for one minute while they write on their lesson sheets.]25.Maia: It has to be energy, right?It can't be force.It has to be energy.(. ..) 26.Laurie: Like Brad was saying, it was like kinetic energy.There's a potential energy and there's kinetic energy.So the kinetic energy is when it first went in motion.So she [Mary] was saying the force is when there's an actual interaction, and pushing the cart, and then energy is left after the cart stays at a constant speed.This discussion illustrates the degree of challenge in teasing out the differences between energy and force, consistent with design principle 2 (learning is a complex process).The instructor then called on groups around the room to answer the summarizing questions.He called on Morgan to answer the first question: While the simulator cart was moving along the track, was there a force pushing it forward the whole time or only at certain times?What evidence supports your answer?She seemed unsure about her answer, suggesting she was still struggling with the distinction between energy and force: 36.Morgan: When I think about the force, I think about the physical hand pushing the cart.Um, which is available [for some of the] time, but I think energy is there the whole time.The question is whether or not energy is force.
In response to her question about whether energy is force, the instructor suggested they move on to the second question (see above), thinking it might help clarify the issue.He called on another student.37. Sara: Um, we know that energy is there because there is this little movement in the object.But, um, we were deciding whether or not there was force, kind of like the same thing [looking at Morgan].I don't think that there's force, just because watching the force graph, as soon as the hand releases there is no force.But, if the force remains with it, then force would be with the energy.
38. Instructor: Okay, someone else have a thought on this?Other people must have been grappling with that.Share your thinking.[Calls on Brenda] 39.Brenda: Well, I would think that if the force graph isn't showing that there is any force, then, you know, there is no force even if, like, even if energy goes with it, then energy doesn't count the same as a force.Because I think it would be picked up by the graph.
40.Instructor: Okay, and the graph is showing only force?
41. Brenda: Yeah.There's no force when it's not touching it.There's no force.
The above excerpt illustrates how the features of the curriculum help create a learning environment consistent with the design principles.Sara used evidence from the graphs to support an answer (statement 37, design principle 3), but she also indicated her uncertainty (design principle 2).Brenda then built on what Sara said (statement 39, design principle 4), helping to clarify that student's thinking (design principle 5).
The last summarizing question, CQ 2-3, gives students a force-time graph and asks them to choose which of three possible speed-time graphs could correspond to it (see Fig. 11).There was very little class discussion and the students then voted on it: With the majority choosing C (the correct answer), the instructor brought the lesson to a close and moved on to the next lesson.The lesson took 29 minutes.

D. U2L03: Motion with a continuous force
In both U2L01 and U2L02, students consider situations in which an object is acted on by an impulsive force in the direction of its motion.This helps students consider whether force is transferred to an object during an interaction and whether the fact that an object is moving implies there must be a force acting on it in the direction of its motion.In U2L03, students consider the following key question: How does an object move when a force of constant strength continuously pushes it forward?
After the instructor described what it means for a force of constant strength to continuously push an object forward, the students considered CQ 3-1, shown in Fig. 12.(In the PSET activity, the students were prompted to predict the shape of this graph themselves.) The videotaped group discussed CQ 3-1.Mary argued for graph C, reasoning that the speed should increase like it did in CQ 2-2, but without leveling off because the force does not stop.Maia initially agreed, but then began arguing for graph B (speed increasing, then constant).Brad seemed to think that the cart can only speed up so much, and then must level off.51.Maia: Yeah, the initial force that you put on it; it has to speed up eventually and then stay constant, right?
The students in the above excerpt drew on their prior knowledge (consistent with design principle 1).Mary (in statement 44) based her argument on what she had learned in the previous lesson.Both Brad (statement 50) and Maia (statement 51) seemed to refer to their everyday knowledge that no object continues to speed up indefinitely.
The class then voted with their clickers.The results were A ¼ 18%, B ¼ 41%, C ¼ 41%, indicating a split in the class between those thinking the speed increases the entire time and those thinking the speed either remains constant the whole time or increases for a short time and then remains constant.At this point, the class period ended.
The lesson continued in the next class period.The instructor began by showing a movie of someone trying to push on a cart with his fingers with a constant strength push the entire time the cart remained on the track, but the resulting force-time graph showed that the strength was not actually constant.The next movie showed that when a cart with a running fan mounted on it pushed against the force probe, the resulting force-time graph was constant.Students then concluded that a fan mounted on a cart would be able to exert a continuous force on the cart with constant strength.After this, the instructor showed a movie of the fan cart moving along the length of the track.The resulting speed-time graph confirmed graph C in CQ 3-1 (Fig. 12).
In response to seeing the speed-time graph with a positive slope, a student in the class asked, ''Wouldn't it eventually level out?''It would certainly seem reasonable to the students that the cart could not speed up indefinitely, that at FIG. 12 (color online).First clicker question in U2L03.some point it would stop speeding up, as suggested in statements 50 and 51 above.To begin to address that issue, the instructor described a scenario where the fan cart was allowed to run along three tracks placed end to end.CQ 3-2 asks students to predict what they think will happen in this scenario.In the fall 2009 class, 53% of the students predicted its speed would increase along the first track, then level off; 39% of the students predicted the fan cart would speed up the entire time; and 8% of the students predicted the fan cart would speed up at first and then begin to slow down.The instructor then showed a movie of the situation.The movie displays time clocks showing how much time it took the cart to move along each of the three tracks; it takes less time for each successive track, indicating that the fan cart speeds up continuously along the three tracks.The fall 2009 students concluded that the cart does speed up continuously.[The students did not determine, however, whether the rate of speeding up was constant along the three tracks.They only concluded that the cart's speed continued to increase.]A number of students in the class again raised the issue that eventually the fan cart would have to stop speeding up, since it could not reach infinite speeds.The instructor acknowledged that and briefly mentioned that, as the cart sped up, the air would exert a backwards force on the cart (air drag), with the effect becoming more noticeable at higher speeds.He said they would consider this effect in detail later during the unit.
The Summarizing Questions consist of two clicker questions giving students the opportunity to synthesize the ideas developed over the first three lessons.Figure 13 shows CQ 3-3.
The students in the focus group, without much discussion, chose answer B. The results for the whole class were A ¼ 0%, B ¼ 84%, C ¼ 0%, D ¼ 16%.The instructor asked, ''Someone who chose B, how would you respond to somebody who chose D?'' A student answered that it is not definite evidence if it is just moving since they saw before that if it is moving at constant speed it does not have a force acting on it [45].The last clicker question, CQ 3-4, asks students to think about what would happen to a moving object if the force acting on it was suddenly removed.See Fig. 14.
The class voted and the results were A ¼ 2%, B ¼ 84%, C ¼ 14%.The instructor then called on someone who chose B to argue against choice C. The student who responded referred to previous evidence, and claimed that if there were no force on it, there was no way it would continue to speed up.This ended U3L03, which took a total of 31 minutes, divided over parts of two class periods.

E. Description of U2L01, U2L02, and U2L03 homework
The LEPS homework is computer based.Each homework activity consists of a ''tutorial'' section where students are introduced to ideas (through readings, diagrams, and/or movies of simulations and experiments) and given practice answering questions (usually multiple choice or matching) with feedback, and then a quiz section consisting of multiple-choice questions.The homework quizzes are intended to be done in a learning management system so that the grading is automatic.On average, students in the fall 2009 field test reported spending approximately 36 minutes working through each assignment, including the quiz.(The range was 15 to 90 minutes.)Screen shots of the three computer-based LEPS homeworks are included in the Appendix.
The U2L01 homework introduces students to force diagrams and units of force (newtons), and then leads students through several examples.The homework quiz asks a series of questions very similar to ones the students had just worked through.The content of this homework is similar to some of the content included in C2A1 of PSET.
The U2L02 homework focuses on comparing energy and force descriptions of interactions.Students select appropriate force diagrams corresponding to moments before, during, and after an interaction occurs (a person giving a cart a quick push).This is the same situation considered in the PSET activity, as shown in Figs. 3 and  4, described in Sec.IVA, except that PSET asks students to create the force diagrams on their own rather than choose from multiple-choice options.The quiz section asks students a few multiple-choice questions about force and energy descriptions of interactions.
The U2L03 homework is similar to the PSET C2A1 homework described in Sec.IV B, except that instead of open response questions, LEPS consists of a sequence of multiple-choice questions, first about the appropriate  speed-time graph, and then about the appropriate forcetime graph.Instead of providing direct feedback to each question, students then watch a movie of a simulation (the same simulation PSET students manipulate themselves) and answer questions involving interpretation of the speed-time and force-time graphs generated in the simulation movie.Some of these questions are multiple-choice versions of similar questions asked in the PSET homework.
The first two multiple-choice questions in the quiz section ask students to choose appropriate force diagrams for different times in the scenario involving pushing a friend on a skateboard.The third quiz question describes a scenario where a hockey puck is given a quick shove to get it moving, and three fictional students' ''explanations'' for why the puck speeds up.The LEPS students must decide which of the three explanations is appropriate.The final quiz question asks students to decide which of a number of statements about the relationship between force and energy is (are) correct.

VI. COURSE EVALUATION
Content learning goals [46] were assessed with a multiple-choice, physics and chemistry assessment (Physical Science Questionnaire, or PSQ).The PSQ consists of 28 items, including 19 items used with permission from Horizon Research, Inc. [47], 5 items used with permission from AAAS Project 2061 [48], and 4 items constructed by project staff.Items were selected to match, as best as possible, the content learning objectives identified during the curriculum development process and were evenly distributed among physics and chemistry.Figure 15 shows three of the questions from the PSQ, all of which focus on force and motion ideas.We will discuss these below.We did not attempt to assess students' efforts at developing, testing and evaluating models, and writing and evaluating explanations.Students' engagement in these practices of science is an important aspect of PSET, but LEPS provides much less experience with these kinds of activities.
In fall 2009 and spring 2010, the PSQ was administered as voluntary online pre-and postassessments in 10 LEPS classes and 17 PSET classes [49].Students' performance on the postassessment did not contribute to their course grades.These classes were taught in institutions that are geographically diverse, of varying sizes (from community colleges to research universities), and with various student demographics, thus providing a credible set of sites for evaluating the impact of LEPS and PSET on student learning.All LEPS field testers reported having prior experience with interactive engagement techniques, most commonly inquiry-based curricula in small courses, all but one had used clickers prior to the field tests, and all were familiar with learning management systems.Most PSET instructors had taught PSET at least once previously.Thus, both groups of instructors were experienced with instructional techniques relevant for the curriculum they were using.While most instructors implementing LEPS in the 2009-2010 academic year were doing so for the first time, as field testers for the new curriculum they had attended a 2-day orientation workshop on the curriculum content, format, and pedagogy during the summer.
The average class enrollment was 68 students in LEPS classes (range of 14-185) and 29 students in PSET classes (range of [16][17][18][19][20][21][22][23][24][25][26][27][28][29][30][31][32][33][34][35].Those students who completed both the pre-and postassessments, and also consented to have their assessment data used for project evaluation purposes, averaged 52% of the enrollment in LEPS classes (range of 36%-89%), and 56% of the enrollment in PSET classes (range of 34%-94%).To check for selection effects, for the largest of the LEPS courses we compared final exam grades for students in the matched data set (pre-and post-test scores, N ¼ 80) with final exam grades for students who were not in the matched data set (N ¼ 110, so the participation rate was 42%).There was no statistically significant difference between these groups: tð187Þ ¼ 1:6, p ¼ 0:11.The final exam was written by the curriculum development team, was representative of the entire curriculum, and was similar in format (i.e., multiple choice) to the PSQ.
Before comparing the results of the PSQ for the LEPS and PSET classes, we first describe the responses of LEPS students and three sample PSQ items.The average pre-test score on the 28 items for 425 students who took both the pre-test and post-test across all LEPS classes was 38:0 AE 0:6% and the average post-test score was 56:8 AE 0:9% (uncertainties are standard error of the mean).Three questions dealing with force and motion are shown in Fig. 15, and the corresponding student responses are shown in Table IV.Success on item 9 required students to realize that energy is transferred during collisions (choices A and B), not force (choices C and D), and that in this case since both balls were moving after the collision that only some of the energy in the first ball was transferred to the second ball (choice A).The research literature suggests that it is not uncommon for students to think that force is transferred during an interaction [33].The pre-test results suggest that about 57% of students (response A or B) recognized that energy was transferred during the collision (either partially or fully), while about 43% thought that force was transferred (response C or D).At the end of the semester, as reflected on this particular assessment, about 74% of students recognized that energy was transferred during the collision, and only 26% still thought force was transferred.
Success on item 14 required students to know that the slope of the speed versus time graph (choice C), and not the value of the speed, was related to the strength of the force acting on the object.As the research literature suggests, students often think that force and speed are directly related, instead of force and the time rate of change of speed.Students who had this particular idea about speed being related to force would probably have chosen graph A in item 14, as did 45.6% of the respondents on the pre-test and 32.9% of respondents on the post-test.By the end of the semester, slightly more than half the students gave a response (choice C) consistent with thinking that the slope of the speed-time graph (the rate of change of speed with time) was directly related to the strength of the force acting on the object.Item 20 provides an example of an item that was quite difficult for students.Success on this item required students to realize that after the ball left the kicker's foot, the only force acting on it was the downward force of gravity (choice A).However, the research literature suggests that students often think that during an impulsive interaction (e.g., kicking the ball), force or impetus is transferred to the object and it continues to act on the object (or is carried by the object) after contact is lost [33].Based on the responses to this item, this type of thinking was common among LEPS students, during both the pre-test and post-test.On the pre-test, more then 77% of the students seemed to think that there was a force from the kick acting on the ball as it was rising after the kick (choices B or C), and only about 8% correctly thought that only the gravitational force was acting.At the end of the semester, as reflected on this assessment, only 15% (twice as many as on the pre-test) of the students seemed to be thinking correctly that only the gravitational force was acting as the ball was rising, whereas about 59% still thought a force from the kick was still acting on the ball after contact with the foot was lost.Thus, it seems that the LEPS curriculum did not successfully address students' commitment to this idea.It is also interesting to note that students might see both items 9 and 20 as dealing with the idea of transfer of force, the former item in the context of a collision between two balls moving horizontally, and the latter in the context of kicking a ball in the vertical direction.The fact that students performed much differently on the two items suggests that students' understanding is very much related to context.
Next we compare the scores for LEPS classes versus PSET classes for the entire PSQ.Average course scores were computed by averaging course scores, where course scores are the average of pre-or post-test scores for all matched students in a course.Thus, the average course pretest score differs slightly from the average pre-test score across all students, because average course scores give equal weight to courses regardless of enrollment.The average course PSQ pre-test score was 38:5 AE 0:8% in LEPS courses and 38:0 AE 1:1% in PSET courses (uncertainties are standard error of the mean).The average course PSQ post-test score was 56:7 AE 1:9% in LEPS courses and 56:4 AE 2:2% in PSET courses.For each course, an average course normalized gain [50] was calculated as the ratio of the actual average gain (%post À %pre) to the maximum possible average gain (100% À %pre) based on matched pre-and post-test scores.In the ten LEPS classes, the average normalized gain ranged between 16.7% and 41.0%, and in every case the gains were statistically significant (p < 0:05).In the 17 PSET classes, the average normalized gain ranged more widely between 4.6% and 53.2%; the gains were significant (p < 0:05) in all but two classes with two of the three lowest average gains.Averaging across courses, the normalized PSQ gain in LEPS courses was 29:6% AE 2:8%.Averaging across PSET courses, the normalized PSQ gain was 29:5% AE 3:5%.(See Fig. 16.)Both gains were statistically significant (p < 0:05), indicating that students' content knowledge had improved [51].There was no significant difference between the average course PSQ gains for LEPS and PSET classes, based on a two-tailed t test: tð25Þ ¼ 0:97, p ¼ 0:34.An f test indicated no difference in variance.
In summary, there seems to be no statistically significant difference between performance in LEPS and PSET classes with respect to gains in content knowledge as captured on the multiple-choice PSQ assessment.

VII. DISCUSSION
Both LEPS and PSET were based on a common set of design principles, and both curricula provide students with opportunities to develop an understanding of physical science concepts through discussions with peers based on scientific evidence (rather than based on claims by the professor or text).However, differences in the course formats required differences in how the design principles were instantiated in the two curricula.In comparison to the PSET materials, the LEPS materials are more granular (shorter, more focused lessons and homework), more constrained (questions with multiple-choice responses versus free response), more directed (less exploration and much more guiding by the instructor), and less hands-on (observing movies rather than conducting experiments).Given these differences, it is interesting to consider the impact on students' experiences in the two courses.Here, we discuss the development of students' conceptual understanding, their engagement with explanations, their experience with phenomena, and their interactions with peers.
Data from field tests suggests that with respect to specific physical science content, students in LEPS and PSET courses make similar gains as assessed by the PSQ (an entirely multiple-choice, content-based instrument).Although both curricula treat the same content, this result is surprising given the structural differences in LEPS and PSET courses.Specifically, LEPS' reduced class meeting time, higher student-to-instructor ratio, and greater relative emphasis on homework (including introducing new content) might all be expected to result in diminished outcomes.One interpretation of this result is that both course formats and curricula support content learning gains, and although PSET further supports development in practices of science such as constructing explanations, conducting experiments, and discussing science ideas, the extra class time devoted to these practices has limited effects on student performance on the content-focused PSQ.Below, we further compare the curricula with respect to the practices of science.Another possible effect (which was not tested in this study) relates to PSET and LEPS students' familiarity with the type of questions on the PSQ.All the PSET content tests during the semester were free response, where students needed to write short answers to questions, construct diagrams, and write or evaluate explanations.The PSET students did not have experience answering multiple-choice questions on course content.On the other hand, all the LEPS content tests were multiple choice, as were the homework quizzes.Thus, one might argue that the LEPS students had an advantage on the PSQ because they were familiar with the format of questions asked on the assessment, and this might counter to some extent the effects of class meeting time and student-to-instructor ratio, which would otherwise favor the PSET students.Finally, we note that the gains represent about 5 items on the PSQ and the average post-test score is 57%, which is less than most instructors would hope.Part of this may be attributed to the fact that taking the assessments was voluntary, that it was done online, and that performance did not contribute to a student's course grade.
As noted above, the PSQ does not address what one might consider the most significant difference between PSET and LEPS, namely, the opportunity for engagement in the practices of science.Therefore, differences between the curricula in student learning associated with these areas would not be detected by the PSQ.For instance, there are significant differences in the two curricula regarding the practice of writing and evaluating explanations.In PSET, students are asked to use physical science ideas to write explanations using the criteria of accuracy, completeness and logical flow.Students use the same criteria to evaluate others' explanations.Since the typical size of a PSET class is 16-30 students, this does not impose a severe grading load on instructors.Furthermore, when explanations are written or evaluated during class, the whole class has an opportunity to discuss them and address questions that arise.In LEPS the situation is quite different.Because of the high grading load involved in assessing written explanations or written evaluations of explanations in a large class, the LEPS curriculum does not ask students to write explanations, and only provides sample explanations for students to evaluate in a multiple-choice format based on the sole criterion of accuracy.when the evaluation takes place during class, especially when the explanation is not accurate, the instructor does engage students in a discussion about what is wrong and how to correct it.But the LEPS students do not consider whether the explanation is complete, nor whether it is clearly written and logically consistent, as the PSET students do in their evaluations.Thus, we might expect less development of LEPS students' scientific discourse skills.However, these skills are difficult to assess and this was beyond the scope of the present study.
Both LEPS and PSET prioritize the role of experimentation and evidence in science, such as using evidence to evaluate claims and establish patterns.Despite this common use of evidence, students' experience with phenomena is very different in the two classes.PSET students are extensively engaged in hands-on experimentation and developing evidence to support or refute their claims.Although almost all of the experiments they do are suggested by the curriculum, they can also vary factors, address ''what if'' type questions, and generally have the freedom (within time constraints) to explore other issues as the opportunity arises.In contrast, LEPS students experience physical science phenomena by watching movies of experiments; they have no opportunities for direct handson exploration of physical science phenomena [52].The difference between watching a movie of an experiment and performing an experiment with your own apparatus has numerous implications.Most obviously, direct personal experience and tactile sense is replaced by an indirect passive experience, namely, watching.When watching a movie, students' interactivity is limited because they cannot physically affect the process (although the instructor can replay, rewind, pause, or slow down the movie), and the questions they can explore are constrained to the ones addressed by the materials prepared by the curriculum developer.The use of movies also changes students' roles; in PSET, some students handle apparatus, some record observations, and others observe.In LEPS, all students now have the same task: watch the movie, then record and discuss their observations.Movies also have benefits: they allow students to experience phenomena in ways that are not possible in person, through the use of time lapse, split screen for side-by-side comparisons, and control of time (e.g., replay, pause, and slow motion).Movies also focus on the intended phenomena and eliminate distractions associated with errors, equipment failures, etc. (though these may also be viewed as important learning opportunities for students).As a logistical matter, showing a movie of an experiment requires far less class time than having students conduct the same experiment themselves.LEPS students were also provided with a Web site where they could replay any movies shown during class, in case they wanted to review the evidence provided by the movie.
One might claim that students develop a greater sense of ownership over the ideas they develop, and a better sense of the practice of science, when they directly experience phenomena, make and test predictions, and pose additional questions for exploration, compared to students who simply watch movies of someone else performing the experiments.We did not, however, gather information to test this claim in the present study.
Finally, there are clear differences between LEPS and PSET with regard to students' engagement in scientific discourse.The PSET curriculum and pedagogy is designed to promote a class norm where students are expected to share their ideas and respond to the ideas of others.Because of the small size of a typical PSET class and because students spend all their class time sitting with their group members, students generally engage in significant and substantive discussion with their peers [19].The instructor does not guide the class through the various questions within each activity; instead the students in each group are expected to guide themselves, and each group member has a responsibility to participate in that process.The PSET students also are expected to share with the whole class their answers to the questions posed in the Initial Ideas and Summarizing Questions sections, and they often do that by using presentation boards.They are also expected to ask questions of other students as appropriate during these presentations.In this sense, the PSET students have the opportunity to experience science as a sensemaking and collaborative social activity.
In LEPS, due to the larger class size, higher student-toinstructor ratio, and shorter class times, opportunities for students to engage in science discourse is less substantive than in PSET.The instructor guides the entire class together as they move through the questions in the lesson; thus, students have much less time to talk about their ideas with their peers sitting next to them.Because of time constraints, there are fewer opportunities for students to share their answers to questions with the rest of the class, and because of social inhibitions in a large class many are reluctant to do so in any case.Therefore, it may be more difficult for students to develop a class norm of sharing their ideas and responding to the ideas of others.Yet, the classroom excerpts presented in Sec.V demonstrate that substantive classroom discussion is possible in LEPS, both within the small group and in the whole class.Finally, LEPS is likely to be taught in classrooms where the physical layout of the room includes rows of desks that may be immovable and tiered.Such a configuration is not conducive to peer discussions and prevents the instructor from effectively monitoring and interacting with students during their conversations, something that is commonly done in the PSET classroom [53].
The preceding comparison highlights the shortcomings of LEPS, as compared to PSET, in regard to providing opportunities for students to engage in the practices of science.The LEPS-PSET comparison is the natural focus of this discussion given the historical development of LEPS from PSET.However, for a large-enrollment course PSET is not an option (as described in Sec.VA); in such cases a standard lecture-format course is a more natural comparison for LEPS.In this situation, LEPS compares favorably in terms of the incorporation of evidence-based reasoning, opportunities for student engagement, sense making, and peer discussions.

VIII. CONCLUSION
We have described how the LEPS curriculum was developed to address the need for an active learning physical science course that can be implemented in large-enrollment lecture-style settings.The same set of research-based design principles guided the development of both the PSET and LEPS curricula, but in the two curricula these principles are implemented differently according to the constraints imposed by the intended setting.This resulted in a pedagogical structure in which short instructor-guided lessons replaced lengthier small group activities, and the evidence used to support and test ideas is provided via movies, rather than hands-on investigations.However, LEPS still promotes student peer interaction as an important part of sense making, via the regular use of ''clicker'' questions.The examples of student dialog we have presented indicate that this format is capable of generating substantive student discussion that shows successful instantiation of the design principles.
Somewhat surprisingly, evaluation of content learning (using a pre-and post-test multiple-choice survey) indicates that there is no statistically significant difference between gains in content knowledge of students who complete a LEPS course and those who complete the PSET curriculum.However, there are other aspects of learning science that LEPS was not designed to address because of the constraints imposed by the course setting and that we therefore did not attempt to evaluate in our study.These aspects include important scientific practices like the writing and evaluation of scientific explanations, the design and implementation of investigations involving physical engagement with scientific equipment, and extensive opportunities for engagement in scientific discourse [54].
Our goal in developing LEPS was not to promote large classes instead of smaller ones.Rather, we recognize the resource constraints faced by many universities (for example, the situation described in [7]) and accept largeenrollment classes as unavoidable.Given this reality, we believe LEPS is a valuable alternative to a lecture-based large-enrollment course.
Finally, we note that LEPS offers the opportunity to introduce college faculty to research-based curricula and pedagogy through our ''educative instructional materials'' [55] in a way that is not too far removed from traditional instruction and without requiring that they rewrite their curriculum themselves.It thereby gives faculty an opportunity to explore and practice using interactive pedagogy that allows for student construction of ideas, possibly supporting faculty transition to and their advocacy of smaller, more hands-on inquiry-based formats [56].

FIG. 3 (
FIG. 3 (color online).Energy diagrams before, during, and after the cart is pushed.Students are asked to sketch the corresponding force diagrams.

FIG. 5 .
FIG. 5. Image from PSET C2A1 homework.Skateboarder moving to the right is subject to a short push.

1 .
Instructor: Do you or your group have an answer [to the question Does the force of the hand continue acting on the cart after it has lost contact?How do you know?] 2. Brad: We're conflicted.3. Instructor: You're conflicted.What are the two arguments?(. ..)

FIG. 8 (FIG. 9 .
FIG.8(color online).Discussion between three fictional students about the relation between motion and force after the hand stops pushing the cart.

27 .
Maia: Energy.[Group silent for 7 seconds.]28.Laurie: But it's not force.The force is actually pushing.[Moving hand forward, as if pushing something.]29.Brad: Does force, is force actually transferred to the receiver?30.Mary: It may not be.31.Maia: That's what it says, from the source to the receiver [moving left hand towards right hand] 32.Mary: Maybe just energy.Right?33.Maia: I think it will be just energy.Because it doesn't, I mean, we just saw [pointing towards screen at front of room], yeah, it's just like when I'm touching it [moving hand forward], that's it.34.Laurie: Well, I don't know because we actually have a transfer of energy.35.Mary: That's true, that's true.
44. Mary: It's C because when we gave it force the last time [she looks at graphs from U2L02 CQ 2-2] it's [???].So, if we're going to do force [pushes hand forward] the whole time, then it's going [slants hand upwards] to go like C. 45.Maia: Yeah, I think it's C too.Because that one [pointing towards graph B] is saying that it speeds up and then stopped [speeding up].It slowed down and it's constant in its speed.46.Mary: I don't think it stays the same speed if you're pushing it.It would go fast.47.Maia: Well, it could be B. 48.Brad: Yeah, I don't know.49.Maia: Because the initial-50.Brad: It could only go so fast.

FIG. 15 .
FIG. 15.Three questions from the Physical Science questionnaire.

FIG. 16 .
FIG. 16.Average normalized course gain on PSQ.Error bars are standard error of the mean.Letters in the course identifiers indicate different courses taught by the same instructor.

TABLE II .
Design principles as implemented in PSET and LEPS.

TABLE I .
Features of PSET and LEPS.
prior knowledge (see Sec. V for examples).Student responses are collected, and the results

TABLE IV .
Pre-and postinstruction responses of LEPS students on specific PSQ items.Numbers are percentages of all 425 LEPS students with matched preinstruction and postinstruction responses.Correct responses are in bold.