Replicating and Understanding Successful Innovations : Implementing Tutorials in Introductory Physics

We report a detailed study of the implementation of Tutorials in Introductory Physics at a large-scale research institution. Based on two successive semesters of evaluation, we observe studentsimproved conceptual mastery (FMCE median normalized gain = 0.77, N=336), albeit with some student discontent. We replicate the results of original studies of Tutorial e¤ectiveness and document how and why these results occur. Additionally, using the Colorado Learning Attitudes about Science Survey we measure the support of studentsexpert-like beliefs about learning physics in our environment. We examine this implementation from a viewpoint that emphasizes varying contextual levels of this implementation, from students engagement in individual tasks, to the situations in which these tasks are embedded, to the broader classroom, departmental and educational structures. We document both obvious and subtle features that help insure the successful implementation of these reforms.

A frequently achieved informal goal of Tutorials is that the undergraduate students should perform as well on post-test evaluations as the graduate physics TA's performed on pretest evaluations before going through the Tutorial materials.We present a case study of a secondary implementation of Tutorials at the University of Colorado.Our study begins by documenting the logistics and e¤ectiveness of this implementation, and then examines our implementation from a broader contextual lens, to better understand why and how these reforms work, and how to assess and improve their impact.

II. CU IMPLEMENTATION
A. Background and course structure Tutorials were implemented …rst in the Fall of 2003 in the introductory Calculus-based Physics course (Physics I) at the University of Colorado, Boulder (CU).CU is a large public research-extensive (primarily research-oriented) University.[25] The student population in the Physics sequence described here is roughly 50% engineers and 20% natural science majors (20% are undeclared, and the remaining 10% are a broad mix of social science, arts, architecture, business, etc).The class is 75% male, and over half are freshmen.Three lectures per week are taught by the lead instructor (SJP) in large halls which seat approximately 300 students.Multiple choice "Concept Tests"are asked frequently in lectures (3-5 times/class) following Peer Instruction [12] methods, using a student personal response system.[26]Each week students attend an additional 50 minute recitation section sta¤ed by a physics graduate student teaching assistant (TA).All recitations in this physics course are scheduled on a single weekday, scattered among small classrooms with traditional seating for 28-32 students.This course serves 500-600 students each semester; the lectures are repeated twice, serving 250-300 students in the lecture hall at a time.There is no lab component associated with the …rst semester course; however students usually enroll in an independent laboratory course concurrent with their second semester of introductory physics.The content and syllabus of this course are largely de…ned by departmental tradition, and follow much of the …rst 15 chapters of Halliday, Resnick and Walker [27] or a similar text during a 15 week semester.
based labs (MBLs), students demonstrated signi…cantly greater improvement on the Force Concept Inventory than those students who did not use Maryland tutorials.
There are three evening midterm exams and a …nal exam, all equally weighted, which together account for 60% of the grade.In addition, conventional end-of-chapter problems are assigned and graded by a computerized system, LON-CAPA.[28]One of these homework questions is written up for hand grading by the TA's every week.Students are given the opportunity to work on homework collectively in the "help-room,"a large TA-sta¤ed room for all physics students to meet in and work on homework.

B. Structure of Tutorials in Introductory Physics
Implementing Tutorials involved the following components.One instructor spent two weeks visiting the University of Washington, participating in TA training classes and Tutorials, and solicited advice from the UW Physics Education Group.Based on that advice, instructors (including the authors) constructed a small space with tables well suited to group-work for half of the sections; the other half of the sections were still held in regular classrooms, with movable desks brought into small circles of four each.In Spring 2004, the dedicated Tutorial workspace was doubled, eliminating the need for the less desirable conventional classroom space.In order to adequately sta¤ the Tutorials, external funding 3 was secured to support an undergraduate Learning Assistant (LA) to couple with the departmentally funded graduate TA in each section, reaching a teacher: student ratio of 1:14 that approaches the suggested ratio of 1:10.[3] A weekly online Tutorial pretest system 4 was set up.This pre-test evaluation was required of all students every week 5 and functioned as an "elicitation" mechanism [14] [15] for the students, while providing essential data on student preconceptions.Mandatory 75 minute preparatory meetings for the TA's and LA's were held each week to discuss the philosophy and practice of Tutorials, including explicit guidance in the use of Socratic questioning as the primary tool for teacher-student interaction in the Tutorials.Most of the preparatory sessions, however, were spent …rst examining student prior-conceptions as represented on the pretests, and then working through the upcoming Tutorial.The undergraduate LA's participated in an additional separate weekly 1-credit hour science education seminar through the School of Education. 6ach week during the 50 minute Tutorial recitation sections, the Physics I students gathered in groups of four at tables, received brief instructions and graded work from the TA's and then worked their way through Tutorial exercises.Students were provided large scratch paper on the tables, appropriate equipment, and were strongly encouraged (by course instructor, TA's and LA's) to work through the Tutorials collaboratively.Following the Tutorial methodology, work during Tutorials was never turned in and graded.Surveys and anecdotal evidence indicate that Tutorials were often not completed in the 50 minutes allotted.Students were assigned 2 to 5 pages of homework each week from the Tutorial[9] workbooks 7   which were graded by the TA's and constituted 15% of the overall course grade.Attendance at Tutorial accounted for 4.5% of the …nal course grade.Twelve Tutorials were run in each semester of implementation, largely following the published sequence. 8Every midterm exam explicitly included a Tutorial-related "long answer question," which comprised 25% of the midterm exam.The course syllabus and web page included basic explanations of the Tutorial logistics, philosophy and purpose.The implementation of Tutorials was nearly identical for each of the two semesters included in this study.The only signi…cant change was a doubling of physical workspace, so that no sections in the second term were in conventional classrooms with individual desks.
The details described in the paragraphs above represent the basic logistics for implementing Tutorials within the broader institutional and departmental frameworks, training the instructor, construction of a physical space for the Tutorials, providing a su¢ cient teacher to student ratio, training and preparing the TA's, assigning credits, and implicitly establishing a new set of expectations for students.In the second section of this paper we will use a contextual constructivist perspective [13] to examine more closely the role played by these aspects, and why and how Tutorials …t into the course and impact student learning.and Practice o¤ered by the School of Education. 7These build on ideas and concepts raised in the in-recitation Tutorial activity. 8Speci…cally, weekly Tutorials covered: acceleration in 1-dimension, motion in 2-dimensions, forces, Newton's II and III laws, tension, work and the work-energy theorem, relative motion, a modi…ed conservation of momentum in 1-d (modi…ed version provided by UW PEG), rotational motion, dynamics of rigid bodies, simple harmonic motion (modi…ed version provided by UW PEG), superposition and re ‡ection of pulses.
First, we present data from the …rst two semesters of implementation (Fall 2003 andSpring 2004) to characterize the impact of this course structure on student learning, and attitudes and beliefs about learning.

III. DATA AND RESULTS
Data were collected in two consecutive terms of the …rst semester of the introductory physics course and included measures of student conceptual gains, attitudes and beliefs about physics and learning physics, and a¤ective response to the Tutorials and course.

A. Student conceptual learning
Conceptual learning in this course was extensively documented at a number of levels.Students completed weekly online pretests that included multiple choice and freeresponse questions.Multiple choice "Concept Tests" in lectures allowed the gathering of data on understanding of speci…c topics covered in Tutorials.Questions were issued both in silent/individual and talking/collaborative formats, and were collected with a student personal response system.[26]Longer term assessment of student conceptual mastery included analysis of student performance on Tutorial homeworks and performance on exams.
Well-documented questions from the University of Washington were used for most of the long answer exam questions. 9Additionally, multiple-choice questions designed to assess the impact of Tutorials were added to each exam.Nationally-normed conceptual surveys of student mastery were issued both pre and post instruction during the …rst two semesters of running Tutorials: the Force Concept Inventory (FCI) [29] in Fall 2003 and the Force and Motion Concept Evaluation (FMCE) [30] in Spring 2004.These were administered in the …rst and last recitation sessions of the term.
Student learning gains as measured by the FCI and FMCE tests fell into the "high gain" category described by Hake.[31] Student performance on the FCI (given Fall 2003) shifted from an average pre-test score of 53% (median 50%) to an average post-test of 81% (median of 83%).The normalized gain of the averages, g = (post-pre)/(100-pre), was 0.6.Matching individual students pre to post gives a median normalized gain of 0.67.The FMCE (given Spring 2004) shifted from a pre-test of 29% (median = 23%) to a post-test of 74% (median = 85%).The normalized gain of the averages g=.63, the median gain matching students pre to post is g = 0.77. Figure 1 shows the distribution of student performance for the Spring 2004 pre and post FMCE.The most common score on the post-test was 100%.
Because these conceptual assessments (the FCI and FMCE) examine only a very limited fraction of the content of our introductory physics course, we assessed student mastery of material covered on each Tutorial (and in each week of class) with tutorial pretests and follow-up exam questions to provide more detailed measure of the impact of Tutorials.Table I shows some representative student scores on measure of student mastery (both in the short term and longer term).
The data presented in of the impact of the twelve Tutorials.The "pre" column represents an average score for online multiple-choice questions given before the material was covered in Tutorial.The data are averaged over several questions for each topic, and across the two implementations.
Statistical errors in these data are typically a few percent, and semester to semester variation is also typically a few percent.Following the tradition of UW publications [20][21] [22][23] [24] we round all results to the nearest 5%.Participation in these pretests ranged from 75-90%  semesters, including …nals) where the relevant questions appeared.These exam questions came weeks to months after the Tutorial. 10(Of course, there are many factors in ‡uencing student performance on these exams, including the Tutorial and other homework problems, lecture, and the class discussion surrounding the question, all of which occur after the "Post (click)"question.)

B. Replication
A primary concern when appropriating tested curricular reforms is the …delity of implementation: how successful are we at replicating the original success of Tutorials?One mechanism by which we may answer this is a direct comparison of results with the original studies by University of Washington (UW).On common long answer questions issued on the midterm and …nal exams, we may compare answers.These results are shown in Table II 2. Students who performed well on the Tutorials (homework) did well in the course overall.Such data suggest that the Tutorials are well-integrated into the rest of the course, an idea that we return to below.

D. Students'Attitudes and Beliefs
In addition to the traditional content within any course, there are extensive sets of attitudes and beliefs about science we teach to our students.The way we conduct our classes send messages about how, why, and by whom science is learned.Such meta-messages have been referred to as the "hidden curriculum."[3] As o¤ered in traditionally taught courses, some of the hidden curriculum is bene…cial (e.g., the message that science is a coherent representation of the world) while other aspects are detrimental (e.g., the notion that women cannot be strong scientists).While decades of physics education research have reformed classroom practices to improve student mastery of conceptual domains, [2][3] these same class environments are often found not to improve student attitudes and beliefs (ABs).In fact, students are usually found to regress from more expert-like beliefs to more novice beliefs over the course of a semester.[3][33][34] [35] In our …ndings and those of prior researchers, it is notable that such retreat in favorable student ABs are seen even for courses where reform pedagogy (including Tutorials) is used and improved conceptual gains are observed.Although not intrinsically manifest in the curricular reforms we implement, there is potential for the development of this hidden curriculum, and it is this issue that we address next.To study the state and evolution of student beliefs in Physics I, we use a newly developed and validated instrument, the Colorado Learning About Science Survey (CLASS), [36] which is based in part on prior instruments used to evaluate student expectations.[33][37] [38] The overall result of the CLASS survey for each of the semesters of the reformed course showed no signi…cant change in student beliefs from beginning to end of term -an improvement over the regress typically seen in introductory courses as measured by this and other instruments.[3][33][34] [35] Because other implementations of Tutorials and tutorial-style approaches have documented the regress of student AB's, [3][33] it is likely that the lack of retreat we measure is associated with a more complex interaction of classroom instruction, and explicit attention to the hidden curriculum.In the second half of this paper, we will examine the role played by, and interactions among the various course elements, student beliefs, and practices surrounding the course.In Figure 3, we show one example of the correlation between conceptual learning and the measured student attitudes about the role of concepts in learning physics.The histogram is binned based on normalized gain on the FMCE (low, moderate, high, very high) and within each bin, the pre and post scores (…rst and last weeks of the term) on a subset of CLASS questions that focus on the "Conceptual Understanding"category 12 are shown.[36] There are two interesting trends in the data.
First, students with higher learning gains (as measured on the FMCE) tend to score more favorably (more "expert-like") on their attitudes and beliefs about the role of conceptual understanding in learning.Second, students in the lowest gain group regressed to more novice-like beliefs over time, while students in the higher gain groups improved.In the top bin (normalized learning gains on FMCE above 75%), the e¤ect size of the pre-post CLASS shift is +.2, a small e¤ect.(For all students in the class, the e¤ect size of the pre-post shift is negligible.)The Pearson's correlation coe¢ cient of overall CLASS score (percent favorable) with …nal score on the FMCE is 0.24 (pre) and 0.34 (post), with nearly identical results using …nal course grades rather than FMCE scores.These correlation coe¢ cients are statistically signi…cant, indicating that attitudes and beliefs about learning may themselves in ‡uence, or be in ‡uenced by, the class structure and learning experience.Re-examining these data by binning the overall score on the CLASS pre-test in terms of percent favorable response (0 = "novice-like", 100 = "expert-like") provides another perspective.[39] In examining the relation between Tutorial Homework scores and students'CLASS beliefs (shown in Figure 4), we observe similar trends as seen in Figure 3, which compared CLASS performance as a function of FMCE gains.The lowest performers on Tutorial homework make no gains (indeed, regress) in expert-like beliefs over the term, while the higher performers on homework improve somewhat in terms of expert-like beliefs.The di¤erence in post-CLASS Concept score between the bottom and top quartile homework performers is large (>.5 standard deviations), and statistically signi…cant (p<.001; 2-tailed t-test).

E. Student A¤ect and Evaluation
Students'a¤ective responses (whether they enjoy the Tutorials or not) are worth noting in a course that is their …rst experience in college science.Related to this sense of enjoyment is whether or not students believe the experience to be useful (whether or not, in fact, it is Questions were asked on a 5 point Likert-scale; "agree" in table corresponds both "4" and "5".
Totals do not sum to 100 because neutral "3" is a valid response.
however, is the sizable fraction of the class that does not enjoy the Tutorial experience.individuals, 14 we argue that no less signi…cant in the learning process are the environmental structures that constitute the educational experience.The local context, surrounding norms, and particular tools, whether a given Tutorial lesson or 2-d velocity arrows and a shared piece of paper used in that lesson, serve to mediate students'actions, and thereby participate in the intellectual processes we seek to promote.
A useful tool for identifying a particular level or scope of activity that shapes the success of Tutorials is the notion of frames of context.[13] We might examine any number of frames: the tightest framing -a student engaged in particular task, such as a student predicting the direction of an acceleration vector; a slightly broader frame -the situation surrounding a particular task, such as a given Tutorial session on two dimensional motion and acceleration; the local culture of the physics course or Tutorial sessions, which is constituted by the collection of many such situations, the broader departmental and university norms, and so forth.Figure 5 illustrates the nested nature of these frames.
While these frames are useful for locating the level of detail we wish to examine, we also recognize that these levels in ‡uence one another.There are dynamic relations among con-stituent elements of the Tutorial system, the associated physics course, and broader departmental, University, and environmental cultures.A Tutorial session is not simply surrounding individual tasks, but constituted by them, as well as shaping the tasks themselves. 15

B. Implementing Tutorials: A Frames of Context perspective
In analyzing the various frames of context necessary for implementing Tutorials, we highlight some of their key features: [13] -Frames are nested and generally hierarchical (and outside more heavily in ‡uences inside than the reverse).
-They are dynamic, representing evolving relations among constituent elements.
-Their boundaries are somewhat ‡uid: frames interact not only at their interfaces, but also via their constituent elements.
-People participate simultaneously in many frames of context.
Below, the implementation of Tutorials at Colorado is examined at each of four nested and increasingly broad frames: task formations, situations, course, and departmental frames.

Task formation
Because of the well-documented practices of the Tutorials, it is relatively straightforward to replicate particular task formations.The particular tasks delineated by the Tutorial workbooks and accompanying homework are designed simultaneously to be simple to implement and to be directed at improving students'mastery of foundational concepts in the 15 Furthermore, it is worthy of note that students participating in Tutorials participate in many other contextual frames.Participants, students, TA's, LA's and the instructor are all members of various cultural groups, family structures, and social circles, each with it own sets of norms, beliefs, situations and particular tasks.One of our goals in the course is to recognize and value these historically developed and productive sets of norms and practices.[43] While we could analyze any number of sets of nested frames of context, in order to analyze critical elements of the Tutorials, we …nd it useful to consider a particular perspective common to the students, instructor and broader institution -notably the academic frame.
domain.Because they are among the most studied educational reforms in college physics, the particular task formations have been demonstrated to be very e¤ective at eliciting students existing ideas about the content domain, and at stepping students through a series of confronting or bridging activities to help the students build more expert-like understanding of the concepts.[15][44] The implementation of similar tasks at CU has led to similar results on assessments of students mastery of particular concepts, as shown in Table II.
These task formations, however, depend upon a variety of other constraints and interactions.Most notably, the students themselves are critical to the creation of e¤ective task formations.While students' prior understanding of these domains have been thoroughly researched, and thereby included in the materials used in these tasks, 16 these studies have been conducted with a particular subset of U.S. students.Tutorials are designed for a calculus-based course and studies of student understanding have been conducted mainly at the University of Washington.Students who come from di¤erent backgrounds/contextual frames (cultural, linguistic, and gendered frames) bring di¤erent tool sets (norms, practices, beliefs and understanding) for engaging in these tasks.In some cases working with di¤erent student populations, it has been suggested that these Tutorials are not as e¤ective at promoting conceptual understanding as demonstrated by Washington or our current work.[8][34] [46] One potential explanation for such …ndings is that the students themselves engage in, shape and interpret these tasks in very di¤erent ways.Fortunately for the present study, the student demographic for students enrolled in calculus-based physics at University of Colorado is rather similar to the student population at the University of Washington.In particular, we observe similar pre-test scores on common conceptual questions.Even so, in weekly observational notes collected by the Learning Assistants and through analysis of course surveys completed by students at the end of term, we have found many students consider the Tutorials an activity to be discharged rather than one that is to be learned from.Furthermore, as shown in Table III, students report not enjoying the Tutorials.Such a view of these tasks likely comes from students'broader perspectives, which are shaped by 16 That is, not only do the material artifacts of the Tutorial worksheets carry with them a particular representation of physics (well vetted by science and history) and pedagogy, but carry with them an understanding of students (what they know and how they know).For example, the particular tasks are designed to elicit common conceptions of the world that students have.A second example is that there are few references to real-world or personal activities.This approach promotes a more abstracted and disconnected way of knowing, which addresses di¤erent populations of students di¤erently.[45] the larger frames of context that include norms developed in other courses and the wider context of the education itself.Not surprisingly, Tutorial tasks are strongly bound to these broader contexts, and most notably to situations -the conditions in which a particular task might take place.

Heuristics of implementing Tutorials at the level of Tasks
The particular Tutorial tasks are well documented and easily implemented (both in terms of infrastructure and practice) and hence exporting Tutorials at this level is straight-forward.
Nonetheless, consistent with prior researchers, we claim the particular tasks are not su¢ cient to promote student learning; how these tasks are situated (whether or not students work alone for example), and what goals are enacted in these tasks (conceptual learning versus development of particular attitudes and beliefs) [3][36] are both critical to successful implementation.Insuring that these tasks are productive depends upon the broader situations in which they are embedded.

Level of situations
The environmental surrounding, the ensemble of task formations over time and the resources present for conducting a particular Tutorial task, constitutes the surrounding situation.Critical elements for implementing productive situations include: the design of a functional space, resources such as tables, papers (used as joint mediating tools), modest equipment, appropriate tests and feedback, 17 and adequately trained sta¤.These features in collection allow for the students to productively engage in the Tutorial tasks.However, at the same time, these elements themselves are critically dependent on the broader framings.The department has to commit to providing appropriate space and the added sta¢ ng requirements: roughly twice the usual number of assigned TA's is suggested.
From a pedagogical perspective, two forms of interaction are essential in these situations: student-student interaction and student-TA interaction.With the appropriate shift of re- 17 It is at the level of situation that we might understand why we might only compare UW and CU student conceptual mastery to 5%.While particular questions might be identical, the situation in which these questions are asked (where these questions are asked (exam, or in class), the question order (which questions precede and follow), and answer order within the questions) all a¤ect student performance.[3] While the particular tasks may be similar (answer a particular multiple choice question), the situations in which these occur vary dramatically enough to recontextualize the tasks (and hence student performance).
sources, Colorado has doubled the number of rooms allocated to the recitation sections so that groups of students might be able to work at tables of four around Tutorial materials and discuss the questions posted in the Tutorial tasks.Students develop an expectation of talking as a form of educational practice (as opposed to passively sitting listening to a TA working through homework problems at a chalk board in front of the room). 18It is worth noting that the tables do not force the students to interact, but do support the tasks that require interaction.In the …rst semester, when some of the students worked at individual desks, students were required to re-arrange the environment physically (working against the original room design) to engage in these tasks by bunching desks together.Had the desks been bolted down, the environment may well have precluded the forms of interaction we sought to support.Situations where students explain their reasoning with their peers, debate approaches and collectively agree upon answers leads to enhanced student mastery of concepts (demonstrated above) and more sophisticated expectations about learning (as evidenced by modest increases, or no decreases in their beliefs).Notably, students also come to recognize the educational value of group work, and in particular such sentiments are captured in students'evaluation of the Tutorials (Table III).
The student-TA interaction, which we advocate as essential in the productive implementation of Tutorials, relies upon the existence of enough sta¤, and upon their adequate training and support.By having two leaders in a class of 24 students, the six groups of four students may each interact frequently enough with experts to insure that their student-student interactions in given tasks are productive and support the broader course culture and norms.The Tutorials build such interactions into the materials (such as the tasks of check-outs, where students check their answers with TA's before proceeding further in the Tutorial) to support these productive learning-situations.Meanwhile, adapting practices at the University of Washington, TA's and LA's must learn how to support and build these learning situations.
The local Tutorial leaders participate in relevant and intersecting frames of context in which they develop expectations and practices that support the Tutorials.It is during a training session each week run by the course instructor that TA's and LA's develop the pedagogical content knowledge [47] appropriate for conducting Tutorials.Furthermore, approximately 18 Such expectations arise from and shape the surrounding culture of the Tutorials and physics course.
Notably, while productive for situations conducive of educational tasks, such expectations sit at odds with other educational cultures in which students have participated.
half the Tutorial leads participate in an additional course on education/learning in physics or the sciences. 19During these two forms of interaction, the Tutorial leads develop the norms, expectations and an understanding of the practices in Tutorials.One of the critical norms of Tutorials, emphasizing student responsibility and agency, is supported by TA's and LA's not disclosing answers during the Tutorial sessions.Students are held responsible for understanding and explaining the content (on graded Tutorial homeworks and targeted questions on exams).In the Tutorial situations TA's and LA's emphasize these norms by supporting student reasoning and understanding more than verifying correct answers.As a result, students come to develop an expectation of learning and their roles in this process, as discussed in II.D above.These participants and forms of interaction are critical in shaping the surrounding culture of Tutorials.

Heuristics of implementing Tutorials at the level of Situations
How students engage in Tutorial tasks (whether they simply discharge their duties of …lling in answers or try to make sense of the concepts at hand) are signi…cantly shaped by students'interaction with each other and with the TA's.Students support each other's understanding by making reasoning explicit; TA's support student engagement in tasks by not disclosing answers or grading student work in Tutorial section (but do grade homeworks and exams).Other elements of the situation enable these productive interactions: locating students in the same room, providing appropriate desks for students to work at, eliciting student conceptions in advance of Tutorials, and understanding that these ideas will be revisited on exams and homework.Each component of these situations is shaped by TA, student, and instructor understanding of the norms of this environment, which are (hopefully) based on a model of how students learn.

Course culture
Extending the …eld of view allows for examination of collections of many situations over time that are localized to a particular group of individuals.One way of describing this frame of context is by thinking of small-group cultures, or what Fine calls idiocultures.[35] In many senses, developing a productive course idioculture is the goal of implement-ing Tutorials at CU -we seek to create an environment which is the productive collection of particular norms, uses of tools (intellectual and material) and their evolution over time.Researchers [49] have also referred to such as "habits of mind." 20A wide variety of norms/practices are supported by the Tutorials and surrounding course: -Interactive, student-centered engagement: the Tutorial sessions, Peer Instruction activities, and homework help sessions, where the students are responsible for identifying particular issues and answering questions they raise (often collectively).
-Students talking: related to a student-centered environment; however, Tutorials and other elements (particularly the Peer Instruction and Help Room) shift students'beliefs about who is to be speaking during o¢ cially sanctioned class time.
-Not knowing, but discovering is valued over answers: By not providing o¢ cial solution sets, by not telling students of correct answers in section, but allowing students to discover and invent answers, the Tutorials emphasize a more sophisticated epistemological frame.[50] This normative value is supported by not grading Tutorials, and notably runs counter to much of traditional instruction and formal grading systems.
-The nature of science and scienti…c inquiry is valued explicitly along with particular content.Students are encouraged to focus on how we know what we know as much as they focus on what we know.
These norms are most clearly established by the course instructor and Tutorial leads.
These norms are established through the practices, tasks and situations described above and are embodied in many of the tools used in these practices.Sometimes these norms are stated overtly -as in the syllabus or in the grading rubrics -and sometimes they are left implicit in the particular tasks and situations in which students are placed.For example, the homework help room does not provide instructions on use, but engenders certain practices and norms such as student talk.While often directed by authority …gures, these norms must be collectively established by students as well.If students do not accept or participate in these normative practices, the local classroom culture does not progress, no matter how insistent the instructor.When these norms sit at odds with existing norms and beliefs of students (developed in other educational environments) these idiocultures clash.For example, as observed by LA's and documented in surveys of student attitudes, to students who are committed to performing in lieu of understanding because they have been rewarded for such beliefs, the Tutorials can seem a blur of busy work and students resist both the added work and nuanced approach of Tutorials.They also report not enjoying Tutorials.Carefully engineered educational environments, however, may foresee this and di¤use the distinction between performing and understanding by placing appropriate questions/assessments in the course (such as Tutorial questions on the exams).As a result of such attention, we observe signi…cant correlation between students' …nal grades (excluding their homework scores) and their performance on Tutorial homework (Figure 2).Nonetheless, such extrinsic focus remains a patch on the large problem of clashing ideology that is established from broader contextual frames such as the educational system's commitment to grading.We hypothesize that such a rift in expectations about learning and the practice of Tutorials may be one of the causes of students'lack of enjoyment of Tutorials.While the experience is evaluated as a positive learning environment (and is borne out by evaluation of students' conceptual mastery), such learning experiences do not have to be enjoyable (like a potent medicine).Student a¤ect is the subject of current and future research.
Without particular attention to these rifts in norms, students'expectations and beliefs about learning physics will tend to degrade.That is, students will come away from the course believing that physics has less to do with their lives outside the classroom than when they entered the course.[3][39] By interweaving the Tutorials with the rest of the course (referencing Tutorial activities in lecture, placing Tutorial-style questions on exams, and linking these with homework), students at CU were encouraged to consider the coherent nature both of the course and of physics.Students hopefully recognize that Tutorials do not sit as isolated or ancillary activity but as an integral activity to the course and to learning (or at least performing).The increase in student performance from the near term ("Post (click)"questions, Table I) to longer-term evaluation (…nal performance, see footnote 11) corroborate the integrated nature of Tutorials with the course.Furthermore, those students who have a more expert-like understanding of conceptual framing and coherence of physics do better in the course, and tend to improve their understanding along these dimensions (see Figures 3 and 4).At the same time, the content of the Tutorials references activities previously covered, suggesting that physics is a coherent domain rather than a disconnected collection of trivia.While the Tutorials themselves do not explicitly reference either the students'lives or particular ties between physics content and the world outside the classroom, both the instructor and the teaching and learning assistants regularly tied the Tutorial concepts to real-world phenomena associated with students' lives.Finally, while subtle, the signi…cant presence and approach to student-TA interaction, e.g.not providing answers or solutions to Tutorials, encouraged students to engage in and take charge of their learning, as well as introducing students to the nature of scienti…c investigation (supporting intellectual discovery over memorization).
Given these norms, one of the main challenges of de…ning and producing replication remains: How does one adopt and adapt a culture of norms and beliefs?The materials and even structural implementation of Tutorials are relatively easy to replicate; however the norms surrounding these cannot simply be stated and exported -rather, they must be grown and adopted.That is, while it might be useful to explicitly state appropriate norms (as we attempt here), faculty must adopt these and appropriate them to make sense at their institutions.Just as students may learn in a lecture environment where information is stated, the learning process is one of construction of ideas and understanding, so too do faculty need to build an understanding of supportive norms of Tutorials at their institution even if the norms are overtly stated.As a result, UW emphasizes the importance of visits to Washington for workshops or training sessions at national conferences.Just as LA's develop an ideological frame and norms by participating as they do, so too might faculty.Because the lead instructor had visited the University of Washington and because other researchers in the CU community had spent time with the Tutorial authors and studied their approach, adapting an appropriate ideological frame was more likely in the CU implementation.Such an adoption of ideology critically depends upon the broader framing of faculty worlds, of course.[51] Heuristics of implementing Tutorials at the level of course culture The norms and expectations established by and for the participants in the Tutorials are among the most critical and subtle features of implementing the reforms.The situational interactions described in the previous section are guided by this localized culture.The Tutorials are an integrated set of activities, including pretests, Tutorial tasks, TA training, homework sets and exams, [9] all of which establish a set of beliefs about learning physics in this course.It is through the norms established in this system of activities accompanied by other explicit e¤orts in lecture, that students and TA's learn critical features of productively engaging in the Tutorials.For instance, ideas are constructed by the students, and answers are reasoned and justi…ed as opposed to corrected.It is also possible in these environments, though by no means inevitable, that productive beliefs about learning physics are established.For example, Tutorials and physics are not disconnected from the rest of class or world at-large.Because the norms and expectations evolve over time, adopting a model of enculturation [48] of participants (from instructor, to TA's, to students) appears to be e¤ective for establishing a supportive culture.
4. The broader lens: department, university, and social frame While various broad frames shape the Tutorial experience, including department, university, and the larger national educational system, these are collapsed for the sake of brevity.
A number of competing interests and norms arise in these broader educational frames: cost of implementation, teaching and learning culture and what each of these means.Twice the number of TA's is required to run the Tutorials as is assigned by the department.While addressed in di¤erent ways at UW and CU, increased teacher:student ratio is required for appropriate Tutorial experiences.At CU, through external funding (an NSF grant) and a novel course on teaching and learning physics, the Tutorials are adequately sta¤ed and prepared.
Fundamentally, however, the existence and training of these Tutorial leaders (alternative to the traditional TA models) requires support from the departmental or broader institutional levels to remain sustainable.Once/if external funding is withdrawn (yet a broader framing on local practice), alternative models need be in place.Two such alternatives are: commitments from the department for additional funding per student, or expanding the class on teaching and learning physics.[52] Each provides the added lead-support required, one through funding, and the other through a service-learning educational experience.[53] Meanwhile, other resources, equipment, faculty time, and space require some funding and institutional commitments; however, these are more closely aligned with the levels of commitment to education that are currently provided.

V. CONCLUDING REMARKS
Clearly, it is possible to replicate the success of Tutorials demonstrated by the curricular authors.Notably, however, such replication is more than a matter of repeating particular activities or worksheets.It is a matter of appropriating activities and surrounding structures: the particular tasks, surrounding situations in which tasks are embedded, and the larger norms, practices and infrastructure required to support the Tutorial situations.Because these activities must …t into new environments (that is, the boundaries at each of these level of detail must match boundary conditions), there is a mix of adoption, adaptation, and reinvention.
In addition to documenting the processes of implementing Tutorials, we apply a new framework for understanding this educational practice in context, referred to as 'contextual constructivism'.This framework speci…cally attends to the social and contextual nature of educational environments and student learning.The idea of frames of context, provides a mechanism by which we might understand how nested features of these environments (from departmental norms to speci…c learning tasks) shape and are shaped by the participants.
We exploit this framing to identify particular levels of context that are relevant -e.g. the need for tables to support certain tasks, such as student discussion of a given conceptand how these levels are co-constituting -e.g.student talk supports a norm of Tutorials: that students are the agents of their own learning and that reasoning is as important as …nal answers.Furthermore, we identify how particular frames of context support or sit at odds with pre-existing frames, such as those that encourage students to focus on answers more than reasoning.In short, these frames of context provide an organizing mechanism for thinking about which features of the system are critical for success, as measured at whichever the level of interest -from that of students (are they learning?)to that of the broader institution (is such a model sustainable?).In applying this model we simultaneously emphasize the critical role of context in student learning and provide a mechanism to build and understand environments supportive of learning.
While we have demonstrated some success at improving students' conceptual mastery and developing productive attitudes and beliefs about physics as a result of implementing Tutorials, several challenges remain.First, we seek to further develop our implementation of these activities so that students' a¤ective responses are more re ‡ective of the joy and productivity of these intellectual tools and processes that we as scientists and educators experience.Next, further investigation is needed to address the increasingly diverse student populations we seek to serve in our school system.While rare, there are some reported investigations into the application and applicability of Tutorials and other PER-based reforms for under-represented populations.[8][34] [46] Finally, while we demonstrated the Tutorials are scalable (or replicable) a major question of sustainability remains.Will other faculty adopt these practices?Do students develop a more positive response to these practices over time?Are there models of sustainable support for the Tutorials and the associated increased demands on the educational institution?

FIG. 2 :
FIG. 2: Correlating overall performance in course (course score includes all graded elements except Tutorial homeworks, expressed as a percent of total possible) with Tutorial homework score.Data shown is for Fall 03.The correlation coe¢ cient is r=0.73 (N=562).The solid line shows a best …t (regression) to the data.Data for Spring 2004 look very similar, the correlation coe¢ cient is r=0.65 (N=513).

FIG. 3 :
FIG.3: Histogram of percent favorable responses in the CLASS "concepts"category pre and post for students with di¤erent normalized learning gains (g) on the FMCE.

FIG. 4 :
FIG.4: Histogram of percent favorable responses in the CLASS "concepts"category pre and post for students in the bottom quartile for Tutorial homework scores (<72.8%) and in the top quartile (>91.5%).
FIG.5: Frames of Context: a particular task is embedded in a larger situation (a Tutorial class activity), which is embedded in a larger micro-culture (a particular class), and so forth.

(
dis)engagement to no longer participating in the course, or even providing negative feedback on the course questionnaire, which in turn shapes faculty practice.Thus, while faculty beliefs and goals may be aligned with reformed practices, systemic barriers to implementing these practices may prevent such e¤orts.[51]Optimistically, Tutorials and related PER-informed activities can a¤ect the broader institutional norms by developing sets of expectations among the participants in the educational system.Students and faculty may begin to reshape their expectations of their roles in a class, and tools such as surveys of students'mastery of concepts or beliefs may begin to become staple practices in the departmental use.Seeds of such a transition may be observed at the University of Colorado where faculty are beginning to more broadly adopt pre and post tests and discuss on a regular basis what the surrounding goals and measures of success are for the suite of undergraduate courses.Heuristics of implementing Tutorials at the level of Department and InstitutionTutorials require the support of the department or broader institution.The traditional model of course support does not provide the necessary resources required for sustainable implementation of Tutorials, which requires more space, sta¤, and training.There are many mechanisms by which such support might be provided, such as external or internal grants, TA training courses, and/or leveraging technology and human resources.These added resources allow the Tutorials to operate as originally designed.Simultaneously, e¤ective PER reforms can feed back to the broader system and bene…t not only individual students, but also departmental and institutional e¤orts in education, such as revising forms of teaching evaluation.Such positive feedback potentially justi…es the added departmental and institu-tional commitments.
Table I are a representative subset of the more complete assessment

TABLE II :
Scores on selected conceptual topics measured on exam questions, rounded to the nearest 5 percent.(UW data list percent with correct explanation, CU data are the average scores on the exam question.See footnote 11.) [24] compare the results of the UW implementation and the two semesters Fall 2003 and Spring 2004.The data show that we parallel published results from the primary implementation at UW.[20][21][22][23][24]In each of these cases we approximate the original results within 5% (the resolution of reported …ndings discussed above).
11C.Correlation StudiesMany interventions were implemented in this course (on-line homework, a version of Peer Instruction with a personal response system, and Tutorials), making it di¢ cult to directly correlate student performance on measures of conceptual mastery with performance on (or attendance in) Tutorials.For example, the correlation coe¢ cient of FMCE normalized gain with Tutorial homework is only 0.22, i.e.Tutorial homework performance is associated with only a small fraction of the variance of the normalized FMCE gain.Nonetheless this positive correlation is statistically signi…cant (p < 0.0001).The modest value of the correlation coe¢ cient may be due in large part to the clustering of many students with very high normalized gain, (almost 30% of the class had normalized FMCE gains above 90%).Similarly, the correlation coe¢ cient of Tutorial attendance to FMCE gain is 0.12, again statistically signi…cant but not large.Once again, this is not surprising, since average attendance (for students with measured FMCE gain) exceeds 95%.More telling is the correlation of Tutorial homework score with overall course grade, shown in Figure

TABLE III :
).Each week in class students were asked to report on various aspects of the course in Student responses to end of term online survey questions, rounded to nearest 5 percent.
Students have an overall favorable response to the utility of Tutorials and the group work, as well as other course components and the course evaluations as a whole.Striking, [54] subtle, but no less signi…cant, is that what counts for education shapes the broader commitment to Tutorials.In traditional institutions, teaching evaluation is at best loosely coupled with student learning.Most institutions, including CU, utilize the end of course questionnaire as the predominant tool for evaluating faculty teaching practice.Plenty of discussion and some research has examined the limitations of such tools.[54]However,it is generally recognized that these course questionnaires often do not re ‡ect measures of student learning.Simultaneously, students themselves have been embedded in an educational system valuing performance (distinct from learning) and have come to value evaluation more than understanding.Thus, students have fairly well established beliefs about what practices constitute an appropriate educational experience.When new and di¤erent forms of practice are introduced, especially those that require more e¤ort, students often resist.These forms of resistance may be realized in each frame of context discussed above, from task