Educational transformation in upper-division physics : The Science Education Initiative model , outcomes , and lessons learned

Stephanie V. Chasteen, Bethany Wilcox, Marcos D. Caballero, Katherine K. Perkins, Steven J. Pollock, and Carl E. Wieman Department of Physics, University of Colorado Boulder, Boulder, Colorade 80309, USA Department of Physics and Astronomy and CREATE for STEM Institute, Michigan State University, East Lansing, Michigan 48824, USA Department of Physics and Graduate School of Education, Stanford University, Stanford, California 94305, USA (Received 28 September 2014; published 23 September 2015)


I. NEED FOR A MODEL OF CHANGE
There is a rising tide of attention to the improvement of undergraduate science, technology, engineering, and mathematics (STEM) teaching.This attention is driven by the increasing emphasis on science and technology as the current and future driver of national economic growth and the results of discipline-based education research (DBER), which is indicating clear opportunities for improvement.We see evidence of this increased attention in numerous calls for action from disciplinary societies, national networks, and federal agencies [1][2][3][4][5][6].
However, as long as the impetus, success, and preservation of change relies on single individuals, rather than organizational structures that support that change, teaching innovations will be inherently fragile, and scaling up those efforts will be challenging.Previous research has demonstrated the high rate of loss of both the number and fidelity of use of instructional innovations [7,8].A focus on individual decision making and creation of instructional materials is also inherently inefficient due to duplication of effort, as faculty continually reinvent the wheel.A new model for change is needed that guides coherent collective efforts by making innovative, effective teaching an institutional goal that is embedded in the established organizational structures.Such a model would allow this rising tide of attention to STEM education to lift our collective boats towards a higher goal, rather than leaving STEM faculty paddling around on their own in an attempt to keep their heads above water.
As a field, scholars in STEM education are working to both create and investigate models of change that realize the potential of educational reform efforts [9], especially those which go beyond the traditional "development and dissemination" model of change [10,11].In this article we attempt to answer previous calls [9,11] for (a) a more sophisticated model of change and for (b) research on the efficacy of these models of change.

A. Overview
The Science Education Initiative (SEI) was an experiment in institutional structure to support change.Led by Wieman, the SEI provided funding at a departmental level, enabling short-term appointment of postdoctoral fellows to partner with faculty on course transformation using a backwards-design model [12].The SEI was implemented at two institutions: the University of Colorado Boulder (CU) [13] and the University of British Columbia (UBC) [14].The CU SEI began in 2005, and the UBC program one year later.A total of seven departments were funded at each institution, including the Physics Department.
The SEI focuses on change at the departmental level and includes partnership among multiple change agents and resources, as shown in Fig. 1.Funding was allocated to individual departments based on the strength of the proposal submitted to SEI Central by the department. 1 While departments were given substantial leeway on how to spend their funds, all departments used funds to hire postdoctoral fellows to partner with faculty on SEI activities. 2  Additionally, each department selected a departmental director who helped to supervise and guide the science teaching fellow's (STF) work, structure interactions between the STF and faculty, and serve as an advocate for the program within the department.A central administrative body-SEI Central-played the role of a highly involved funding agency, and provided advice and training to STFs and departmental directors throughout the program.
STFs had typically earned a recent Ph.D. in the discipline and were interested in education research and (or) reform.Most STFs came with limited education research experience-in some disciplines DBER-trained candidates were quite scarce at the time, plus these positions were not intended to be primarily research positions.In addition to strong content knowledge, selection of STF candidates focused on evidence of strong general skills necessary for this unique role (e.g., interpersonal, group facilitation, time and task management, organizational, communication, etc.).The position often provided a pathway into educational reform and in fewer cases, DBER, for disciplinary Ph.D.'s.This model contrasts with that of a typical physics education research (PER) postdoctoral experience, which would be concentrated on a specific research agenda, and explains the relative absence of PER-trained STFs (only 1 at CU Physics).STFs thus required training in researchbased instruction and education research, provided by SEI Central.The need for a carefully designed training course, 3  and a community of STFs, makes the SEI model potentially challenging to implement in institutions without DBER knowledgeable faculty.
The desired outcome of the SEI was to (a) achieve measurable impacts on the number of courses using evidencebased instruction, (b) sustain those course changes over time, (c) demonstrate impacts on student learning, (d) gain broader acceptance, use, and expertise in evidence-based teaching among faculty, and (e) fuel a cultural shift within departments such that evidence-based teaching is considered the norm.We used two main strategies to pursue these outcomes.First, by having departments lead their own course transformation efforts, we expected faculty would share common expectations for outcomes and instruction in the transformed course, and thus that the new course structures and materials would be more likely to be maintained.Second, by using course transformation as the main vehicle for change, we expected that faculty development would occur naturally as faculty partnered with STFs on transformation of courses, and that shifts in the practices of these individual faculty would lead to a general shift in departmental norms for instruction.As we show later in the paper, the overall structure of the model was found to be generally successful.However, later modifications helped account for the lack of institutional incentives for change.

B. Focus of this paper: SEI in physics at CU
This paper outlines the model and outcomes of the CU SEI in the Physics Department ("SEI Physics") and discusses how features of SEI Physics contributed to those outcomes.
While most departments funded by the SEI focused on introductory courses with some progressing to upperdivision courses, the SEI effort in the CU Physics Department focused on the upper-division level from the outset.The set of courses forming the focus of SEI Physics were the canonical courses taken by majors after the freshman year: Classical Mechanics and Mathematical Methods (2210), Electricity and Magnetism I (3310), Electricity and Magnetism II (3220), and Quantum Mechanics I (3320).Additionally, the Advanced Laboratory (3340/4430/5430) experimental optics course was the focus of a series of grant-funded efforts, which included partial funding from the SEI, and Modern Physics (2130/2170) was transformed prior to the SEI.This paper reports on the first four courses as the focus of the SEI.An annotated bibliography of work arising from these efforts can be found in the Supplemental Materials [17].
SEI Physics differs from the SEI implementation in other departments in several ways (described in Sec.IV) largely due to the presence of a PER group and significant external funding, allowing focused postdoctoral effort on each course over a period of years.These differences had diverse implications, as we will show.

A. Course transformation philosophy overview
The majority of SEI projects focused on course transformation, including the development of learning goals, identification of student difficulties, and creating materials and assessments.Below, the process of course transformation is described in detail, with a focus on upper-division physics.
Our approach to course transformation is based on a philosophy of backwards design [12].First, define learning goals and objectives, then determine how well students are meeting those objectives, and finally create learning experiences to help support student success.Figure 2 shows the three central questions directing SEI course transformation efforts.Course transformations take at least two, and more typically three, semesters-one semester for a planning period, one for the bulk of the course transformation itself, and a third during which there are substantial refinements but involving much less STF time.Faculty typically continue to make minor adjustments and refinements over the next few semesters as they do in any course.In SEI physics, testing of instructional materials occurred over longer periods of time.More detail about the SEI approach to course transformation can be found in prior publications [18,19].All physics course materials can be found in Ref. [20].

B. Learning goals: What should students learn?
The first step in course transformations was to establish faculty working groups to generate consensus learning goals [21][22][23].Learning goals are explicit statements of what a student should be able to do at the end of a course, and may address conceptual understanding, skills, metacognition, and beliefs.Learning goals may address the course as a whole (e.g., "Students should be able to translate a physical description of a junior-level electromagnetism problem to a mathematical equation necessary to solve it.")or learning about a specific topic (e.g., "Students should be able to explain how conductors shield electric fields").These goals are then used in course design, enabling creation of a coherent course in which the goals, assessments, and instruction are aligned [12].In SEI Physics, we achieved much higher rates of faculty engagement in the process of creating learning goals than in many other departments for a variety of reasons [23], although development of goals in Quantum Mechanics was particularly problematic due to the lack of canonical course content or approaches [24].The result is a set of ∼10 course-scale learning goals and lists of several objectives relevant to each topic in the course.

C. Student difficulties: What are students learning?
One aspect of the SEI's model of change is to identify where students encounter challenges in learning the material, so that those challenges can be addressed through instructional intervention.PER has a long history of such investigations in the lower division [25].This section describes, briefly, how we have investigated student difficulties in the upper division; a more detailed overview can be found in Ref. [26].
The investigation of student difficulties includes observations and field notes during lecture, observations from student homework help sessions, student solutions to exam and homework questions, student responses on conceptual assessments, and interviews with students.These documented student difficulties are shared with faculty and used to develop or improve clicker questions, tutorials, and other materials.
The expertise gained over the long time scale of our transformations, and the resources of the PER group, have allowed physics to engage in relatively sophisticated analysis of student difficulties at the upper-level.At the beginning of our work, very little existed in the area of upper-division difficulties, but now a body of literature exists, including work from CU [27][28][29][30][31][32]  020110-3 [33][34][35][36][37][38][39][40].These investigations have provided insight into various concepts, producing lists of difficulties in various topics including Ampere's law [30], Gauss's law [28], and vector calculus [29].More recently, we have developed an analytical framework to investigate findings more strategically and to organize these difficulties more coherently [32] and analyze particular areas of difficulty [27,31].

D. Material development: What instructional approaches improve student learning?
There are many ways to go about this part of the transformation-the most important consideration is that the course materials be aligned with the learning goals and the results of research into student difficulties.STFs act as education experts in this work, suggesting and developing instructional materials based on known techniques.In SEI Physics, STFs typically observed a traditionally taught semester of the course prior to the transformation (not always the case in other departments), and the transformed course was developed by the STF and a PER faculty member.See Refs.[41,42] for more information about materials developed in SEI Physics, Ref. [43] for a discussion of tutorial development, and Refs.[44,45] for a discussion of the use of concept tests in upper-division physics.
Once developed, materials for a course were compiled into an organized course archive.The UBC program led an effort to create an online archive [46] so that materials across departments and institutions would be centralized and organized into a common structure.We note that it was challenging to create a model that worked for all possible cases and was easily used, and (regrettably) this online structure serves more as a resource for SEI staff than for faculty.In the Physics Department, materials are primarily passed to instructors via a downloadable zipped folder, which was later formalized into a web structure; see Ref. [20].
Course material packages in SEI Physics were not static; over the course of the project, instructors (both PER and non-PER) modified and (or) added to the archives, which were updated periodically.While this reflection and revision was perhaps more focused and deliberate in the case of PER instructors, several non-PER instructors also made modifications-often in terms of clarifying or adding clicker questions.Once the SEI funding ended, however, there was no clear mechanism or responsibility for updating the archives in SEI departments.

E. Assessment: What are students learning?
To quantitatively assess the success of course transformations, one needs a standardized and validated measure of student learning that is aligned with learning goals.For more about the process and challenge of creating such validated assessments, see Ref. [47].
In physics, student learning has typically been measured at the introductory level using validated conceptual assessments like the Force Concept Inventory (FCI) [48] and Brief Electricity and Magnetism Assessment (BEMA) [49].Unfortunately, very few validated assessments had been developed for the upper-division level when the SEI began its course transformation efforts.To address this, we set out to create new, primarily open-ended, assessment instruments specifically designed to target our consensus learning goals.We developed four upper-division conceptual assessments, each targeting the content of one of the core courses being transformed; more information on our assessments and their development can be found in previous work [50][51][52][53][54][55] with an overview in Ref. [56].Assessments were developed for Electricity and Magnetism I (CUE [50]), Quantum I (QMAT [53]), Math Methods and Classical Mechanics assessment (CCMI [55,57]) and Electricity and Magnetism II (CURrENT [54]), with multiple-choice formats being developed for the CUE [51] and QMAT [52].
Our upper-division assessments, while invaluable parts of the course transformation process, have some significant barriers to implementation, including time to administer and to grade, and variation in content coverage among faculty.For more discussion of such barriers, and solutions, see Ref. [56].
Development of such assessments was a more explicit focus in SEI Physics than in other departments, due in part to the course-targeted model of transformation, the existence of the well-established PER community, and the interests of STFs.These assessments required a significant investment of time, but were valuable in supporting the research base of our upper-division work, and there was external grant support for this work.

IV. THE CONTEXT OF THE SEI IN PHYSICS
In addition to focusing on upper-division transformation, SEI Physics was somewhat unique among departments in terms of its history and the influence of the PER group.Here we describe those contextual factors, many of which appear to have impacted the model and outcomes of SEI Physics.

A. History of course transformation
The CU Physics Department has long engaged in scholarly transformation of its introductory courses using interactive engagement techniques and the use of research and analysis to inform its efforts.Peer Instruction [58] was first used in lecture in 1997, soon becoming standard across all the introductory courses, and the Washington Tutorials [59] have been in use in recitation since 2003.In 2003, Wieman hired Perkins as a postdoctoral fellow to collaborate on course transformations in Physics of Everyday Life (PHYS 1010-1020) and Modern Physics (PHYS 2130) [60] using a scientific approach to teaching, including the creation of learning goals, assessments, and instructional materials.Success in this early effort led Wieman to create the SEI.Thus, while SEI Physics was not funded until 2007, scientific teaching efforts had been under way for more than a decade.
Because of its history of innovative teaching, SEI Central had anticipated that the CU Physics Department would be a leading contender for SEI funding.However, there was greater up-front resistance among some faculty in the CU Physics Department to making a departmental commitment to carry out the vision of the SEI, particularly the aspect of carefully developed "departmentally owned" and maintained courses along with measures of student learning (though many faculty in the department are quite accepting of research-based educational techniques).In other departments, such resistance sometimes arose during the course of the SEI once it was clear what would be required in order to achieve the goals of their SEI proposal.Thus, physics faculty are not necessarily intrinsically more intransigent, but previous experiences had already given rise to established opinions about transformed teaching, including the difficulty of getting universal adoption of instructional and assessment techniques by faculty and the importance of maintaining faculty choice.Thus, the initial the Physics Department proposal explicitly said that there would be no departmental expectations established for how future faculty would teach the SEI-transformed courses, and it so was not funded. 4But as the CU SEI proceeded, it became clear that other, funded, departments were not living up to their commitments to varying degrees (e.g., assigning courses under development to temporary faculty or failing to come to agreement on learning goals or instructional methods to use for a core course).As the Physics Department remained interested in SEI participation, an agreement was negotiated for more limited funding.
The focus on upper-division physics was not an explicitly stated goal in the SEI Physics proposal, but rather the goals of the SEI in the Physics Department were subsequently aligned with the goals of a simultaneously funded NSF proposal [61] that aimed to transform Electricity and Magnetism (E&M) and Quantum Mechanics, a project led by a clear champion who was and is a PER faculty member.A focus on the upper division was the natural next step in the department for many reasons: There had already been extensive work in the introductory level, resulting in near maximum gains on conceptual assessments such as the FCI [48] and Force-Motion Concept Evaluation (FMCE) [62], and there had already been extensive discussion and broad faculty interest in the goals and design of the upper-division courses.

B. Unique factors in the CU Physics Department
Across departments at CU and UBC, the CU Physics Department represented the most rigorous adoption of the SEI course transformation approach (Sec.III), including the most successful effort to create learning goals that represented a true departmental consensus and development of validated assessments tied to those goals.Course transformations in SEI Physics reflected a focused, long-term effort, which often required 3 or more years of postdoctoral time.
This increased depth and rigor in SEI Physics is due to several factors.First, the CU Physics Department started out in a rather different place from other science departments at CU or UBC, as described above.Second, after a few years, the SEI departmental director role was taken over by a PER faculty member, who had also been the primary person working with the STFs on the course transformations throughout the effort.Third, the existing PER group formed a strong academic base for those STFs, providing expertise in creating assessments, measuring student outcomes, and generating publications.Lastly, efforts in the Physics Department were part of a much larger, research agenda; while the funding from the SEI (∼$460 000) was less than that of several other departments, this funding was supplemented through multiple NSF grants oriented towards course transformation, for a substantial total funding level of (∼$1.3 million).Thus, the PER influence provided expertise lacking in many departments, but this associated research agenda also colored the course transformation process in the Physics Department, as will be discussed below.
Another contextual factor that we found to be important is that, unlike many departments in the SEI, there is frequent rotation of faculty among courses in the Physics Department, such that an instructor may only teach a specific course a few times in a five-year period.High levels of faculty rotation have important implications for how to achieve sustainability and ownership: Once course materials are created, multiple faculty members need to be brought "on board" in order to sustain those changes.Thus, courses in the Physics Department were typically transformed in collaboration with PER faculty, with those course packages handed off to subsequent instructors rotating into the course.In contrast, in other departments, STFs worked with a broader pool of faculty, many of whom would transform a course and then teach it repeatedly.
Additionally, STFs in other departments typically worked on several courses over their tenure as a STF and sometimes on more than one in a given year.STFs in SEI Physics typically worked on a single course during the tenure of their position.
Table I provides an overview of the contextual factors underlying the SEI in the Physics Department, as well as several potential consequences of these contextual factors, which will be considered when interpreting outcomes from SEI Physics.

V. HOW WELL DID THE SEI ACHIEVE THE DESIRED OUTCOMES?
Here we present data on the success of the model in impacting courses, students, faculty, and departmental culture.In order to provide insight as to the relative success of the SEI Physics, as well as the impact of contextual factors on these outcomes, we compare the outcome data in the Physics Department with two additional departments.In Sec.VI, we will use this outcome data to drive a discussion of the implications for upper-division course development, sustainability, and research.

A. Methods
This section draws from multiple data sources.For data that are not described in previous work, we describe the research methods below.The data are necessarily limited, as the university funds that supported the SEI were provided to carry out instructional changes, not to do extensive research on outcomes.

Course and faculty impacts
As part of the internal assessment of the SEI project, SEI project leaders conducted interviews with departmental STFs and departmental directors in 2009, 2013, and 2014 to gather data on how courses and faculty had been impacted by the SEI.Interviews were typically timed to coincide with the departure of one or more STFs, and typically required 2 h.Using a structured spreadsheet, information was gathered on each course and each faculty member in the department, such as whether learning goals or clicker questions had been developed for that course, or whether a faculty had participated in learning goal discussions or made substantial use of the STF.The exact data collected are detailed in the Supplemental Material [17].These data were further validated through follow-up discussions with departmental directors and STFs and cross-checking with annual reports from each department.The Institutional Review Board (IRB) reviewed use of this data and all others involving human subjects for this study.
Course or faculty "impact" was defined as the total number of changes cataloged through these interviews for an individual course or faculty member; impact is thus an interval measurement.The total number of possible changes for a given course was 14, and the total number of possible changes for a given faculty member was 9. Additionally, these data were categorized to align with the original aspects of the SEI model: learning goals, assessment, and instruction.This impact measurement allows comparison of the impact on faculty or courses across departments, but does not allow comparison of the impact in different categories (e.g., learning goals versus • Little prior work to guide efforts • Small classes (small N) pose challenges for research studies a PER faculty leadership began formally at the start of year 4 when author S. J. P. was designated departmental director, but SEI Physics efforts were informally led by PER in earlier years.assessment) within a department.This is because it is harder to achieve a "perfect score" in some categories (e.g., instruction) than others, because each category contains different numbers of items.Additionally, this method simply counts the impacts, rather than evaluating the quality of those impacts, which is a limitation of our data.The impact data show considerable, nonsystematic variation across the seven departments, due to local variables.We chose departments C and D as comparisons with the Physics Department, because they represent cases in which the SEI impacted the teaching of a small fraction of the faculty (department C) or a very large fraction (department D).

Student enrollment data
Student enrollment data for each course were obtained from the Office of Planning, Budget, and Analysis.In cases where the course offerings have changed since the SEI was involved in the department, we used the course offerings at the time of the end of the SEI project.The student enrollment was averaged across the past two academic years (AY 2012 and AY 2013).

SEI midway faculty survey
In Spring 2010, a short survey 5 was given to faculty in all the departments of the CU SEI to document their level of interaction with the program.Of the 162 faculty who were asked to participate, 114 faculty responded, for a relatively high response rate of 70%.

Physics instructor survey
All physics instructors who had taught a course that had been the subject of a full SEI transformation effort were invited to participate in a survey regarding their use of materials 6 and the impact of the SEI on their instruction in Summer 2014.The survey was developed to assess the longevity of SEI materials in the transformed courses, as well as to assess the influence of several factors found to be important in the faculty change literature [7,8,65,66], such as time required to use educational materials and the tendency to make modifications.Out of 17 instructors, a total of 13 unique faculty responded to the survey or participated in an interview.Author S. J. P. was not included in this analysis, due to the deep nature of his involvement in the course development.

Course transformation over time
Course materials included learning goals, student difficulties, lecture notes, clicker questions, in-class activities [tutorials and (or) whiteboards], out-of-class tutorials, online preclass questions, homework, exams, and conceptual assessments.We gathered data on the level of use of each of these materials in each course since the beginning of SEI involvement, using (a) the physics instructor survey, (b) faculty interviews, and (c) observations from members of the PER group.This resulted in complete data on a total of 23 semesters of instruction and partial data on an additional 5 course offerings, out of a total of 31 course offerings since the development of SEI materials in each course.Information was gathered as to whether a material was used not at all, a few times, periodically, or consistently.For sustainability analysis, material use was recoded as "used" or "not used": Materials that are intended to be used only a few times were coded as used if any amount of use was reported (learning goals, student difficulties, homework, exams, conceptual assessment).For other materials meant to be used consistently, only periodic or consistent use was coded as used (tutorials, clickers).Records for which we had incomplete information were dropped, as were courses taught by the developers.Each course was then given a "fidelity score" representing the sum of all materials used.This fidelity score can be compared to the "ideal" implementation of the first semester, as taught by the developer (typically using 7 or 8 materials).

External evaluation interviews
As part of an external evaluation of the SEI, a CU researcher conducted in-depth, semistructured interviews with samples of individuals involved in the change initiative: SEI leadership, including institutional administrators, project leaders, department directors, STFs, SEI-engaged faculty, and the chairs of SEI-participating departments. 7Interview protocols explored individuals' knowledge of the change effort, their role within it, their experiences in SEI, their attitudes and beliefs about teaching and learning, self-reported changes in these as a result of involvement in the change initiative, issues of autonomy, motivation, and resistance to SEI, and whether and why the initiative was seen as successful or not.
Individuals were solicited to participate in an interview and provided a letter of consent for the study via university E-mail.Out of 65 individuals invited, 54 were interviewed for an overall response rate of 83%.Interviews were conducted individually, lasted 1-2 h, and were digitally recorded and transcribed.These data were released to the SEI team (authors S. V. C. and C. E. W. only) under a separate IRB protocol, dependent upon individual consent, with the provision that (a) the individuals remained deidentified, (b) administrators and former or current STFs employed at CU were not included, and (c) individuals were allowed to redact their statements.Given these restrictions, invitations were provided to 42 individuals, and a total of 24 individuals agreed to release their transcripts to the SEI.

B. Impacts on courses
A main focus of the SEI overall model was the transformation of individual courses.Through that transformation, it was expected that student learning would be improved and that faculty would change their teaching practices in these and other courses.We consider several metrics of course impact (the number of courses and students and types of impacts made) across all three departments, and we take a more intensive look at the instructor decision making in the Physics Departmentwhat materials they used, what modifications they made, and what changes were sustained over time.

Comparison departments
Department C is taken as an example of an only moderately successful implementation of the SEI model, where only a small number of STFs were hired, and there were significant problems with support of the chair and faculty buy-in to the SEI, including distraction by a major structural change to the department.STFs struggled to gain traction in this department, with the bulk of the work eventually centered on recitations in a single introductory course and partnering with a few friendly instructors, most of whom were not tenure-track faculty.The department as a whole is seen as relatively unaffected by the SEI, though changes in the recitation have been maintained.
In department D, by contrast, STFs worked in several introductory courses, impacting a large number of students, and partnered successfully with a broad range of faculty.In this department, the SEI coincided with an existing push in the department to redesign their major.Faculty in the department were deeply engaged with the SEI, with tangible support and involvement from the chair, who explicitly expected that faculty would use SEI-created materials.This department has continued to be involved in educational transformation, engaging in a national initiative, and hiring one of the STFs as a permanent instructor.They have also continued to work directly on SEI-related efforts to align their learning goals across courses to create a more coherent curriculum, with course releases granted for this purpose.

Number of courses and students
One measure of impact is the total number of courses impacted in some way by reforms.A total of 46 courses were the subject of focused STF effort ("full involvement") across the CU SEI as a whole-of which 6 were in the Physics Department.An additional 57 courses were the subject of partial involvement, of which 5 were in the Physics Department.This cost and extent of transformation varied across departments and courses.The average investment was $145 000 per course with full STF involvement, not including the Physics Department.In the Physics Department, the cost was $212 000 per course with full STF involvement, reflecting the more intensive, sustained research effort in that department.
Across the three case-study departments, a similar number of courses were the subject of "full involvement" in each department (4-6).As described earlier, in SEI Physics each course transformation was the focused project of a single STF over several years, with little effort spent on other projects.This focus has an impact on the types of changes made in those courses, as we show below.
These six transformed courses in the Physics Department represent a small fraction of the overall student enrollment served by the department (6%) but a sizable fraction of the student enrollment in upper-division courses for majors (38%).There have been 1601 (not necessarily unique) students enrolled in our SEI fully transformed courses in the 7 years since the SEI began.In other departments, the percent of the department's total student enrollment accounted for by courses with full involvement was much higher, ranging from 27% to 55%, reflecting the focus on the high-enrollment introductory courses.Student enrollment in transformed courses is represented graphically in Fig. 3.
Students taking upper-division physics courses have a high probability of experiencing transformed pedagogy due to the SEI.In fact, the only core course for our majors that has not been the subject of some level of transformation effort at this time is Quantum II (4410).When including reform efforts prior to the SEI, the vast majority (89%) of the total annual student enrollment across all courses for physics majors occurs in courses that have been transformed to include active learning.However, not every instructor chooses to use the active learning materials when teaching those courses.

Types of impacts in courses
To quantify what was actually accomplished within course transformations in a given department, we calculated the average number of impacts in courses from documentation gathered during interviews with STFs and faculty, such as whether learning goals were developed or group work added (see Sec. VA and Supplemental Material [17]).The average number of course impacts in each department is shown in Fig. 4. Averages are calculated within the subset of courses in which the SEI worked, as well as across all courses in the department-to provide insight into how the full body of departmental course offerings was impacted by the SEI.
The focus of SEI Physics on extensive transformations in a small number of courses is reflected in this graph by the large average number of changes in courses with SEI involvement in physics.SEI Physics also had a (somewhat) large number of changes when averaging across all upper-division courses (Fig. 4, right); because SEI Physics was working within a smaller set of courses (the upper-division curriculum), it was easier to affect a large percentage of courses within that curriculum.
When we investigate the types of practices that underlie the large number of course impacts in the Physics Department (data not shown), we find that the differences in the number of changes is largely due to a larger number of instructional changes in courses in SEI Physics (average changes 7.9 AE 0.63 changes) compared to departments C (3.9 AE 0.95) and D (3.5 AE 0.63), rather than to changes documented in learning goals or assessments.SEI Physics followed a rather systematic approach to course transformation, with many common elements across courses (e.g., clickers and tutorials), supported by many years of postdoctoral effort, which likely underlies this quantitative difference.
Another measure of the different focus and resources available in physics is the higher publication rate in physics (∼33 publications to date) compared to other departments (typically 5 or fewer, with the exception of Molecular Developmental and Cellular Biology-17 publications-a department which also has faculty active in DBER).

Sustainability over time
Simply because a course has been transformed does not mean that those course transformations will be sustained in future iterations.While data across departments are still being collected, results from the SEI-wide "midway" survey are promising on this aspect; out of 108 instructors who had made instructional changes, 94 (87%) said that they planned to repeat those same changes or had already done so.In SEI Physics, we were able to directly explore the question of course sustainability in each of our transformed courses.
Course fidelity.-Inorder to provide a rough picture of "how transformed" each course remained over time, courses were given a fidelity score representing the use of SEI course materials, which included learning goals, student difficulties, clicker questions, in-class activities [tutorials and (or) whiteboards], out-of-class tutorials, online preclass questions, homework, exams, and conceptual assessments.In Fig. 5, these results are plotted for each transformed course.The first instance is the course as taught by the developers, which serves as an upper bound (and the measure of the fully transformed course as envisioned)-all instruction by developers is indicated by a star.The first semester without STF support is also indicated.
Variability can be understood on a case-by-case basis: For example, Courses 1 and 2 were taught multiple times by faculty who were "friendly" to the SEI, so the level of transformation maintained itself over time.In Course 3, reforms successfully sustained over several semesters, as reported earlier [67], suggesting early success-but then several instructors taught the course who chose not to use the SEI materials and methods.This return to traditional teaching may have contributed to a return to the traditional norm over time.In Courses 3 and 4, however, we see that it is possible for a course to return to a high level of reform even after having been taught traditionally for a few semesters.Withdrawal of STF support, as well as later instruction by a developer, has variable effects-though in Courses 3 and 4 we can see that instruction by the developer is followed by a return to a somewhat higher level of reform by the subsequent instructor.
Interviews have suggested that, across departments, instructors who have not been involved with the SEI course transformation efforts, and subsequently rotate into an SEItransformed course, are often more reluctant to use those materials and approaches than instructors who have had SEI involvement.This suggests that after the SEI tenure in a department ends, courses that continue to be taught by instructors who were involved in the transformation process are more likely to maintain changes than courses for which many non-SEI-involved faculty rotate in to teach.
Interpreting fidelity scores.-Wecan use the above data to estimate sustainability rates (instruction by the developers is not included in these estimates): 67% of the different semesters of the transformed courses in physics maintained the use of at least half of the instructional materials and approaches developed.However, if a higher level of fidelity is required (70% of the instructional materials and approaches), the rate of sustainability drops to 46%.We choose 70% as a cutoff because this allows for dropping of two course elements, which could still represent a relatively consistent approach with the developer.
In comparison, previous work at UBC [68] showed that practices were sustained nearly 100% in courses where faculty developed or adopted different teaching practices with the personalized coaching of a STF.This study also showed that these instructional changes were almost always applied in new courses that the faculty member teaches.In department D, STFs partnered with faculty as a coach, as described in Ref. [68], and interview data suggest that course sustainability in department D was similar to that reported in Ref. [68].
Among those physics instructors who choose to use the SEI materials, their fidelity of material use is quite high, consistent with results from other departments.It can be seen from Fig. 5 that some instructors have chosen to opt out of the SEI approach: 17% of semesters used one or fewer course elements (usually indicating that they used our conceptual assessment at our request).If we remove these nonparticipating instructors from our sample of interest, the remaining instructors used the SEI course packages with relatively high fidelity (average fidelity score 68%), on average dropping 2-3 course elements.Reasons for low fidelity scores were idiosyncratic, but most often dropped were preclass assignments, exam questions, and (about half the time) learning goals.Exam questions were presumably not used for fear of encouraging student cheating.Online preclass assignments (i.e., online questions aimed at encouraging students to think about material before class) were seen by faculty as administratively challenging, and are also the only course materials receiving a consistently low rating by students (see Sec. VI C)factors that likely lead to their lower use among faculty.Student learning goals, ostensibly, provide a guiding compass for course instruction, and it could be problematic that these are not used consistently.Because learning goals were developed through working groups at CU Boulder, these instructors may have felt less need to consult themhaving either been involved in these working groups or trusting that the course elements were aligned with those goals.In contrast, faculty from other institutions using the materials consistently rate learning goals as some of the more useful aspects of the course package and state that the goals were very useful in guiding their instruction.
Several elements were very strongly maintained in the physics courses at CU: Among courses making any use of SEI materials, clickers were used by all instructors, and tutorials by all but three instructors, which is a positive outcome.Conceptual assessments were frequently used, but often this was at the request of PER group members.
Handing over the course package.-Giventhat SEI Physics was reliant on "handing off" course packages to new faculty rotating into the course, we were interested in examining how course materials can best be provided to instructors so as to maximize the chances of a positive impact on the course and students.This is of particular interest given documented difficulties with such development-and-dissemination models [7].Organization and posting of online course archives was a significant focus in SEI Physics.This was due in part to the need to provide an easy-to-use package for the faculty rotating into the transformed courses (which is not as big of a concern in departments with less faculty rotation), as well as the broader, dissemination goals of the external grants that partially funded this work.Interviews, observations, and instructional experience allowed STFs in SEI Physics to develop increasingly sophisticated course archives, and course packages were organized both by material type (e.g., all clicker questions placed together, all tutorials placed together) and by content area (e.g., all materials on Gauss's law placed together) and provided as a zip archive and online [20].In interviews, faculty state that the organization of the course archive is an important aspect of the ease of adoption of course materials, and that a poorly organized archive can pose a significant barrier.An important open question is how to best support a dynamic archive, where faculty can easily add to the existing materials (e.g., creating a broader bank of homework problems) or make suggestions for modifications, and how to provide personnel for maintenance of the archive.Currently, maintenance depends on PER faculty and postdoctoral fellows.
The course archives themselves are only part of the course package hand-off.In order to inform instructors of the course package, one of the authors (S.J. P.) contacts instructors in the semester prior to their instruction, sending a link to the course package via E-mail and offering to meet to discuss it (which faculty take up to varying degrees).
Faculty were asked about the impact of this type of interaction with the course developers.Among faculty who used more than just a few of the SEI materials, meeting with the course developers was rated as significantly more helpful than communicating with the prior instructor of the course (see Supplemental Material [17] for data), suggesting that discussions with the PER developer are different from the typical course hand-off between instructors.
The nature of these conversations is likely to have an impact on course material use as well (e.g., whether a particular course element is emphasized or not).How course developers can most productively work with instructors who rotate into SEI-transformed courses is an area that is not well understood and, likely, would have benefited from additional attention during the project.
Modifications to materials.-Basedon previous reports in the literature [7,65,66], we expected that faculty would want to make modifications to our materials, and hence we provided course materials that were easily changed.Faculty, on average, indicated that they made modifications to the materials about half the time.Faculty comments (N ¼ 7) indicated that modifications were nearly always made for concerns of content, rather than pedagogy, for example, to match the way that content was presented in class or to address specific examples from lecture.While it is likely that pedagogical changes were indeed made (and perhaps not at the forefront of instructors' minds when they consider modifications), we think it is noteworthy that instructors' modification decisions are most explicitly driven by content-related concerns.Thus, educational developers may find it useful to remember that faculty are highly motivated to help students learn content-and so embedding conversations about teaching in the particular content at hand (as opposed to focusing on general pedagogical principles) may create higher engagement.
Faculty indicated that the aspect of the transformed courses that they found most challenging were the implementation of the tutorials.Many faculty found it difficult to coordinate tutorials with course content, and faculty who did not choose to use the materials reported a strong concern about time.Tutorials were sometimes given as an optional out-of-class recitation, adding logistical complication.In all cases, tutorials were eventually adapted or developed as shorter, modular activities that could take place within a lecture period.Recommendations for inclass tutorial development can be found in Ref. [43].
Summary.-To summarize these data on sustainability, we find that subsequent physics instructors exhibited varying levels of "fidelity" to the course as envisioned by the developer, sometimes not using any materials, but often using several critical items (such as clickers or tutorials).High levels of faculty rotation in the Physics Department (resulting in continual handing off of course materials to new faculty), as well as the fact that course materials were largely developed by STFs in partnership with PER instructors, likely play a role in the challenge of sustaining practices within transformed courses.Physics faculty's interest in active learning was demonstrated by their lack of explicitly pedagogically driven modifications to course materials and their decision to use tutorials as an in-class experience when institutional constraints limited their ability to provide tutorials as a separate section.An organized course archive plays a role in the ease of adoption of course materials, as do conversations with developers, though the lack of continued maintenance of this archive poses another barrier to sustainability.

C. Impact on student learning
Did the SEI efforts impact student learning in transformed courses?In this section, we report on student learning in SEI Physics as measured by conceptual assessments and exams, as well as students' self-report.
In other departments, we have varying amounts and quality of evidence of student outcomes, because of the varying emphasis on assessment, and so are unable to provide comparison data.Rigorous assessment of student learning gains and comparison across different teaching methods was more feasible in physics due to the existing expertise of the PER group, a history of conceptual assessments, the long time scale of the course transformations, and the somewhat lockstep adherence to the SEI model of course transformation.SEI Physics was also particularly attentive to collecting both baseline and post-transformation data, but it was not easy-logistically or sociologically-and created tension in the department.In all departments it was problematic to demonstrate evidence of improved student learning as courses are transformed, as this requires administration of the instrument over several semesters, and baseline data is often difficult to gather.Usually, assessments were not available until after the course approach had already changed.Where assessments were developed and administered in other departments, and baseline data exist, we consistently see evidence of increased learning.
In SEI Physics, we will focus on our Electricity and Magnetism I assessment (the CUE [50]) for several reasons: it has the largest pool of data to draw from, it is the oldest assessment, and the availability of CUE data from earlier, nontransformed classes (now an increasing rarity at CU).The CUE is an open-ended assessment emphasizing students expressing their reasoning and is accompanied by a complex grading rubric requiring grader training.Figure 6 shows that courses using our transformed materials consistently scored higher on the CUE, the conceptual assessment for Electricity and Magnetism I. Treating courses as data points, transformed courses averaged 58.0 þ 2% while traditional courses averaged 42.3 þ 3%.These crossinstitutional, long-term data provide the strongest evidence of increased student learning as a result of upper-division course transformations at CU.
While scores on the CUE show that the course transformations positively impact students' conceptual skills, a common concern voiced by some faculty was that transformed courses would produce students with poorer calculation skills.To address this concern, we gave a subset of common exam questions to three semesters of our junior Electricity and Magnetism I course: one traditional and two transformed [41].The questions were scored by an independent grader using a common rubric.We found that students in the transformed semesters scored equally well on the calculation elements, and better on their ability to explain their reasoning, or justify an answer.These data show that, in electrostatics, the course transformations do not sacrifice students' calculation abilities, but they do enhance performance on conceptual and sense-making questions.
One drawback to these course-wide assessments is that they do not provide information as to the individual contributions of different aspects of the transformed courses FIG. 6. Histogram of average course score on the CUE demonstrating improved performance for courses using CU's transformed materials (11 courses, 3 institutions, N ¼ 329 students) relative to courses using only traditional lecture (10 courses, 5 institutions, N ¼ 303 students) across seven institutions.to improved student learning, which makes it difficult to counsel faculty as to the importance of any one aspect of the transformed courses.Various lines of evidence indicate that tutorials are beneficial both to students [69] and to instructors [67], especially in supporting sophisticated problem-solving skills in the upper division.Otherwise, only student attitude data (see Table II) point to the impact of particular course elements on learning.
To explore student attitudes about the course transformations, we gave end-of-term surveys to students in our transformed courses.These surveys asked students on a Likert scale how useful each class component (e.g., clickers, lectures, tutorials, textbooks, etc.) was for their learning.The results of these surveys for 11 semesters of our transformed courses are summarized in Table II.Additionally, 90% of N ¼ 301 students responded that lecture using clickers was more useful to their learning than pure lecture.These data suggests that, at least at CU, student resistance to interactive engagement in upper-division courses is not a concern.

D. Impact on faculty
One major assumption of the SEI was that departmental faculty, through partnership with STFs on course transformation, would engage deeply in scientific teaching strategies, resulting in impacts on their instruction over time-fueling large-scale cultural shifts within departments.In order to determine whether this outcome was achieved, we analyze whether a significant fraction of departmental faculty were involved in the SEI, what types of instructional impacts are reported, whether and why faculty choose to use SEI materials, and whether faculty workload and time are deterrents to the use of SEI materials.

Process of faculty development
While course transformation was an important focus of the SEI, an underlying goal was to encourage and support faculty in the use of evidence-based teaching methods.In some departments, STFs might be working on up to three courses at once [68], resulting in engagement with many individual faculty over time-we find it typically takes about two years for a faculty member to become a consistent, effective user of active learning strategies.
In SEI Physics, while most course transformations were the result of focused work by STFs and PER faculty, in the semesters immediately following a course transformation the STF worked quite closely with instructors to help them implement the changes and reflect on progress.STF support tapered in subsequent semesters, with the assumption that increasing institutionalization of the reforms would reduce the necessity for additional instructor support (an assumption that was not borne out in practice).Additionally, in early iterations of the course transformations, NSF funds were available to support co-teaching by PER and non-PER instructors, which included and fully supported a non-PER, SEI-friendly faculty member in teaching and contributing to the transformed courses.Early interviews [67] suggested that these co-teaching experiences were powerful professional development experiences for those faculty.However, we also note that, as SEI involvement in courses was reduced, at least one of those faculty "regressed to the mean," returning to more traditional instruction [67].

Number of faculty impacted
Across all departments, 135 faculty members have modified their teaching in some way, representing 47% of teaching faculty.In the Physics Department, a moderate number of upper-division faculty have modified their teaching (20 faculty, or 34% of those teaching upper division) or made use of the STF (48%).
A useful gauge of departmental involvement is the number of faculty who made no use of the STF: In department D only 16% of faculty made no use of the STF, in contrast to physics (52% of upper division instructors made no use of STF) and department C (37% made no use of the STF).
We can also examine the percent of teaching faculty who partnered with an STF to transform a course (as opposed to consultations with the STF or contributing to a working group), and thus had significant opportunity for changing their teaching practice: The fraction of faculty involved directly in course transformation is greater in department D (11, or 44% of teaching faculty) than in the Physics Department (8, or 14% of faculty teaching upper division) or department C (8, or 19% of teaching faculty).An additional 17% of physics upper-division faculty worked closely with a STF as they taught with the transformed materials.
Together, these data suggest that the SEI model in department D resulted in reaching a broader fraction of the faculty than in either the Physics Department or department C.

Types of faculty impact
As with course changes, we calculated the average number of impacts on faculty in each department, as captured during interviews with SEI staff (see Sec. VA); see Fig. 7.The success of department D in reaching both a large fraction of faculty (25, or 84% of teaching faculty) and affecting a large number of practices for those faculty is shown by the higher average impact across all faculty.The converse can be seen for department C.But what kinds of changes were made by these faculty?As with courses, we cataloged whether faculty involvement included learning goals, assessments, or instructional changes (data not shown).Department D was particularly successful in impacting faculty's instruction (68% of teaching faculty have modified some aspect of their teaching, many of whom added clickers as part of those instructional changes).In the Physics Department, due to the high priority placed on having faculty give our conceptual assessments in their classrooms, greater impacts are observed in the "assessment" category.
Our online survey in 2014 (see Sec. VA 4) asked how the SEI may have affected teaching practices.About half (N ¼ 7 faculty out of 13 responding) indicated that their instruction was "significantly" impacted by the SEI materials, while 4 indicated that their instruction was impacted "a little bit"; the remaining 2 faculty did not use the materials at all.Instructors who had made some use of the SEI materials (N ¼ 11 out of 13), on average, "somewhat" agreed that the SEI project had affected their teaching in this course or other courses.Results from the physics survey are compared with earlier results from departments C and D in Table III.Faculty in SEI Physics report several changes to their instruction, which they plan to carry forward into other courses.
A caveat of the above analysis is that, as noted above, many active learning practices were more common in physics than in other departments at the start of the SEI, which could result in some undercounting of instructional change in physics (since only new instructional practices are cataloged in the data on the number of faculty changes).It is difficult to determine the extent of such undercounting, though three faculty responding to the survey commented that their instruction had not been greatly impacted by the SEI because they were already using such techniques prior to the SEI.Additionally, 4 of 9 faculty who used clickers as a result of the project indicated that they had used them before, and hence this was not a change in instructional methods for them, although this was their first time in using them in the upper division.
Overall, these results suggest that the impact of the SEI on physics faculty was positive, including the introduction of many techniques into the upper division that previously had only been used in the lower division.However, in department D the number of faculty impacted across the department, and the number of changes reported for each faculty member, was much higher than that in SEI Physics.These data support our qualitative observations, in which we have seen more sweeping changes in department Dwhere the SEI was more deeply integrated into departmental culture and resulted in ongoing faculty-led education initiatives-than in physics, where the SEI was treated as a separate PER project.However, it is difficult to gauge how the observed impacts in the Physics Department might have been different if, as in department D, there was no existing precedent of active learning.

Faculty decision whether to use the SEI materials
In SEI Physics, course sustainability followed a "if you build it, they will use it" model-making course archives available each semester to the faculty member assigned to teach the course, and relying on faculty involvement in earlier discussion of learning goals to generate a sense of FIG. 7. Average number of faculty changes in the three casestudy departments, with "PHYS-UD" reported as an average across faculty teaching upper-division courses only."Impact" is the total number of changes cataloged through interviews (see Sec. VA).Faculty impact has a maximum of 9. ownership in the project.As shown in Fig. 6, the results of this approach varied greatly each semester.We were interested to identify the factors influencing instructor use of the SEI materials in physics.One possible factor was seniority: we correlated the number of years employed at CU at the time of instruction with the number of materials used in the course package, for the most recent semester in which an instructor taught a SEI course.Consistent with earlier results by others [8], we found no relationship between seniority at the university and the use of the SEI materials.In some instances, experienced faculty adopted a wide variety of SEI materials, and in other cases, experienced faculty chose to use their own existing materials.Many new faculty chose to use the SEI materials and expressed gratitude for their existence.
Faculty who had chosen not to use the materials (N ¼ 2) were asked about their reasons for this choice.Both were veteran instructors of 15 þ years at the university, and both indicated that they had not used the materials because "I already had my own materials for this course," and "these (SEI) materials did not match with my teaching style or philosophy."Additionally, one instructor indicated that they take a different approach to the content that made it not feasible to use the SEI materials.Neither instructor indicated that time (either preparation or instructional time) was a factor in their decision, or that they did not feel that the SEI materials would be effective.It appears that these instructors were happy with their existing course approach, and saw no reason to change: The SEI offered a solution where there was no perceived problem, especially given the additional time that incorporation of new methods would require.There may even be negative attitudes towards course transformation among such faculty: Some faculty with traditional instructional approaches have previously voiced fears that, if they did not use the SEI materials, they would be given lower priority for teaching of these (desirable) courses, and felt that this would not be fair.

Workload and time
We know that "time" has been reported as a deterrent to using instructional changes [7,11], so those physics instructors who did use our materials were asked how their workload teaching with the transformed materials compared to that without the materials.For those instructors who had not taught that particular course previously (N ¼ 8), they estimated (on average) that their workload would have been "a little more" if they had not had access to the SEI prepared materials-in other words, that their teaching load decreased due to the SEI.(We had insufficient responses from instructors who had taught the course before to make determinations about how the SEI materials impacted their workload.)This perceived time savings may be a factor in faculty decisions to use readymade materials.
This reduction in preparation time in SEI Physics is in contrast to the results from the midway survey, in which faculty across all departments indicated that their teaching preparation time had increased due to SEI efforts: Roughly equal numbers felt that their course preparation time had increased (49) or stayed the same (44)-none felt that their course preparation time had decreased.Thus, the SEI Physics model, where faculty involvement in the transformations was more limited, has the benefit of reducing faculty time, but this lack of involvement likely has a negative impact on sustainability.

Summary
We have presented several lines of evidence-the number of faculty impacted, the average impacts across all faculty, and faculty self-reporting-suggesting that SEI Physics did impact faculty practice in the upper division.One outcome is that instructional practices that were preexisting at the introductory level (such as the use of clickers and tutorials) were successfully introduced into the upper division with significant impacts on student learning.
However, results in department D showed that changes in teaching methods of a much larger fraction of the faculty in a department, with potential long-term impacts on departmental culture, are possible within the SEI model.In the Physics Department, a smaller subset of faculty were invested in and impacted by the SEI and its outcomes.Some physics faculty saw the SEI materials as a time saver, rather than a time commitment, but this was not true of those who had existing materials and were not motivated by a commitment to active learning.Existing faculty attitudes towards active learning, due to the departmental history with course reform, could have led to a lack of serious consideration of the SEI materials by some experienced instructors.Given the changes that had taken place earlier in the Physics Department, one could argue that there was a lack of "low-hanging fruit" (in terms of faculty who might be interested in changing their teaching practices but had not yet done so) in the Physics Department compared to other departments.

F. Impact on departments and institutions
The goal of the SEI was for transformed courses to become the norm across multiple departments.At this point, this lofty goal has not been realized.Transformed courses are certainly common, but in most cases, including the CU Physics Department, there are no departmental expectations that instructors will use the SEI materials, and use is variable.
In the Physics Department, we do have evidence of steady, nonsystemic change in the department: The use of interactive techniques, particularly the use of clickers [45] in upper-division physics, has increased since the presence of the SEI.Students have also expressed dissatisfaction when encountering traditional instruction after experiencing a transformed course [45], resulting in some student-led pressure to use clickers and other interactive techniques in the upper division.Also, while focused material development and refinement occurred in the four main courses (Classical Mechanics, Electricity and Magnetism I and II, and Quantum I), a nontrivial effect of these transformations was the incorporation of some of these teaching methods in follow-up courses: Electricity and Magnetism II (later continued by the PER group), Quantum II, Classical Mechanics II, and Thermodynamics.In each of those courses, faculty individually developed materials without the assistance of a STF, or chose to use some of the materials created by other faculty members.We believe this activity in non-SEI courses is indicative of a larger culture shift in the department regarding the use of interactive techniques in upper-division courses.
Across all departments in the SEI, faculty report that their departmental culture has changed due to the SEI.In the SEI midway survey (in 2010), out of 97 faculty who were not new to the department, 75 (77%) indicated that the number of discussions around education and teaching had increased, and that these conversations occurred either monthly or weekly at the time of the survey.Additionally, 70 (72%) of those faculty indicated that the nature of those discussions had changed (presumably guided in a productive fashion by the SEI), and 88% of faculty respondents felt that the influence of the SEI on their department had been positive or very positive.As a tangible measure of this positive feeling towards the SEI, most faculty (104 out of 112; 8 neutral) indicated that they would support the continuation of a STF in the department if using external funds.While this number drops by approximately half (54 out of 108; 35 neutral) when asked if they would use department funds for this purpose, SEI Central was pleasantly surprised by this result, given the many serious needs for the use of limited departmental funds.Indeed, three departments have hired STFs into regular instructor positions.
Table III summarizes the data presented on departmental impacts in the three case studies, along with some of the contextual factors that we suggest are potentially important in influencing these outcomes based on our observations and the progression of the effort in each department.

VI. SUMMARY A. Viability of the SEI model
We have presented an experiment in large-scale change in undergraduate science teaching: the Science Education Initiative.Outcome data indicate that large numbers of courses, students, and faculty were impacted in positive ways by this departmentally focused model of change.Some evidence shows steady change in faculty practices and departmental norms across departments.Thus, we conclude that the SEI model has the potential to support institutional change, including within upper-division physics, though the ability to realize this potential depends largely on local factors within each department.The SEI model was later enhanced to include support mechanisms, such as greater STF training, departmental accountability, and faculty incentives, in order to address this local variation and the lack of institutional incentives for teaching.For more detail on these enhancements to the SEI model, see Ref. [70] as well as discussion by the National Research Council [71].

B. Outcomes in the Physics Department
The major focus of this paper is on the SEI model in upper-division physics at CU. High-quality, comprehensive, course materials were developed for core courses in upper-division physics, along with an extensive research base on student ideas, conceptual assessments for measuring student learning, and multiple publications that have advanced the field.A large fraction of our majors' courses have been the subject of transformation efforts, enabling our students to experience active learning throughout the major, from lower to upper division, with demonstrated impacts on student learning.The course materials have been used by instructors at CU and multiple institutions.Thus, we have demonstrated that instructional change and educational research in the upper division is feasible and productive, and that the SEI model of course transformation can be a powerful guide in this process.
However sustainability of these changes has been inconsistent, and we cannot consider any of our courses to be permanently "transformed."From the outset, the Physics Department had no expectation that a faculty member would use SEI-developed materials, and some faculty have resisted these transformations, and so use has been variable.However, the majority of faculty do choose to use many of the course materials, with most choosing to use clicker questions and tutorials, and many other essential elements.We see sustained student learning gains over time in courses using the SEI materials.
The SEI in the Physics Department reached a limited fraction of faculty (especially in comparison to more successful departments), considering either the department as a whole or only those teaching in the upper division.SEI Physics was seen more as a PER project, with postdoctoral fellows working with PER faculty rather than serving as a general departmental resource.Most faculty using our materials were those who were already interested in active learning or who did not have course materials from previous instruction-seniority was not a predicting factor of faculty use of transformed course materials.While the department did not undergo a "sea change" in their instructional practices, many faculty used instructional methods, such as clickers, for the first time in upper division, and we have some evidence that these practices are becoming the norm.Thus, overall the impact of the SEI on physics faculty was positive, but somewhat limited in scope.

C. Factors influencing outcomes in SEI Physics
Physics was unique among SEI-funded departments, as outlined in Table I, with positive and negative consequences.In order to shed light on contextual factors in SEI Physics, we have provided two contrasting case studies.The impacts and relevant background factors in each department are summarized in Table IV and discussed below.

Departmental history
Physics had a history of course reform, which led to preexisting opinions about educational transformation among faculty; some had decided a priori that they would not use SEI materials, whereas others were already friendly to such techniques.This existing experience with active learning was productive, since many faculty brought experience with and interest in these techniques into the upper division.However, some faculty also expressed skepticism about making interactive teaching the norm in the upper division.In departments C and D, most faculty had not yet formed strong opinions about educational The work from 2 of these courses were reused in two separate courses that use the same content.b Several courses in this department were transformed by faculty "champions" with little STF time, whereas the majority of STF time was spent on two courses which were the focus of the SEI.transformation.It was also possible for more sweeping changes to take hold in these departments, since they were starting from a more traditional place.

PER-developed courses
The PER expertise and dominance of the course transformation resulted in high-quality, extensive course materials, researched and validated assessments, welldocumented student learning outcomes, and rich contributions to the field.This existing expertise was largely lacking in the other departments.Faculty in the Physics Department cited fewer concerns about workload in teaching transformed courses than did faculty in other departments, likely a result of the PER team having done most of the timeconsuming development work.However, this structure also resulted in fewer faculty being involved in the development of the course and sometimes less invested in using the resulting course materials.

Faculty rotation
In the Physics Department, faculty rotate through courses quite frequently, which partly drove the PER-led model described above.In departments, such as C and D, where faculty teach the same course many times, it is easier to sustain course transformations than in the Physics Department where rotation between courses is more frequent.In the Physics Department we were able to learn a great deal about making transportable course archive packages, though maintenance of these archives is a challenge.How best to structure conversations with faculty rotating into these courses and provide ongoing support to these faculty over time is also an open, unsolved question.

Focus on the upper division
Work in the upper division leveraged faculty interest in these courses for majors, resulting in engaged faculty and productive discussions around learning goals for these courses.In contrast, several other departments choosing to focus on introductory courses (including department C) had difficultly gaining traction due to overloaded curricula, multiple decision makers, and low faculty investment.While work in the upper division may sometimes be a good catalyst for faculty engagement, this focus does have some compromises as well-limiting involvement to faculty who teach upper-division courses, impacting fewer students because of the smaller class sizes, and lowering the availability of prior work to guide our efforts.The existing PER expertise helped offset the lack of prior work.

Departmental leadership
Departmental leadership is one of the critical factors influencing outcomes across SEI departments.Where the SEI was marginalized by departmental leadership and not aligned with departmental priorities (department C), STFs were able to gain traction in just a few courses, working with a subset of interested faculty.However, where the SEI was well integrated into department structures and priorities (department D), high levels of impact were achieved, with some long-lasting effects on courses.In the Physics Department, the chair was supportive, but not as actively involved as in department D, and did not explicitly require faculty to use materials.The SEI department director in the Physics Department was also (eventually) a PER faculty member, which likely contributed to the SEI being seen as a PER-led project.
Together, these factors contribute towards the mixed, complex results that we have observed in SEI Physics in terms of (a) course sustainability, (b) impact on faculty practice, and (c) change in departmental culture.

D. Implications
This work has several implications for the field.One is that the SEI model can be effective at promoting course transformation, though outcomes depend strongly on local factors.Our work indicates that developers should consider local factors in a department before initiating change.What are faculty's attitudes towards teaching and learning?How does the Chair respond to educational efforts?What is the structure for faculty course assignments?How will course materials be archived, transmitted, and maintained?Is there an active PER or DBER community to support this work, and how is that community viewed and integrated within the department?Are faculty interested in upper-division course transformation, and is there leadership to support that interest, or are there other priorities drawing their attention?How do you leverage existing DBER experience and yet support investment from the broader faculty?
Perhaps the main lesson for upper-division developers is that we can expand upon our typical PER model of course transformation.Rather than working as a research partner with PER faculty, a postdoctoral fellow could partner with a broader pool of faculty, acting as a departmental resource and a coach, and thus as a change agent within the department at large.Such a shift in thinking also expands the possibilities of candidates for such a position (i.e., not restricted to those with PER degrees, interested in PER faculty positions).By working with individual faculty members, over time, based on their needs and interests, deep and lasting faculty change at the local level may be achieved.

ACKNOWLEDGMENTS
While it is impossible to thank all those involved in the SEI, we would explicitly like to thank the STFs, departmental directors, Chairs, and faculty at CU and UBC, across departments, for their engaged interaction and support of the SEI.We appreciate the support offered by various administrators at CU and UBC, without whom the SEI would not have been possible.Many thanks to

FIG. 1 .
FIG. 1.The SEI model and its intended impacts.Partnership between several players-driven by science teaching fellows (STFs) within departments-was aimed at both course transformation and faculty change.The strength of interactions between players is indicated by the weight of the connecting arrow.Course transformation was the explicit focus of the program; faculty development was equally important as a program goal, but implicit in the model.

FIG. 4 .
FIG.4.Average number of course changes in the three casestudy departments, with "PHYS-UD" reported as an average across upper-division courses only."Impact" is the total number of changes cataloged through interviews (see Sec. VA).Course impact has a maximum of 14.

FIG. 5 .
FIG. 5. Fidelity scores in the four transformed courses, over time, representing the total number of instructional changes included each semester.The first semester represents course development.Because of some missing data, not every instructional instance is represented-essentially anonymizing this data and so obscuring the identity of instructors.The first semester without STF support is indicated with a circle, and instruction by the developer is indicated with a yellow star.Course 4 has had continued postdoctoral support.

TABLE I .
Unique aspects of the CU SEI Physics, along with attendant consequences.
• Opportunity to impact many faculty • Resulted in well-organized course archives • Sustainability issues Focus on upper division • Opportunity to impact faculty who primarily teach upper division • High priority on courses by faculty • Individual instructor has control over entire course (single sections, few follow-up courses)

TABLE II .
Percentage of students who responded favorably or unfavorably to the statement that a particular component of the class was useful to their learning on an end-of-term survey.Total N does not include students who selected 'Not Applicable'.

TABLE III .
Responses to departmental surveys in 2010 (departments C and D) and 2014 (Physics Department, including 11 users and 2 nonusers of materials).

TABLE IV .
Summary of impacts across case study departments.Impacts include courses with full or partial STF involvement and faculty with some SEI involvement.