Curriculum development in studio-style university physics and implications for dissemination of research-based reforms

Over the past few decades, a growing body of evidence demonstrates that students learn best in engaging, interactive, collaborative, and inquiry-based environments. However, most college science classes are still taught with traditional methods suggesting the existing selection of research-based instructional materials has not widely transformed undergraduate education. SCALE-UP is a renovated pedagogy and classroom environment that has achieved a greater impact than most, used extensively throughout the United States and abroad. SCALE-UP is not a simple collection of lesson plans or a textbook that can be easily adopted, and instead instructors are encouraged to customize main pedagogical principles to their unique instructional situation. This flexibility along with promotion of instructor autonomy may have assisted its spread. This paper uses case studies of five successful secondary implementations in the United States to examine how instructors gather information about reform, create a curriculum, and achieve sustained use. Many people learned about research-based resources that formed the composite of their curricula through interpersonal connections. Time constraints and misunderstandings between developers and instructors limited which resources were chosen and how they were used. Once instructors created a “working form” of the curriculum, three out of five instructors did not make significant changes. This could lead to the preservation of a more conservative curriculum. Implications include that disseminators should articulate core principles of the reform that should be retained to uphold the integrity of the reform as well as the areas where adopters have flexibility to innovate. Strategically involving other secondary users in the dissemination process could facilitate important interpersonal exchanges that could provide an additional layer of support for faculty.

Over the past few decades, a growing body of evidence has demonstrated that students learn best in engaging, interactive, collaborative, and inquiry-based environments [1].While educational researchers have produced many educational reforms and collected evidence documenting their positive outcomes [2], traditional lecture is still the most common mode of instruction especially in science and engineering [3,4].Prestigious groups are increasingly expressing dissatisfaction with the rate of implementation, adoption, and spread of research-based instructional strategies [5][6][7][8].Educational researchers are finally starting to acknowledge that they should devote energy to understanding higher education change processes if they want existing innovative pedagogies to increase their impact.
While business, management, and communication fields have developed a wealth of theories to inform change efforts, these theoretical frameworks remain underutilized by STEM education leaders and researchers.Recently, researchers have conducted literature reviews and constructed summaries that attempt to capture higher education processes [9][10][11][12], however, very few researchers have applied these frameworks to understanding educational change efforts.While bringing theoretical frameworks to the higher education community is important, if we want users to adopt reformed based instructional practices, it is even more critical to examine change process in authentic higher education settings.

A. Introducing the SCALE-UP reform as an appropriate case study
To help fill this void in the literature, this paper describes how five secondary sites achieve sustained, widespread use of a radical, structural educational reform in higher education introductory physics courses.Investigating secondary reform efforts is still relatively new so it is important to describe how secondary adopters naturally attempt this process and what difficulties they face, so developers can structure reforms to eliminate these issues.This study will examine what motivates implementers to seek reform, how they collect information about the reform, how they put the reform into use and achieve sustained reform.
Since we aspire to promote successful instructional change, we chose a reform that has achieved a largerthan-average impact: Student Centered Active Learning Environment with Upside-down Pedagogy (SCALE-UP) [13].This reform modifies both the pedagogy and classroom environment to minimize lecture, promote interaction between students and instructors, and heighten student engagement.Students are seated in groups at special tables, round or other shapes that promote face-to-face interaction.Whiteboards on walls serve as public thinking spaces, projection capabilities share student work, and the lack of an obvious "front" of the classroom encourages the instructor to serve as a "guide on the side" instead of the primary information source.
A recent census found SCALE-UP inspired reforms are used in at least 314 departments at 189 higher education institutions in over 20 countries worldwide [14].Many suggest several hypotheses for why SCALE-UP has spread more extensively than most reforms.We hypothesize the redesigned classroom starts departmental conversations and stand as a physical symbol of change that can aid the reform's sustainability and spread.The role of the redesigned classroom is discussed elsewhere.
Here, we seek to investigate SCALE-UP's appeal to instructors from a curricular point of view.Dissemination literature has found that higher education faculty members prefer innovations that allow them to have some academic freedom and ownership of the curriculum [15].This provides a good balance between completely prescribed curriculum, which does not acknowledge instructors autonomy in the classroom, and an idea that is too abstract for practical significance.SCALE-UP falls under this category, which might have aided its extensive spread.It acknowledges and empowers faculty participation in the change process because it provides basic pedagogical goals instead of specifying a curriculum.This also means SCALE-UP can be, and has been, adapted for use in more than a dozen disciplines, in a variety of class sizes and institution types.This study seeks to explore this hypothesis by examining how instructors participate in the change process and create a curriculum for use in a SCALE-UP environment [16][17][18].SCALE-UP's pedagogy encourages instructors to (1) create a cooperative learning environment that encourages students to collaborate with their peers, questioning and teaching one another, (2) minimize lecture in favor of research-based activities and, (3) coach students to help them answer their own questions and let them present their results to the class for review.For example, SCALE-UP physics students may work on miniature laboratories with physical manipulatives, solve intriguing real-world problems, and use technology to visualize abstract phenomena.

B. Goals of this work
At the most basic level, this paper will provide practical advice and examples for people interested in adopting SCALE-UP at their own institution.It is important to provide examples of how other secondary sites reformed their curriculum to accompany the redesigned classroom because changing the learning environment will not automatically lead to learning gains and other positive outcomes.Specifically, Cummings et al. [19] conclude that without activities promoting student interaction and engagement, "the use of the studio format alone does not produce improvement in conceptual learning scores as compared to those measured on average in a traditionally structured course" (p.44), a finding that was echoed by Ithaca College's first efforts at SCALE-UP implementation [20].Thus, it is critical that SCALE-UP adopters reform their pedagogies and curriculum in addition to the classroom space and this paper will provide information about that process.
Furthermore, this study will lead to important theoretical insights for people interested in the dissemination of research-based materials and achieving widespread change.Examining why secondary users chose SCALE-UP then examining how to adapt the reform can help test the hypothesis that instructors prefer flexible curricula they can modify.Investigating how sites achieve sustained use can probe the relationship between adaptation and accomplishing satisfactory change.Monitoring the role of the developer and secondary users can lead to recommendations for disseminating customizable materials that allow faculty to productively participate in the reform process.

II. LITERATURE REVIEW
A. Choosing a framework While fields like management, higher education, and communication have been conducting studies to investigate change processes, this is a relatively new area of interest in STEM education and there is no widely agreed upon theoretical framework to view this process.Henderson et al. [21], characterized common theoretical frameworks based on a literature review of 191 relevant journal articles relevant to undergraduate STEM education.This literature review will help narrow down a framework for this study.
This work classified change strategies in four categories based on two criteria: (i) whether the strategy seeks to change either individual instructors or environments and structures and (ii) whether the change strategy has prescribed or emergent outcomes.The four main categories of change strategies are presented in Table I below.
Characterizing SCALE-UP within these categories is not straightforward.At first glance, SCALE-UP would appear to fall under the "environment and structures category" because it requires a reformed classroom and structural changes that are usually beyond the power of an individual instructor.However, once instructors have access to a reformed space, changing the pedagogy and curriculum usually becomes an individual change effort.In all of the cases examined in this work, individual faculty had autonomy in creating curriculum, with little to no interference from higher administration.Thus, we will focus on change processes that affect individuals, shown in the first row in Table I.Applying the second criterion is easier: the SCALE-UP reform prescribes intended outcomes, for example, minimizing lecture, increasing interaction, increasing real world problem solving, etc.Thus, the pedagogical aspect of using SCALE-UP lands in the first quadrant of Table I: disseminating curriculum and pedagogy.
With this characterization in mind, we can bring in a basic framework for understanding the change process.The literature review recommends using the diffusion model, which focuses on distributing a quality product to as many individuals as possible.It also describes how most change agents tackle undergraduate STEM reform [22].Under this model, STEM change agents create highly structured and specific interventions that can be easily used by others, under the assumption that faculty have limited time and expertise to develop innovative teaching strategies.In the past, STEM-based change agents might have relied on mass-market techniques (e.g., publishing articles, giving talks) to distribute their innovation, however, mounting evidence suggests that the most successful diffusion occurs through interpersonal interactions [23,24].

B. Three stage change process
Diffusion of innovations is a robust theory that allows for some influence from the system, lots of adaptation, phased adoption, and many other aspects of change.Models of a dissemination-based change process [25][26][27] usually involve at least three stages to move an individual toward changing their behavior.We will summarize three basic steps of the diffusion model, with details discussed as relevant.Our research questions will probe each stage of the process as shown in Table II.

Step 1: Motivation to change
The first step to changing one's approach to teaching involves looking at instructional practices from a new perspective or understanding [28].To motivate implementers to invest the effort to change, policy makers must create a sense of dissonance in which agents see problems with  current instruction until they are willing to supplant or fundamentally change current practice.Once agents recognize problems with the existing model then they can restructure existing beliefs and knowledge to make sense of the new idea [29].
It takes effort to change so people need to believe that change is worth the effort.This shift in mindset can begin when instructors realize that their students are not learning as much as expected.For example, the field of physics education research (PER) gained momentum when instructors began administrating the Force Concept Inventory [30] to their students, revealing their students often did not understand basic concepts that instructors believed had been sufficiently covered in class.
Rogers' Diffusion of Innovations framework also describes different characteristics of adopters based on when they adopt the innovation.The "early adopters" tend to be more adventurous, well educated and well connected to the research community and are attracted by the idea of trying something new.In order for the reform to spread to the early majority, it must appeal to the "early majority" with different characteristics.This population is more hesitant, more influenced by peer pressure and they want to adopt a reform that works.It might require more effort and different marketing strategies to convince this population to change.
Sometimes the departmental and institutional culture can discourage instructors from spending energy improving their instruction.At some institutions, scientists' professional identities are linked more tightly to research over teaching [31] and this can be "an invisible and underappreciated barrier to undergraduate science teaching reform" (Ref.[31], p. 339).A position that prioritizes research responsibilities can cause professors to be reluctant to spend time and effort improving their teaching, especially using instructional strategies that deviate from the way they learned science.Thus, it is essential for departments and/or institutions to encourage and reward (or at least, not penalize) instructors for investing this effort.Our study will investigate what motivates instructors to change and whether the department or institution affects this process.
Step 2: Gathering information During the next step, implementers learn about the existence of the innovation and seek further information that includes three basic kinds of knowledge.First, respondents need "awareness knowledge" that the instructional strategy exists.Then, they can gather "how-to knowledge" about how to use the strategy properly.This is important because even though faculty might be experts in their discipline, they may not have expertise in instructional reform [32].This type of knowledge is especially important for multifaceted, complex innovations like SCALE-UP.Finally, users need "principles knowledge" about why the strategy works.Without principles knowledge, instructors may interpret a reform as prescribing certain activities without a coherent idea of the underlying rationale.This may especially affect radical reform efforts like science-byinquiry, student-centered classroom and SCALE-UP, which intend to revise not only classroom activities but also the cognitive practices of scientific reasoning and the social interactions of learners and their instructors [33,34].With complex reforms like these, developers may need to provide more support.
"Such reform cannot be accomplished by having teachers learn only the surface form of reform practices.It requires grappling with the underlying ideas and may require deep conceptual change, in which teachers rethink an entire system of interacting attitudes, beliefs, and practices" (Ref.[7], p. 417).
Principles knowledge helps users overcome any challenges that may arise because it requires a higher order of thinking about the implementation process than just replicating what the designer did.
All three kinds of knowledge can help achieve successful implementation so this study will probe where implementers received information about the reform and what type of support they sought.Also, familiarity with PER and innovative teaching strategies can be linked to possessing deeper principles knowledge so we will discuss what instruction looked like before SCALE-UP to see if instructors had awareness, how to, and principles knowledge about the composite instructional strategies.
Step 3: Implement the reform During implementation, an innovation is put into practice.An implementer may run into challenges at this stage and need technical assistance from change agents and others to reduce uncertainty about using the reform.
Reinvention also occurs as the reform is put into practice, which Rogers defines as "the degree to which an innovation is changed or modified by a user in the process of its adoption and implementation" (Ref.[27], p. 180).Rogers claims the more reinvention takes place, the more rapidly an innovation is adopted and becomes institutionalized because it allows the user to take ownership of the innovation.Since reinvention seems key to achieving sustained and widespread use, we want to study this process in more detail.
Fortunately, Henderson and Dancy provided a tool to describe the degree of modification to research-based materials based on the involvement of the external change agent and the instructor.This will allow us to characterize how much reinvention occurs when faculty integrate other research-based tools into the SCALE-UP curriculum.

C. Characterizing the degree of reinvention:
Adoption-invention continuum This continuum spans four categories, from an adoption pole to a reinvention pole, as shown in Fig. 1.At the lefthand extreme, the change agent develops all the materials and procedures for "as is" adoption by the instructor.At the opposite extreme of invention, the instructor may be inspired by the idea of an existing product but basically develops everything with minimal external influence.In between, for adaption and reinvention, an external idea influences the instructor but they develop important aspects of the strategy independently, usually without the developer's assistance.The resulting instructor-developed strategies may or may not be in line with research-based practices from the literature [35][36][37][38][39][40].
This continuum will help identify how research-based materials are adapted and contribute some insight to why.For example, in some cases, the degree of adaption may depend on the nature of the innovation itself.Some tools were designed for immediate classroom use and are unlikely to be modified, such as assessment instruments like the Force Concept Inventory.In contrast, pure adoption is not possible for other materials and reforms.For example, Physlets [41] was developed to be a flexible technological resource for classroom demonstrations, designed for use according to the instructor's preferences.SCALE-UP also falls under this category-it outlines main pedagogical goals but leaves the choice of content and structuring of activities up to the instructor.While the nature of the innovation can influence whether a reform can be strictly adopted, the instructor's background, perceptions about the situation, and instructional goals can introduce further modifications.This study intends to probe how secondary users handle this flexibility.
To summarize, both the three-step change process framework and the adoption-innovation spectrum will influence the analysis.The change process framework will be used to understand the steps that lead up to implementation and guide the research questions (as shown in Table I).This study will investigate what motivates instructors to change their instruction, how they chose composite research-based materials and adapt these in use and how they achieve sustained use.The adoptioninvention continuum will be brought in as an additional tool when thinking about the interaction between the user and materials during the implementation stage.

A. Data collection
Since we want to understand how people navigate the curriculum development process effectively, we examined "successful" secondary sites that had a direct link to the SCALE-UP developer and development site (Robert Beichner at North Carolina State University).
We defined "successful" as have achieving sustained use of the reform and 100% usage in introductory physics courses.We focus on calculus-based first semester mechanics of the introductory physics sequence because this course can be compared across institutions and many research-based materials are available for this course.
All of these institutions have dedicated studio spaces, with 7' round tables (seating nine students per table for a total of 45-117 students), whiteboards on walls, projection capabilities so there is no obvious front of the classroom and students have access to computers.Although some institutions use different acronyms for their implementations, for this study, all reform efforts will be referred to as "SCALE-UP" to preserve anonymity.Additionally, all of the interviewees will be referred to with male pronouns to further protect anonymity.Interviewees will be referred to with a letter to describe their institutional association and a number to distinguish between interviewees.
These institutions were also selected because they span the adoption-invention continuum.As shown in Table III, the institutions toward the left (Institutions A and B) attempted to replicate the reform as closely as they could while those to the right (Institutions D and E) purposefully added and modified significant elements.Also shown in Table II, our sample includes instructors who have and have FIG. 1. Adoption-invention continuum (Ref.[15]).
not formally studied physics education research prior to initiating the change.Thus, faculty involved in this study represent likely adopters of reform: both people who are actively involved in the PER community as well as more "typical" higher education professors who have an interest in improving instruction but no formal background in physics education research.Table III also includes some relevant information about when SCALE-UP was adopted and the Carnegie Classification [42] of the school.The amount of research activity can give some indication about how much time faculty have to devote to teaching and the percentage of undergraduates at the institution can indicate whether graduate students are available to serve as teaching assistants in the classroom.
Since this study primarily concerns the process of curriculum development, we focus on the initial implementation of the reform when it first came to the physics department at an institution.Thus, we interviewed "key implementers" who were heavily involved in bringing SCALE-UP to the university and developing curriculum to teach the first courses.These curricular packages developed by our interviewees were offered as a guide to future instructors so they significantly influenced the reform at each institution.We do not focus on how subsequent instructors modify this curricular package because that adds an unnecessary layer of complexity and we want to focus on how the "pioneers" of SCALE-UP at an institution develop the first working curriculum.
We interviewed at least two people (usually faculty and occasionally an administrator) at each institution, with the most knowledgeable representative interviewed at least twice.The open-ended interviews lasted an hour or more and the first session gathered information about the general initiation story; for example, where did you hear about SCALE-UP?Why did you decide to try SCALE-UP?How familiar are you with physics education research?What did your instruction look like before and after reform?
The second interview was conducted after the completion of class observations and review of course materials to gain a more thorough understanding.Questions focused on more specific details of curriculum development and concerned observations from the other data sources.For example, what resources did you consult?How did you choose a textbook?Did you consult PER resources?Which ones?Why or why not?
In addition to interviews, we collected course materials from the key initiator, including syllabi, daily Power Points, lesson plans, laboratory guidelines, quizzes, and exams.These were the materials that got passed to subsequent instructors, who could choose to use and modify them as they wished.As an additional reminder, this study does not examine what happens as the reform spreads within a department to these subsequent instructors.We also reviewed additional materials generated by the institution, including any related research literature, final reports to funding agencies and any relevant information on the Internet.
Site visits incorporating between two and six classroom observations were conducted at each institution.Often, the key implementer was no longer teaching the course so only subsequent instructors could be observed.Initially, we tried videotaping the class but too much activity in the large classroom made it difficult to create an accurate, comprehensive record.Instead, the observer took detailed, minuteby-minute notes on class activities, instructor, and student engagement.
Since the primary focus of this study is how the first key initiators developed a curriculum, the class observations were not used to detail subsequent changes made by secondary instructors.Instead, class observations were used to gain a deeper understanding of the SCALE-UP implementation and formed a basis for discussion during interviews.In addition to guiding the discussion, the triangulation of data sources verified statements made during interviews.To further verify and validate results, summary reports regarding the implementation effort at each institution were sent to the interviewees for further verification of accuracy and completeness.Additionally, each interviewee was sent a draft of this manuscript for approval.

B. Analysis
All data from an institution were combined to create a coherent representation of the reform effort at an institution and major initiation events were summarized in a timeline.Course materials, publications, and interviews were used to create a list of composite curricular influences.The extent of modification was then classified on the adoptioninnovation spectrum based on interviewee responses to clarifying questions and verified by class observations and course documents.These summary documents were supplemented with qualitative information from interviews to create an exploratory model surrounding the process of curriculum development.First, the researcher highlighted key statements from each interview.These were used to identify common threads, including when different interviewees expressed similar ideas.For example, we noticed several instructors often wanted to use preexisting activities but quickly abandoned this idea if they could not use materials immediately.Next, the researcher applied information from the literature and theoretical frameworks to come up with explanatory models.Results from this level of analysis and relevant literature will be presented in the discussion section, broadly organized by research question.

IV. RESULTS
A. What was the perceived problem with prior instruction?How familiar was the implementer with educational reform and research-based instructional strategies?(RQ1) In the five case studies examined here, reforms were initiated for a variety of reasons, primarily because instructors were self-motivated to improve instruction.Specifically, instructors at institutions A and B were not overly concerned with the current status of instruction but felt like trying something new to increase the engagement of students.Instructors at institution C and E had extensive exposure to physics education research and knew that integrating research-based strategies could improve the learning experience for their students, even though there was nothing drastically wrong with the status quo.Only Institution D initiated the change in response to low attendance and high failure rates.More details are provided below.

Institution A
Before SCALE-UP, Instructor A1, who had no formal PER background, had taught algebra-based physics using lecture supplemented with clicker questions and modeling problem solving at the board.He wanted to increase engagement, recalling, "my motivation was that students need to do more to construct their own learning."

Institution C
Instructor C1 had participated in and helped facilitate a reformed university physics teaching seminar for eight years prior to coming to Institution C. Before SCALE-UP, instructor C1 had been incorporating Peer Instruction [43], Think-Pair-Share [44], and University of Minnesota Context-Rich Problems (CRPs) [45,46] in accordance to research recommendations.When he learned about SCALE-UP, he felt having a more hands-on physics course where the students would share their thinking with the class resonated with the mission of the institution.He believed the reformed room could better facilitate the activities he was already doing and that studio instruction could improve the efficiency of faculty time.Specifically, in his primarily undergraduate institution, a faculty member had to run the 24-student lab sessions.Switching to an integrated lecture lab in a 99-student room actually reduced faculty time in a way that provided additional continuity without harming the student experience.

Institution D
Instructor D1 had been teaching the mandatory second semester of the introductory physics sequence and he was frustrated with the high failure rates (∼10%-15%) and low attendance (∼40%).He felt like he "mastered" lecturing but realized that it did not matter how well he presented the material if the students did not show up for class.He claimed SCALE-UP looked like a "better" way to teach and thought it could encourage the students to come to class without having to explicitly mandate it.As a well-respected physicist and senior faculty member, he found it relatively easy to gain support from administrators because they also wanted to reduce high failure rates.Colleagues were interested because studio instruction would reintroduce a laboratory component to introductory physics that had been missing since the 1970s.Instructor D2 was recruited to help design and teach the pilot sections.He had years of experience teaching a small, experimental physics course so he had witnessed the benefits of collaborative learning and interactive instruction.

Institution E
Instructors E1 and E2 had both received degrees in physics education research and been exposed to studiostyle instruction during graduate school.From the beginning, they wanted to bring studio instruction to Institution E, but as lecturers, both knew they could not implement these methods without an influential supporter in their department.Instructor E1 explained the help of someone higher up was necessary to make physical alterations to the room and dramatically change the pedagogy especially since physics courses were running "good enough" although interviewee E3 describes that there was room for improvement.Before SCALE-UP, physics "lectures were all running on an independent track each taught by a different person who was doing their own thing, recitations where TAs are droning at the blackboard, lecture, traditional labs."Although dropout rates were not a major concern, there was a general sense of discontentment about the state of the courses: students were frustrated about having to complete "cookbook labs" and conceptual gains were not as high as they could be.
When the department head (E3) became interested in PER, spent his sabbatical reading the research literature and wanted to support SCALE-UP, the reform could finally become a reality.

B. How do implementers learn about and gather
information about the reform?(RQ2) All the institutions learned about SCALE-UP through interpersonal interactions, including direct contact with the SCALE-UP developer (which was part of the selection criteria).Institutions A, B, and C learned about SCALE-UP during talks or workshops during national meetings.Instructors from Institution D learned about it through collaboration on another PER-related project.Instructors from Institution E were familiar with the reform from their graduate studies but could not proceed with reform until their department head became familiar with PER.More details are provided below.

Institution A
Instructor A1 first heard about SCALE-UP when its developer spoke at a national meeting.He was so impressed at the exciting new technique to engage his students that he invited the SCALE-UP developer to present a workshop on campus.Before implementation, the faculty member visited NC State to observe SCALE-UP classes.He directly interacted with Beichner to discuss possible textbooks and other details about implementation, including classroom design, structuring groups, etc.

Institution B
At Institution B, two interested faculty members (B1 and B2) became interested in SCALE-UP when they heard Beichner speak, one at an off-campus workshop and the other at an on-campus seminar.They decided to visit NC State to observe SCALE-UP classes and meet with local experts.These faculty members had no official background in PER, apart from reading literature and hearing PER researchers speak on campus (developers of Peer Instruction and University of Washington Tutorials [47]).After bringing one of Beichner's graduate students to campus multiple times for consultation and faculty training, the pilot classroom (four 7-foot tables that sat 36 students) was ready for use starting in Fall 2009.Through informal conversations with research colleagues, they were able to connect with other SCALE-UP secondary users and get ideas for a few isolated activities.

Institution C
After starting a new faculty job at Institution C, Instructor C1 learned about SCALE-UP when he attended the National Science Foundation New Faculty workshop in 2003.Instructor C1 proposed adopting the reform during the college's 5-year planning assessment and found administrators to be enthusiastically receptive.Administrators at Institution C invited the developer to campus to speak, ran a pilot class in 2005 (where students had access to some data collection equipment and laptops) and began using a fullscale (99 student) studio classroom in 2006.
Originally, Instructor C1 considered using resources from NC State but the server of the SCALE-UP database crashed during implementation which eliminated this option.He recalls, "We were expecting a guidebook so that Bob could say, 'here is SCALE-UP, now go'.At the time, and I think still, there's no guidebook."Although he knew enough about the reform not to expect a Workshop Physics [48] curricula or a specific textbook, he still expected more structure.He recalls that it took three years of conversations with the developer before he realized how "loose" the reform is.When other potential SCALE-UP sites ask Instructor C1 for advice, he emphasizes, "SCALE-UP is not a curriculum, it is a mode of instruction that integrates lecture, lab, and recitation that allows you to take advantage of these other approaches."He elaborates that his implementation takes advantage of research-based materials by "using more think-pair-share, more tutorials but the core of it is cooperative group problem solving."He was familiar with these other innovations from his work with the teaching seminar before implementing SCALE-UP.

Institution D
Instructor D1 first learned about SCALE-UP when he visited North Carolina State University while collaborating on a different research project.This project was his first foray into PER and Instructor D1 still claims that he's not an expert in the field.In the interview, he declared, "I am not a PER person."Before SCALE-UP, Instructor D1 used Peer Instruction in lecture hall, potentially because his colleague D2 (a permanent lecturer) had been collaborating with the developer of Peer Instruction on another project.Instructor D2 had a history working with an experimental, activity-based elective physics course on campus.They convinced administrators to build a classroom and ran two, off-term trial sections in 2001 and 2002.
Although instructors from Institution D admit NC State strongly influenced the classroom design, they always planned to create their own content to fit the student, faculty, and institutional culture.They wanted to design materials "that made it worthwhile to make students come to class and feel like they were learning and at the same time, make it easy for faculty to teach in this environment" (according to Instructor D2) in anticipation of student and faculty pushback.

Institution E
Since instructors E1 and E2 already knew a lot about studio instruction from graduate school, they only consulted the SCALE-UP webpage for information on room design but did most planning independently.To prepare for SCALE-UP implementation, the department head worked with E1 to move course instructors toward a common curriculum and they transformed the recitations to Minnesota-style cooperative group problem solving in 2009.The next summer, funds were secured to convert an old laboratory into a 45-person SCALE-UP classroom where instructors E1 and E2 co-taught a section of calculus-based physics in Fall 2010.
C. Where do curricular materials come from?What are the roles of the instructor and the change agent in the implementation process?How do the instructor and the change agent participate in the implementation process?(RQ3) Institutions A and B initially aimed to replicate NCSU SCALE-UP as closely as possible but abandoned that goal when using existing resources was problematic.Institutions C and E had instructors with formal training in PER and felt confident combining different research-based innovations according to their own expertise.Instructors from Institution D wanted to create their own curriculum from the beginning so it would satisfy the student, faculty, and institutional culture and minimize potential resistance.More details on each effort are given below.
In all of the implementations studied here, instructors incorporated other research-based materials into their SCALE-UP implementations.We will consider the other reforms "composite research-based reforms" because they influence the SCALE-UP curriculum that secondary users developed.Table IV below summarizes (i) the curricular influences, (ii) source of information, and (iii) extent of adaption for the development site and the five case studies examined here.Looking at the letters, notice that very few institutions learn about composite reforms from reading the literature and tend to learn about them through connections with colleagues.Looking at the colors, notice none of the course materials were directly adopted, with the exception of Electromagnetic Visualizations created by Instructor D1 at his institution.Looking at the underline, you can see that most institution had incorporated smaller research-based changes prior to SCALE-UP that they incorporated into the studio reform.

Institution A
When Instructor A1 switched to SCALE-UP, he intended to follow the NCSU model as closely as possible; "My initial goal was to clone what you did at NC State.It was kind of tried and true… so I wanted to pick it up and use it here."The developer handed him the collection of the activities used at NCSU but he found that it was hard to sort through and directly implement in his classroom.He remarks, "I found that taking NC State SCALE-UP and plunking it here wasn't that straightforward because things were in this disarray.So I personally ended up using very little of Beichner's stuff because it was too hard to sift through."Furthermore, Beichner dissuaded instructor A1 from simultaneously making the switch to the Matter & Interactions [49] (M&I) textbook, which is the basis for most NC State activities.
Instructor A1 independently found and chose Knight's textbook [50], because it presented the material well and found its workbook problems could be easily adapted to in-class exercises.When looking at the Knight textbook and workbook [51], Instructor A1 found "things were so TABLE IV.Composite research-based reforms for SCALE-UP implementation.The letter denotes the source of information: E (educational experience in graduate school, etc.), C (colleague from within or outside the department), and L (literature).Underline indicates that it was preexisting in introductory courses.Color denotes position on adoption-innovation spectrum, as shown in the legend below.compelling, well organized and problems were good" and he found that he could modify questions from the Knight workbook into the SCALE-UP format of prereading quizzes, clicker questions, and homework problems with minimal changes.
To develop curriculum, he had to come up with new slides, modify laboratories, and coordinate class activities.Previous semester's slides "had too much content to be useful," except for good sample problems or clicker questions.The instructor also had "to take away the cookbook labs we had and introduce a little bit more of the sort of openended labs."During the first semester of SCALE-UP, he tried a VPython [52] activity that the students enjoyed but never had time to do computational activities again.He felt that his implementation was short on hands-on activities since he lacked creativity to develop them.

Institution B
Instructors B1 and B2 claim their SCALE-UP implementation started off "really orthodox, the way things were done at NC State", especially since one of Beichner's students came to help with the first implementation on campus.Over several years, especially when their colleagues started teaching it, some of the formality "was dispensed with-the groups, the nametags, roles, we at least assigned them and used them to different degree, mostly because of laziness but also because the outcomes didn't seem worth the effort," according to instructor B1.
When choosing a textbook, these faculty members considered M&I since NC State was using it but Instructor B1 remembers that they "decided it was a little too radical and it would make it hard to switch back and forth between the two classes, maybe a little too different for us."The instructors decided on Understanding Physics [53], as "a compromise between the very radical M&I and traditional and it did come out of the PER community so there was a sense that this is written with a PER sense in mind."Later, this book was abandoned for its lack of mathematical rigor and the instructors tried two (more traditional) textbooks since.
Instructors B1 and B2 worked together to create their own curriculum, alternating between one writing the slides and the other editing.They felt they had to "start from scratch" and do a "complete restructuring" from previous teaching to come up with SCALE-UP activities and coordinate with the new textbook.They abandoned traditional Pasco labs and tried to invent labs that were simpler in terms of procedures and equipment.However, the instructors never developed a full sequence of experiments so they removed the integrated lab and are still working to create more reformed lab activities.

Institution C
Since Instructor C1 had been using research-based instructional materials in previous courses, he found it easy to move to a room that better facilitated what he was already doing.The scheduling switched from three 50-min lecture sections and a 75-min recitation to three 2-h meetings.To fill the time, he expanded the CRPs to fill longer sessions, for example, increasing the difficulty by providing more distractors and excess information.
At Institution C, calculus-based physics has a separate, formal laboratory to refine experimental skills so he created hands-on activities to use during class that did not replace the lab.These activities allowed students to "explore phenomena before instructor discussion and explanation.An example of these activities is having students use carts and tracks to explore the effects of collisions… followed by Think-Pair-Share questions using clickers."Some of the activities were found on the Internet and Instructor C1 developed others himself.

Institution D
Institution D claimed they wanted to develop a reform that fit their university, faculty, and student culture so they never attempted to replicate the NCSU reform beyond the room design.Thus, after building the classroom, interactions with the developer were minimal and mostly included conversations about implementation difficulties that were not related to the curriculum (for example, problems with students working in groups).
The instructors began by developing content.They wrote a book that students could access online for free and they designed 19 experiments.Some experiments were standard from lab equipment manuals while others were adapted from the experimental course that Instructor D2 helped teach in prior years.For example, the students in the experimental group typically built the apparatus themselves but this step was eliminated for SCALE-UP because of time constraints and the large class size.Originally, the reform started with a mix of lecture and clicker questions, which were developed in-house, inspired by Mazur's ConcepTests.Instructor D1 worked on integrating the visualizations he developed, which were Java applets which could be used as is in the course.

Institution E
Even though instructors E1 and E2 had "best case scenario" of extensive backgrounds in studio instruction and familiarity with a variety research-based materials, they claim it took a surprising amount of time and effort to create a SCALE-UP curriculum.Planning each day was a collaborative effort, where both instructors gave suggestions and made modifications to create something better than they felt they could have created individually.They borrowed almost all of their materials, except one previously developed lab, from external sources.The tests and textbook (Halliday, Resnick and Walker textbook [54]) in SCALE-UP were the same as those in the traditional section so students could seamlessly switch delivery modes for the second semester course.They modified preexisting labs to be more open ended and incorporated Peer Instruction and CRP problem solving, like the reformed traditional sections.Instructor E1 estimates 10%-20% of the material was incorporated from studio activities instructor E1 remembers from graduate school, without any major modifications.He consulted the SCALE-UP website primarily for specifics about room design, not ideas for activities.Instructor E2 incorporated a smaller percentage of lab activities from his graduate school studio experience but modified them to provide more elaborate instruction and eliminate specialized equipment.They assigned a small number of modified Knight workbook exercises to fill extra time.

D. After coming up with an initial curriculum, what
changes are made to subsequent versions and why?How do sites achieve sustained use? (RQ4) 3 out of the 5 institutions only made minor changes to instruction after the first successful pilot section of the course.On the other hand, Institution D purposefully started with a relatively conservative initial reform effort then continued to refine the innovation in subsequent semesters.In contrast, Institution E made dramatic changes to the pilot SCALE-UP structure because they wanted to spread the benefits of studio instruction to all their students which required creativity because of limited classroom space.Since all students could not use the studio classroom full time, they developed a mixed lecture-studio model that could accommodate all introductory physics students.

Institution A
After a successful pilot course, instructor A1 felt satisfied that he achieved an active class using modified Knight materials, and did not search extensively for more resources."I didn't have the time to go and seek out research-developed resources….I look at the slides each year but a more external search, like looking through the PER literature for tangibles or class activities, I never did….not by design, by laziness." Instructor A1 has not taught the course recently but SCALE-UP has been going strong in his department and he passes his notes to new instructors.After the pilot, the classroom doubled to fit 54 to give students more space, especially for experiments.In 2013, all introductory classes switched to a SCALE-UP format, almost naturally, without even an official faculty vote since there were enough instructors interested in teaching this way, space was available, and data verified the value of this methodology.

Institution B
The pilot at Institution B worked well with regular and honors students so they expanded their classroom to accommodate 90 students and continued to use a similar curriculum.Now all introductory calculus-based physics classes are taught with SCALE-UP.They discussed the possibility of eventually moving to the Matter and Interactions textbook but the instructors do not anticipate that will happen in the near future.They do hope to reincorporate a reformed lab once they create more lab activities.

Institution C
After Instructor C1 taught SCALE-UP once, he did not make major modifications to the curriculum.Textbooks were chosen to be consistent with the rest of instructors for the introductory physics course sequence, which had a slight impact.In 2007, Instructor C1 slightly modified how he administered reading quizzes after reading a research paper [55].
Overall, the instructor found his history with interactive teaching, autonomy to develop the course independently, and small class sizes (30-40 student in calculus based) made transitioning to SCALE-UP easy.Learning gains consistently met the instructor's expectations based on national benchmarks suggested by Hake's study [56].Converting the algebra-based physics courses took multiple iterations and more dramatic curricular changes before they could achieve satisfactory learning gains, potentially because of challenges integrating the lab component and dealing with a different student demographic.

Institution D
In contrast to other institutions, Institution D continued to make modifications every year to increase interactivity.Data collected from SCALE-UP pilot sections indicated they improved attendance, learning gains, reduced failure rates and students were generally happy.In 2003, they converted the on-term sequence to this format, which was met with significant resistance from students.Instructor D1 fought most of the political battles and Instructor D3 (a new tenure-track faculty member) took over managing the course.Instructor D3 introduced board problems and initially groups tried to solve them as a table of 9, which meant some people stood around with nothing to do.He realized that raising the projection screens could dramatically increase board space so students were more actively engaged, working in groups of three.
Instructor D2 began enlisting undergraduates to work as Teaching Assistants (TAs).He assigned each TA to focus on helping few tables so they could build familiarity with the students, act as role models (especially to underrepresented students) and proactively reduce potential problem situations.This helped mitigate student resistance.The instructors also reduced the number of experiments and chose activities that were "not so much about data-taking but about learning the concepts behind whatever the phenomena was" (Instructor D2).In 2014, they integrated resources from their Massive Online Open Course into their in-person class to improve reading quizzes and provide additional reference materials.Surveys and course evaluations revealed the students enjoyed these additional resources.

Institution E
After a couple successful semesters, in 2013, the department head decided to switch a mixed Lecture-Studio model so instead of "one third of the students with 100% SCALE UP style, we would have 100% of the students with two-thirds studio style."Specifically, students attend two 50-min lectures per week (which include clicker questions but are largely traditional) and two 110-min studio sessions per week, where students do active learning, cooperative group problem solving, and work in small groups on lab experiments, interactive computer simulations of physical phenomena, and scientific modeling.
The mixed model secured additional faculty buy-in because the teaching workload became equivalent to a regular course and saved on departmental expenses.Switching to the lecture-studio model involved a new textbook (Knight primarily for its presentation of material and the convenience of using Mastering Physics, not because of the workbook) but 80%-90% of the SCALE-UP material carried over.They had to modify the timing of the SCALE-UP activities to fit the new schedule and some had to be eliminated to make room for some modern physic topics but, generally, SCALE-UP activities could seamlessly be moved into studio.
According to E3, now with the lecture-studio model, "the pedagogy is established and that's the way we do things and we're not going back."Since there is not enough space on campus to create enough studios for all students, at least with this mixed mode, SCALE-UP interactions and activities are institutionalized for the foreseeable future.

V. CONCLUSION A. What motivates these instructors to change?
Research question 1 reveals the instructors in this sample possess many characteristics of Roger's early adopters.In four of these cases, instructors decided to adopt SCALE-UP because they wanted to try something new in the classroom whereas only Institution D was motivated to change because of a problem (high DFW rates and low attendance).All of these reform initiators had personal contact with the reform developer, found his argument compelling, and the data he showed impressive.These instructors willingly invested the time and effort to change without external incentives.Instructors C and E specifically mentioned that release time would have helped while developing the course but they were willing to endure the extra work because they believed in the cause.
All these instructors are the best-case scenarios of intrinsically motivated educators who are willing to go beyond job requirements to initiate a change.All of them had support from administration to build a classroom and try this.Once a classroom was built, these instructors usually had the autonomy to make changes as they saw fit, with the exception of choosing a textbook, in some cases.The department head was only actively involved and provided leadership in the reform at Institution E; in the other four, it was a bottom-up effort from faculty.
While it is promising that some instructors are willing to initiate a change effort without external incentives, these instructors may be more the exception rather than the norm.This may not be enough to motivate subsequent adopters, especially as the reform is moved into the mainstream.Diffusion of Innovation theory describes characteristics of adopters based on their position in an adoption curve.The first 16% of adopters, Rogers characterizes as early adopters who tend to be social and respond to the newness of an idea so they are willing to take risks and are not deterred by things that may not work.The interviews revealed that all the interviewees had characteristics of Rogers' adventurous early adopters.
Foote et al. estimates diffusion of SCALE-UP in physics may be at a tipping point between early adopters and the early majority, who tend to be conservative, cautious, more influenced by peer pressure and want to adopt a reform that will work.
Although early adopters, like the instructors in this study, may be willing to develop a curriculum and navigate challenges, in order to spread SCALE-UP to the more hesitant majority, the developer may need to provide additional support and guidance.Thus, a limitation of this study may be that the results primarily apply to selfmotivated and adventurous early adopters and the reform process may look different for people who need external motivation to change.

B. How does familiarity with PER influence implementation?
Responses to research question 2 revealed that this sample also exhibited the qualities of Rogers' "early adopters" because they were knowledgeable and well connected to scientific sources.Since sites for this study were chosen to have direct connections to the reform developer, these "early adopter" qualities also apply to our sample.Although instructors at only two institutions had formal training in PER, they all were interested in the field and sought out talks, workshops and conversations with colleagues to learn more about research-based reforms.Four learned about the reform through relevant workshops or talks and one learned about the reform because he was undergoing his own PER project.As seen in Table III, people learned about most of the researchbased reforms that they integrated in the course through interpersonal connections and conversations with colleagues.Since they were already interested in and connected to the PER community, these conversations were possible.An isolated individual without colleagues interested in PER or resources to attend relevant conferences might have more trouble.
Even before implementing SCALE-UP, all of the sites were using some reformed methods in their physics courses, with all five sites using clickers or Peer Instruction.Since they all had personal connections to the reform developer, they reached out for advice when it came to choosing textbooks but also had the knowledge and confidence to innovate independently.
The reform literature emphasizes the need for implementers to recognize an existing model as problematic and then change agents must support adopters as they restructure their existing beliefs and knowledge to make sense of the novel idea.Since our sample was already using research-based strategies and motivated to change, there was not a strong dissonance between past and reformed instructional methodologies.Furthermore, these instructors were generally knowledgeable and well connected enough to reform their courses with only occasional support; for example, Institution B asked the developer for textbook recommendations.
While all the sites were able to implement SCALE-UP relatively independently, these interviews revealed that some instructors harbor misconceptions.For example, instructors (at three of these institutions) repeatedly mention they embarked on the reform believing that SCALE-UP was a ready-to-implement package that could be brought to their institution, even if they did not switch to the textbook the development site uses.It is unclear where these misconceptions came from but they seem to have persisted.It is possible that faculty members got an incorrect picture from the start, which could have acted as a filter through which they understood subsequent information.For example, two sites believed that SCALE-UP was a ready-togo curriculum until they began implementing the reform.Because of this, two of these institutions started by consulting NC State resources but quickly abandoned use when the format was too disorganized, minimal, or unintuitive for immediate implementation.Instead of spending the time to modify these materials, instructors preferred to create their own, as will be discussed more below.
In another case, Instructor B1 seemed to think various reforms were mutually exclusive, which limited the existing PER resources they incorporated into their reform.
"I guess we didn't see it as mix-n-match, like there were different schools of thought, different ways of going about it.I think we looked at the McDermott books and we thought SCALE-UP was the best, the way we're going to go and if there are things we could import from the other groups, we would do it but some of it didn't seem like it would naturally work." Interaction with the developer was not intense enough to reveal and correct these misconceptions, so this belief was left unchecked and affected the implementation process.
Having to convince skeptical, mainstream faculty may encourage adopters to structure their message to create this dissonance and ensure adopters understand the reform based on its core intent ("principles knowledge").However, building this understanding is a necessary but insufficient condition for successful reform.Even if implementers understand the reformer's intent, they may lack the necessary skills and resources to put those understandings into practice ("how-to knowledge").Thus, in addition to unpacking the key change ideas and justifying the changes, reform developers need to show how the reform ideas may be applied in practice, potentially using vignettes or examples.

C. What factors affect instructors' efforts at curriculum development?
Although some instructors thought they wanted an external package created elsewhere to work for them with minimal modifications, they ended up modifying (and in most cases creating) materials that aligned with their personal style, preferences, and skills.We saw that only the electromagnetic visualization applets were directly adopted, whereas all other resources were adapted to fit the new context.
Instructors at three of these institutions embarked on reform under the impression that SCALE-UP was a readyto-implement package, so they first tried to use NCSU materials.NCSU materials basically described modular activities, but did not describe what knowledge students should have before doing the course, what difficulties instructors may anticipate, and how it fits in with larger goals especially since the activities were designed for the M&I textbook, which presents information in a different order, with a different emphasis.Instructor C1 commented that he expected more of a lesson plan with detailed descriptions of the instructional context.
The NCSU lacked a description of the pedagogical content knowledge (PCK) [57,58] underlying the design of each activity, which the faculty might have benefitted from.PCK is the knowledge of subject matter, explicitly for the purpose of teaching.This includes knowledge of student's difficulties and prior concepts as well as relevant instructional strategies and assessment materials.Unless they have a background in PER, most higher education faculty do not think about the content of their courses from this perspective.This is the kind of information that reform developers should explicitly provide, since typical faculty do not have the same background and probably do not have the time to accumulate this understanding independently.
When the format of the resources was unintuitive for immediate implementation, instructors preferred to create their own instead of spend effort editing the current ones.Except for Institution E, almost all of the instructors cite having to start from scratch to create a course appropriate for use in a SCALE-UP environment.For example, Instructor D3 recalls "blowing up" the previous course since "nothing that we had been doing that was useful and nothing anybody else is using that was useful in that direct sense".
One instructor (A1) found that Knight's textbook and workbook was a more organized resource that could be easily modified into the NC State format of reading quizzes, clicker questions and hands on activities.All the other institutions found they had to piece together activities they had been doing (Peer Instruction, CRPs) and then develop additional materials from scratch.Except for the instructors that had already been familiar with PER, no interviewees extensively searched through the literature because of limited time.
Institution D consciously adapted the room design from NC State but felt that they were better off creating resources that fit the institutional, student and faculty culture to prevent pushback from faculty and students.
This has interesting implications for adoption of research-based materials.Much of the literature on effective dissemination recommends that developers provide resources that can be adapted to an instructor's local situation.However, this study revealed that instructors wanted activities they could implement immediately or they preferred to invent their own.Hutchinson and Huberman [59] echoed this finding, citing a study of ERIC users [60].The researchers found "one-stop shopping" was critical in ensuring use.If a source was listed in a bibliography and was also available at the time of inquiry, it would typically be used.Thus, secondary users may benefit from access to activities in an immediately usable form, as long as the context for use is articulated.
Perhaps a compromise would be to keep the overarching curriculum flexible but improve access to well-organized and user-friendly suggestions for composite activities.Although the SCALE-UP website shares NCSU SCALE-UP activities, these activities remain in a form that Institution A and B did not find helpful.The developer has made an effort to solicit activities to post on the website from other secondary sites but has not received much of a response.When we asked Instructors E1 and E2 about sharing their resources, they claim the activities they use are not in shareable form.They would potentially be sharing materials if they were compensated with release time or funding to modify their resources into a usable form by others.
An unexpected advantage of materials that could not be directly implemented is that it promoted change that went deeper than superficial use of preexisting resources.All of the institutions studied here recognized that they needed a brand new curriculum when they switched to SCALE-UP.Encouragingly, they all underwent radical reform and none attempted just to edit previous lecture notes or add in isolated activities.However, many of these users had no official training in creating research-based materials so these new materials may not always uphold the intent of SCALE-UP reform.The implications section below gives an example of an alternative dissemination model that helps instructors make sense of a reform and build skills to innovate independently.

D. How do instructors achieve sustainable use?
With the dramatic switch to a restructured classroom, most sites mentioned wanting to initially avoid radical curricular changes to minimize unfamiliarity during the pilot.Changing curricula gradually had the advantages of allowing instructors to develop a course they would feel more confident teaching and to maintain a control for data collection.
While these are good reasons to transition gradually, instructors have a finite amount of time so four sites stopped making major changes once a "working form" is established.This preserved a reform that may be more conservative than it was originally designed to be.Perhaps, on-going incentives from the department could help instructors to continue to innovate and refine their curriculum.It seems without this, instructors felt satisfied enough with their change efforts and did not want to invest further time and energy.
For example, instructors from institutions A, B, C, and D mention making very minor adjustments in subsequent years.After developing the course, subsequent preparation primarily involved running through the slides.Even with the switch from SCALE-UP to Lecture-Studio, Institution D still directly used the vast majority of preexisting material.When instructors from Institution A and C first heard about SCALE-UP, they thought they may switch to M&I after getting used to teaching in the new format.However, once they got used to SCALE-UP, they admit to abandoning that idea, primarily because of a lack of time and "laziness" since they established a working reform.The exceptions to this are Institution B who is actively working to improve their labs (but they felt satisfied with the other parts of the class) and Institution E which took a more phased approach to appease students and faculty.
In general, a cautious approach to curricular development seems to aid sustainability but when an initial implementation attempt gets preserved, it may result in a reform that may be more traditional than intended.This echoes previous findings about the adoption of SCALE-UP at secondary site specifically [61] and research-based reforms in general.With more initial support from developers, adopters may feel more comfortable adopting a less conservative curriculum and avoid encountering this situation.

VI. IMPLICATIONS AND AN EXAMPLE FOR AN ALTERNATE DISSEMINATION MODEL
The results and conclusions of this study imply that if developers want their innovations to spread successfully, they cannot expect that spreading awareness of their reform at secondary sites is sufficient to create successful use.SCALE-UP is a radical innovation that requires a restructuring of the classroom but also significant changes to teaching practice.In addition to spending time changing the course and syllabus, instructors need to spend time learning about the reform and how to implement it.For radical innovations like these, developers should provide more support since, in the cases we studied here, time constraints affected the final form of implementation, even for selfmotivated early adopters.If we want to change mainstream instruction, secondary users will need even more support.Then, the question is, how can the developer best support these secondary sites?
A. Developer must articulate critical components of the innovation and where secondary users have flexibility to innovate The Increase the Impact project has created workbooks to encourage reform developers to make successful propagation a priority from the onset of reform development.This workbook discourages reform developers from treating dissemination as an optional afterthought and instead guides instructors through thinking about usability when designing their project.For example, it encourages developers to understand the subtleties of their innovation, identify potential adopters and their instructional system.For example, most STEM educational efforts are aimed at higher education faculty that lack the PCK that may seem intuitive to reform developers.Therefore, developers should explicitly frame resources to fill this gap.Then, they encourage researchers to develop their product while interacting with users, disseminate interactively and support adopters throughout the change process.Throughout this process, researchers need to keep reviewing the critical components of change, and the resources and characteristics of potential secondary sites.
To ensure modifications made by secondary users uphold the intent of the reform, developers also need to make sure adopters understand what they can change without compromising the integrity of the innovation.Unlike secondary teachers, many physics faculty may have never created curricula and activities.Increase the Impact recommends developers construct an Innovation Configuration that describes (i) critical innovation components, (ii) possible variations for each component, and (iii) acceptable ranges of implementation for it to be considered the same product [62,63].If the user believes they can modify aspects of the innovation that are actually critical to success, this can result in failed implementation and continuation.As part of this step, the developer could provide users with a selection of other PER innovations that complement the SCALE-UP reform.Also, they can provide specific activities, supplemented with relevant PCK about each.Then, secondary adopters will have access to a selection of existing activities.

B. Adopters benefit from interactive interactions
Hutchinson and Huberman [57] found the best single predictor of knowledge use and gain is the intensity of contact between dissemination and receivers.Effective dissemination "…involves frequent contact, some faceto-face interaction, and an exchange between dissemination specialists and participants that lasts more than a few months over time" (Ref.[57], p. 17).This sustained interactivity helps ensure the two populations have a shared understanding of the reform, which might have prevented misunderstandings that emerged in these case studies.
Considering the number of SCALE-UP sites (currently over 250), it would be unrealistic to expect the developer to directly communicate with each.Fortunately, engaging other secondary sites in the dissemination process has dual benefits of (i) transferring the dissemination responsibility from the developer to several others and (ii) improving the effectiveness of communication, since persuasion is often more effective amongst similar people.Rogers explains, "when two individuals share common meanings, beliefs, and mutual understandings, communication between them is more likely to be effective" (Ref.[27], p. 306).Attempting conversation between two dissimilar people can lead to STEM faculty's skepticism of innovations developed in different contexts or by people they perceive as dissimilar to themselves [64][65][66].This may be especially important moving forward since the "early majority" is influenced by peer pressure and wants to ensure a reform will work for someone like them before they decide to adopt.
Finally, a knowledge imbalance in the curriculum development process can lead to expert-novice differences that contribute to misunderstandings that developers might not anticipate or resolve.Just like physics students who can resolve misconceptions, better talking to a peer than their instructor, faculty who recently attempted reform may provide more relevant guidance than the reform developer.Since these new SCALE-UP instructors share a similar background, vocabulary and past experiences with new users, they may be better able to predict and avoid miscommunication.The challenge for developers will be to find ways to prepare secondary users and engage them in the dissemination process.

C. An example of alternative dissemination model: POGIL Core Collaborators Workshop
Although it would be nice provide extensive training in PER and PCK to instructors so they are confident and comfortable designing some of their own curriculum, this is not an option for most higher education faculty.Thus, a compromise would be a focused, interactive workshop where participants learn some of the design principles behind a reform, but more importantly have the chance to practice creating curriculum related to the reform with supervision and immediate feedback from developers.
An example of an alternative dissemination model that puts most of these recommendations into practice is the POGIL Core Collaborators Workshop [67].POGIL [68] (Process Oriented Guided Inquiry Learning) groups students in teams of three or four to work on carefully designed, scaffolded problems in chemistry or other subjects.POGIL developers experimented with using Core Collaborators Workshops (CCW) to aid with dissemination and develop a community of practice [69] around this reform.They invited approximately twenty motivated participants from the same discipline (biochemistry) for a multiday workshop with the goal of improving available materials and supporting faculty in transforming their classrooms.Participants collaborated with developers to create additional modules and assessment.During this process, the participants received feedback from developers and built the confidence they needed to innovate independently while expanding resources for the larger POGIL community.
Research shows that attending one workshop usually does not lead to lasting changes in teaching [70] but the elongated exposure and feedback during this workshop aimed to build a deeper level of understanding and a supportive community to promote long-term transformation.Although a grant only provided funding for two CCW workshops, these events united geographically separated people and "provided a community and a network of support for the pedagogical shift that might not exist at their home institutions" (Ref.[63], p. 410).The study showed that people who completed the workshop were more likely to act as leaders in other reform efforts.

VII. FUTURE WORK
This paper reports results from an exploratory study with the best-case scenario of intrinsically motivated, knowledgeable adopters.Integrating reformed teaching in the mainstream will require convincing more conservative and reluctant faculty members who may need more guidance.This paper focuses on the initial curriculum development efforts of the reform initiators in a department.To investigate what happens when the reform reaches more typical instructors, one should study what happens to curricula when hesitant faculty are required to teach the course in this manner.Since the departmental and organizational context would be similar, a study like this would allow researchers to investigate the implementation and curriculum development process for more conventional faculty.
Future work can continue to investigate how to mitigate and eliminate the divergent expectations of educational researchers and faculty that can serve as a barrier to reform.For example, how can we frame reforms to make the role of the change agent and the reformer more explicit?How can we frame reforms to include ready-to-go research-based tools within an adaptable basic structure?Answering these questions will require a continued dialogue between reform developers and implementers to identify the needs and preferences of both populations.Examining how implementers understand reforms and adjust them to personal preferences and situational constraints will help develop dissemination techniques that promote usability and sustainability of research-based reforms.

TABLE II .
Steps of the change process and related research questions.
RQ2: How do implementers learn about and gather information about the reform?3.Instructor implements the new practice.RQ3:Where do curricular materials come from?What are the roles of the instructor and the change agent in the implementation process?RQ4: After coming up with an initial curriculum, what changes are made to subsequent versions and why?How do sites achieve sustained use of the reform?

TABLE III .
Institutional profiles based on Carnegie Classification, all institutions are full time, 4-year institutions with a student population ranging from 7000-27 000.A Research University (RU) is classified as such based on research expenditures, the number of research doctorates awarded, the number of research faculty, etc.A Master's College (MC) is defined by the Carnegie Classification as an institution awarding at least 50 master's degrees but fewer than 20 doctorates.