How faculty learn about and implement research-based instructional strategies : The case of Peer Instruction

Melissa Dancy, Charles Henderson, and Chandra Turpen Department of Physics, University of Colorado–Boulder, Boulder, Colorado 80309, USA Department of Physics and Mallinson Institute for Science Education, Western Michigan University, Kalamazoo, Michigan 49008, USA Department of Physics, University of Maryland–College Park, College Park, Maryland 20782, USA (Received 30 November 2014; published 22 February 2016)


I. INTRODUCTION
Over the years, funding agencies, such as the National Science Foundation, have invested hundreds of millions of dollars in education research and development, resulting in numerous research-based teaching methods, technologies, and curricula to increase student learning and engagement in undergraduate science, technology, engineering, and mathematics (STEM) [1,2].In many ways, physics has been a leader among other STEM disciplines in the development, testing, and dissemination of new teaching methods [1,2].However, many physics faculty still do not implement these methods [2][3][4] and those who do generally make significant adaptations [2,[5][6][7].Little research has been done to understand the nature of secondary implementations or how developers can go beyond simple "dissemination" to achieve educational transformation.
Without a better understanding of how to create sustainable educational transformation, the full value of the enormous expenditures of money and effort will not be realized.
We are engaged in a multiproject research program to better understand educational transformation in higher education STEM.Improved understanding will lead to the development of change processes that can increase the use of research-based teaching in a way that is effective, widespread, and sustained.In this paper we explore how physics faculty learn about teaching innovations, how they use these innovations, and the interplay between these two.We focus on the Research-Based Instructional Strategy (RBIS) of Peer Instruction [8,9].
In particular, we answer the following questions: (1) How reliable are faculty self reports of their use of Peer Instruction?(2) When faculty modify Peer Instruction what components do they describe using, modifying, and abandoning?(3) What do faculty report knowing about Peer Instruction?(4) How do faculty report learning about Peer Instruction?
(5) Is there a relationship between the method of Peer Instruction exposure and the extent to which an instructor implements components of Peer Instruction?

II. BACKGROUND
In this section we briefly introduce (i) the instructional strategy of Peer Instruction (PI), (ii) the diffusion of the innovations change model [10], which we use to organize our analysis, and (iii) the disseminating curriculum and pedagogy change strategy that is implicitly used by most STEM change agents [11].

A. Peer instruction
In this paper we report on a study of physics faculty selfreported knowledge and use of the RBIS of Peer Instruction [8,9,[12][13][14].Peer Instruction was chosen for this study because it is the most commonly known about and used RBIS for introductory quantitative physics [3,4].A web survey of a randomly selected sample of U.S. physics faculty found that 63.5% of faculty report knowing about PI and 29.2% of faculty report using PI [4].These were the highest levels of knowledge and use of all of the 24 RBIS asked about in the survey.
PI is similar to, and shares many significant characteristics, with other RBIS.Therefore, it is expected that findings from an analysis of the use of PI can significantly inform broad scale model building and recommendations for impactful educational transformation.
PI was developed by Eric Mazur for use in his large lecture introductory physics courses at Harvard University.In a PI class, the instructor delivers short lectures (7-10 minutes) followed by a multiple-choice conceptual question.Students first think about the question and answer it individually (often using a personal response, or "clicker", system), then discuss their answer with a nearby classmate, and, finally, revise their answer.Based on student responses to the multiple-choice question, the instructor may decide to move on to the next topic or to continue with the current topic.Studies by Mazur and his colleagues have shown that PI is successful in improving student learning of physics content [8,9,12,15,16] and reducing the gender gap [15].Similarly positive results have been found in a variety of secondary implementations in physics classrooms at a wide variety of institution types [13,17].The PI pedagogy has also spread to other science disciplines [18][19][20][21][22].
B. Individual decision making in the change process: Rogers' stages of the adoption process Rogers [23] proposes that decision making related to adoption of an innovation occurs over time in a series of five stages (Fig. 1): Knowledge about the innovation, Persuasion about the benefits of the innovation, Decision to use the innovation, Implementation of the innovation, and Confirmation of continued implementation of the innovation (Ref.[23], p. 162).We find Rogers' framing helpful for understanding the change process and in identifying ways to support reforms, and we will use these five stages as a framework within which to organize our discussion.It is important to keep in mind, though, that we do not consider there to be firm boundaries between these stages.Rather, we expect that an individual can be engaged in activities from multiple stages at any given time.Likewise, these stages are not necessarily linear in time as a faculty member may circle back and around as they generally progress though the stages.
Our previous work indicates that current change efforts have been reasonably successful at stage one: increasing the level of knowledge that physics faculty have about RBIS [24].As shown in Fig. 2, in a large scale survey of physics faculty, nearly 90% report awareness of at least one RBIS [24].Additionally, current reform efforts appear to be effective at persuading faculty of the value of RBIS (stage two) as indicated by more than 70% of faculty reporting interest in using more RBIS than they currently do [24].Likewise, for stage 3, over 2=3 of all faculty report having tried a RBIS [24].
The problem appears to mainly lie in the final stages of the adoption process.Over 1=3 of all faculty who try an innovation discontinue use [24].This represents the biggest loss point.Change strategies focused on the faculty who try but discontinue use is likely to be more impactful than trying to persuade the small number of faculty who chose not to try a RBIS at all.It is important, therefore, to better understand how faculty implement RBIS once they make the decision to do so.We explore this question in the context of PI.

C. The disseminating curriculum and pedagogy change strategy
A review of the literature on change strategies in higher education found that STEM-based change agents most commonly use change strategies based on disseminating curriculum and pedagogy [11].Similar to prior work, e.g., Ref. [25], the literature review also found that change strategies are often used implicitly, a feature which impedes improvement of change strategies.
The disseminating curriculum and pedagogy change strategy is characterized by the development of researchbased strategies and curricular materials by experts in education research followed by the dissemination of these strategies and materials, typically through journal articles, books, websites, talks, and workshops.In the dissemination phase, change agents usually focus on informing faculty of the shortcomings of lecture-based methods, introducing the new method, using data to demonstrate the superiority of the new method, and then providing materials to support faculty use.Note these activities line up very well with the first three stages of Rogers' Innovation-Decision process discussed above (knowledge, persuasion, decision) but do not directly support the final two stages (implementation and continuance), which are also the stages where we find current reform efforts to be having the least success.
The disseminating curriculum and pedagogy change strategy makes sense intuitively.It places the burden of developing materials and methods on those who have the most expertise, interest, time, and resources to develop them.However, this change strategy contains implicit assumptions that we find to be problematic.
(1) The disseminating curriculum and pedagogy change strategy assumes change is the result of individual decision making; i.e., that faculty can change if they want to.In reality, enacted practice will be a complex mixing of both individual decision making and environmental affordances and constraints [7].An instructor may find value in a research based reform but not implement it, or significantly change how they implement it based on the structures around them (i.e., student perceptions of the innovation, content coverage expectations, teaching norms in their department, class size, class room design, etc.).(2) The disseminating curriculum and pedagogy change strategy assumes faculty can take curriculum and methods developed elsewhere and easily implement them into their unique situation.Given the wide range of institution types, student attributes, local expectations and norms, and faculty personalities, it is unlikely that any RBIS can be effectively implemented broadly "as is."Some amount of customization will always be necessary.While the disseminating curriculum and pedagogy change strategy has achieved some success (it appears to be effective at increasing knowledge and motivation to use among faculty) it has not resulted in the overall goal of large scale and sustained reform.In the discussion section of this paper, we will further consider the disseminating curriculum and pedagogy change strategy and suggest aspects of a more effective strategy.

III. RESEARCH DESIGN AND METHODOLOGY
The findings reported here are based on interviews with 35 faculty with varying levels of self-reported knowledge and use of Peer Instruction.Details of the research procedures are reported below.Note that this paper focuses on the interview results related to instructors' knowledge and use of Peer Instruction.Other papers use this same set of interview data to focus on instructors' perceptions of the affordances and constraints of using Peer Instruction [26], and how instructors and institutions judge teaching effectiveness [27].

A. Interview participants
Purposeful sampling.-Ourinterview sample was drawn from faculty who completed a web-based survey as part of a previously published research study [4].The American Institute of Physics Statistical Research Center (SRC) led the sampling efforts for the web-based survey to target three types of institutions: (i) two-year colleges, (ii) four-year colleges that offer a physics bachelor's degree as the highest physics degree, and (iii) four-year colleges that offer a graduate degree in physics.We selected our interview sample to draw equally from the three institution types.In addition, we used a subset of the faculty's survey responses to further inform and justify our purposeful sampling of faculty for our interview study.The following factors self-reported on the survey were used to guide our purposeful sampling: faculty position, knowledge and use of research based instructional strategies (RBIS), years of teaching experience, and gender.
We choose to interview only full-time permanent faculty.We focused on full-time permanent faculty for three reasons: (i) full-time permanent faculty were the primary target audience of the web survey, (ii) full-time permanent faculty are the primary target of most RBIS dissemination efforts, and (iii) the institutional situation of full-time permanent faculty is different from part-time and nonpermanent staff or faculty.
We choose to interview faculty concerning Peer Instruction because it is the most commonly known about and used RBIS for introductory quantitative physics [3,4].The following survey question concerning the faculty's knowledge and use of PI was used to inform interview sampling: Survey Question: Please indicate your level of familiarity with each of these research-based instructional strategies.With respect to Peer Instruction, respondents were categorized as follows: • I currently use all or part of it → PI user • I have used all or part of it in the past → Former PI users • I am familiar with it, but have never used it → Knowledgeable non-PI users • I've heard the name, but do not know much else about it → Not knowledgeable about PI • I have never heard of it → Not knowledgeable about PI Using information from parallel survey questions on different RBIS, we decided to break down the category of "Knowledgeable non-PI Users" into two groups: (i) Those that are nonusers of PI, but use other RBIS and (ii) those that are nonusers of PI and use no other RBIS.In the context of the PI interviews, we decided to interview the nonusers of the second type (those who don't use any other RBIS, but have some knowledge of PI).Table I shows the survey sample that fall into each of these knowledge or use categories by institution type.
In our interview sample, we aimed to interview 4 faculty from each institution type (two-year college, BS or BA physics, graduate physics) within each of the three targeted knowledge or use groups (user, former user, knowledgeable nonuser).As part of our broader interview study, we also interviewed faculty concerning their knowledge and use of Workshop Physics (WP) [28,29].Since the population of faculty that was knowledgeable about or using WP was smaller than the PI sample, we drew this sample prior to selecting our PI interview sample.Faculty that were chosen to be interviewed about Workshop Physics were removed from this pool prior to selecting the PI interviewees.
Last, within each group we selected interview participants to create diversity based on gender and years of teaching experience.The median number of years of teaching experience for the sample of all permanent faculty surveyed was 14 years.So for each group of 4 we drew a sample of 2 that fell below the median and 2 that fell at or above the median.For each group of 4, we drew a sample that included one female and three males, which somewhat oversamples women from the population of physics faculty (14% of physics faculty overall are women [30]).
Once the target interview sample was selected, faculty were contacted through email to request a telephone interview.Faculty were offered $75 to participate in the 1 hour interview.Faculty who did not respond to the initial email invitation or follow-up emails were contacted by phone.A small number of faculty in the target interview sample were nonresponsive via email or phone (7 out of 51).

B. Response rate
Fifty-one faculty were contacted to participate in the interview study regarding Peer Instruction.Nine faculty declined to participate and seven faculty did not responded to our inquiries by email or phone.Thirty-five faculty agreed to participate in the interview study, resulting in a response rate of 69% participation.Within the PI sample, there are no statistically significant differences between institution type (two year college, physics BA, and physics graduate) and their likelihood to be interviewed via a chisquared test (p ¼ 0.495).Our interview sample also well represents the targeted representation of female faculty (at ∼25%).
We began our interviews by asking the interviewee to confirm their self-reported user status of PI from the web survey (PI user, former PI users, or nonuser of PI).If they

C. Interview procedures 1. Features of peer instruction
Prior to conducting the interviews, we constructed a short narrative description of Peer Instruction.This description was shared with several PI experts, including the developer, and edited based on their feedback.The revised description was used to design a section of the interview protocol aimed at generating detailed information about the instructors' implementation and knowledge of PI.Based on the narrative descriptions of faculty and the researchers' initial narrative description, we developed a list of nine distinct features of PI; see Table III.It should be noted that these features are not meant to be a definitive implementation guide.They were selected to reflect important aspects of PI as suggested by the developer that we felt could be reasonably inferred from faculty responses during a telephone interview.

Interview protocol
Each interview was conducted by one of us.Prior to the interview we created a one-page profile of the interviewee based on the interviewee's survey responses so this information was easily available to the interviewer during the interview.The interviewers followed a semistructured interview protocol.The semistructured interview protocol asked faculty to describe the following: (a) how and when they became aware of PI, (b) how and why they began to use PI, (c) what they see as the strengths and weaknesses of PI, (d) what their instructional practices are in introductory quantitative physics courses, (e) how they use PI specifically, (f) how they implement (or modify or discontinue) specific features of PI, (g) how they perceive aspects of their departmental context, and (h) how they know if their instruction is working.
Potential follow-up prompts were described in the protocol to encourage faculty to elaborate, however, the interviewer improvised specific follow-up questions based on relevance to consensus research questions and the research team's emerging interests.Question (f) of the protocol described above probed directly for specific predefined aspects of the PI RBIS as listed in Table III.The interviews typically lasted about an hour, were audiorecorded, and subsequently transcribed.(or aspects of PI)? (i) What reasons do faculty give for using, modifying, or abandoning PI (or aspects of PI)? (j) What do faculty see as the strengths and weaknesses of PI?The interviews were analyzed using emergent coding with the assistance of qualitative analysis software [31].The research team first analyzed four interviews collaboratively to develop an initial coding scheme.The coding scheme was then refined, with definitions becoming more fully explicated, through analysis of additional interviews.Throughout the iterative revision process, a codebook was created to capture consensus definitions of codes.As significant changes to the coding scheme were made, coding of prior interviews was revisited.Each interview was independently coded by two researchers.Each researcher coded the interview independently.Differences in coding were resolved through discussion.In these discussions, researchers considered not only the final coding designation, but also compared the evidence from the transcript that informed the coding decision.Descriptions of codes associated with specific research questions will be presented as they become relevant in the discussion of the research findings.

IV. RESULTS AND DISCUSSION
Below we report our findings, organized by research question.
Research question 1.-How reliable are faculty selfreports of their use of Peer Instruction?
What does it mean when an interview or survey respondent says that they use Peer Instruction (or any other named instructional strategy)?One goal of the interview study was to determine how well faculty selfreports of use or non-use of PI matched more detailed self-reports of their actual instructional practices.
As previously described, our interview sample included a mix of self-identified users, former users, and knowledgeable nonusers of PI.As part of the interview process each faculty member was asked specific questions to ascertain more detailed information about what they actually did in their classroom.These questions were designed to be as unambiguous as possible.For example, if an interviewee said they had students talk to each other in class, the interviewer would ask follow-up questions (such as how often?, about what?, for how long?, in assigned or self-formed groups?, with what guidance from the instructor?) to develop a clear, detailed description of what was happening in the classroom.
This allowed us to code evidence that argued in support of or against an interviewee's use of each feature listed in Table III.The coded information about that feature was then assessed holistically and an assessment was made by the researcher according to whether the instructor used, partly used, or did not use each feature.At least two researchers compared their independent assessments of each interview and reached consensus based on discussion.From this analysis we were able to compare interviewees self-reported user group confirmed at the beginning of each interview (15 PI users, 6 PI former users, and 14 PI nonusers) with the specific features of PI which they actually described using or not using in the interview context.These results are shown in Fig. 3.
Note that users report using more features on average (6=9) than the other groups, which is what one would expect.Additionally, former users report using more features on average (4=9) than nonusers (1=9) which is also consistent with what one would expect.This suggests that even when an instructor drops a particular innovation their practice likely still retains some of the innovation's components and does not simply fall back to traditional instruction-which is encouraging.
However, looking at individuals, it is clear that some self-reported PI users use very few features of PI (for example, 3 interviewees used only 1 or 2 features) while some self-reported nonusers use many features.The discrepancy appeared to result from different interpretations of what PI is.For example, some faculty thought of PI in very general terms as any situation where "students work together" (even if they only worked together informally outside of class).This appeared to lead some faculty to classifying themselves as PI users when an expert in PI might classify them otherwise.Likewise, faculty who were more knowledgeable about Peer Instruction were less likely to classify themselves as users unless they were robustly and accurately using PI as promoted by Mazur, even though they were using enough PI features that they might be characterized as a PI users under looser criteria.A more detailed analysis of faculty knowledge about PI is reported in a later section of this paper.
We concluded that the self-reported user status does not tell us much about use of the nine PI features.Thus, we developed our own user status categorization (Table IV) for use in the analysis of the remaining research questions.
Using each instructor's detailed descriptions of their own classroom practice, we characterized each interviewee's implementation of PI and developed the following researcher-assigned user groups: High PI user (uses 7-9 PI features), Mixed PI user (uses 1-6 PI features), and non-PI user (uses 0 PI features).Table IV and Fig. 4 show how these groups were represented in the interview sample and how the researcher-assigned groups relate to the selfreported user status.
Notice that less than half (7=15 ¼ 47%) of the selfreported PI users were categorized as high users.Similarly, only about 2=3 (9=14 ¼ 64%) of the self-reported PI nonusers and 1=6 (1=6 ¼ 17%) of the self-reported former users were categorized in the nonuser group.As discussed above, this has implications about a mismatch between the interviewee's interpretation of PI and how an education researcher might view PI.
However, it should also be noted that most of interviewees were using some PI features (i.e., their instruction was not wholly traditional) but were not using Mazur's Peer Instruction with fidelity-they were what we classify as mixed users.This leads to an important implication for researchers who tend to consider use of a RBIS in binary terms-that is, an instructor is either using a RBIS or not.As it turns out, the most common way that instructors use a RBIS is to use some parts and not others.By talking about RBIS use in binary terms, we are missing important information about elements of the RBIS that may be in use.Looking at RBIS use in terms of specific elements allows us to ask and answer important questions.What elements do instructors eliminate or modify, and for what reasons?What elements do instructors tend to retain even when they move away from other aspects of the RBIS?To what extent do mixed implementations (a common feature of secondary implementations) still result in strong learning gains and to what extent are mixed implementations missing important features that reduce the benefits of the original RBIS?
Summary.-While self-reported user status is not meaningless (the general trends were what would be expected) there is substantial noise in the result.Or, put another way, faculty self-reported user status of Peer Instruction is not particularly useful in characterizing their actual practice.Care should be taken to collect self-reported data in a way that asks about use of specific instructional elements rather  Given that few faculty use all nine features of Peer Instruction, it is important to understand which features are most likely to be used and which are most likely not to be used.Below we report results for the 50% (n ¼ 18) of our interviewees who were identified as "mixed users."High users, by definition, used most of the nine features and nonusers, also by definition, did not use any PI features so an analysis of these groups is not particularly interesting for this research question.
Figure 5 below shows the features used, partially used, or not used by faculty in the mixed user group.A feature was coded as "used" when it was clear from the interview that this feature was used consistently; it was coded as "partly used" when it was clear that it was used but was used inconsistently; it was coded as "not used" when it was clear the faculty never used this feature; it was coded as "undetermined" when there wasn't enough information in the interview to make a clear determination.Again, all results were independently coded by two researchers with any disagreements resolved through discussion.
These results show that some core features of PI are more likely than others not to be used by mixed PI users.For example, only 20% of mixed users report consistently having students think about and commit to an answer as an individual before discussing in a group, 40% of mixed users report inconsistently or never having students discuss their ideas in groups and only 10% of mixed users consistently have students vote after discussing their ideas as a group.While arguments could be made about just how essential each of these components are, these departures from recommended use of PI could potentially affect the positive outcomes that PI offers.PI features most commonly used by mixed users include interspersing in-class tasks, not grading in-class tasks and having students discuss.Each of these features is used by more than 60% of mixed users interviewed.Note that the mixed user group is made up of faculty who say that they are PI users, faculty who say that they are former users of PI, and faculty who say that they are nonusers.Thus, we suspect that the features that are most commonly used by this mixed user group are those that more naturally appeal to faculty and, similarly, those features that are not commonly used by this group are those that faculty may be more resistant towards.
Summary.-Faculty do not use certain features that may be essential to the success of Peer Instruction implementation.It is not enough to persuade faculty to adopt a RBIS, they also need help clearly understanding and implementing the core features of the innovation.
Research question 3.-What do faculty report knowing about Peer Instruction?
Given that few faculty use all nine suggested features of Peer Instruction, it is important to ask what they know about suggested features for PI implementation.To what extent could lack of use of specific features be explained by limited knowledge versus a conscious decision to modify implementation?
We noticed that many faculty used the term "peer instruction" in a literal way to refer to students teaching each other or having students learn from each other by working in groups.In some cases they would switch between this literal usage of the term and a more specific FIG. 5.The percentage of mixed PI users that use, partly use, or do not use each of the PI features based on descriptions of classroom practices in interviews.For some interviewees there was insufficient information to characterize their use of a particular feature; these cases are displayed as "undetermined" in the figure .meaning, but in other cases their only usage of the term was its literal meaning.Faculty in this second, literal meaning only, group indicated no familiarity with Eric Mazur or his brand of Peer Instruction.We suspect this literal use of names may be fairly common across other RBIS (e.g., cooperative group problem solving [32,33] or problembased learning [34] could easily be interpreted in a general and literal way).Documenting the more general meaning that faculty gave to the term Peer Instruction was important for communication purposes.
We specifically asked faculty to describe what was meant by Peer Instruction.A minority of faculty (26%, N ¼ 9) only demonstrated use of the term peer instruction in a literal way as a broad assortment of in-class and out-ofclass activities where students worked with one another.For example, many faculty described Peer Instruction as something that occurs in tutoring rooms in their departments where students help other students with course material.The rest of the faculty we spoke with had knowledge of PI as a specific instructional strategy.Within this more knowledgeable set, one group of faculty (20%, N ¼ 7) indicated that they knew that PI referred to a specific instructional strategy, but did not mention any specific features of PI beyond having students work together.Some faculty (37%, N ¼ 13) indicated that they saw PI as a specific instructional strategy and described a few features of the PI instructional strategy.A smaller group of faculty (17%, N ¼ 6) indicated that they saw PI as a specific instructional strategy and described all or most of the PI features.Overall, about half (46%) of faculty who reported to be familiar with PI did not indicate awareness of any specific features of PI beyond getting students to work together and about half (54%) had more in-depth knowledge of PI.
We also sought to document the specific features of PI that each of our interview participants was aware of and reflective about.In describing faculty's awareness of PI features, we did not require faculty to connect the given instructional feature with PI specifically.In this way, when an instructor discussed the importance of giving conceptual tasks in class and gave an example of such a question, but did not clearly describe this as a feature of PI, this interviewee was still coded as aware of and reflective about using conceptual tasks.If the interviewee seemed somewhat familiar and reflective about the feature, they were coded as an somewhat aware of and reflective.For example, one interviewee mentioned that students have difficulties that instruction can address, but never described how this related to specific tasks for students to work ondue to this ambiguity this interview was coded as somewhat aware of and reflective about tasks drawing on students ideas.Other faculty were designated somewhat aware of and reflective about using conceptual tasks if they didn't provide any concrete examples or if they providing examples that contradicted one another concerning the nature of conceptual questions (i.e., where one example question posed was clearly numerical calculations without elaborated conceptual interpretation).If the interviewer explicitly brought up PI features (we often explicitly asked about features the interviewee did not mention, but not always due to time constraints) and the interviewee explicitly stated that they were not aware of the PI feature, the interviewee is coded as not aware of PI feature.Faculty were coded as "not mentioned" for each feature that did not come up in the interview at all.Although not as conclusive as "not aware", we argue that a not mentioned code indicates that this feature was not strongly associated with PI for the interviewee since the feature did not arise throughout the ∼1 hour discussions about classroom practice and with an emphasis on Peer Instruction.It is important to note that these codes represent knowledge about PI features, but not whether or not the instructor uses these features.
Figure 6 shows that faculty have different levels of awareness of the different features of Peer Instruction.Almost all faculty were aware of the PI feature of having students discuss with each other.This is not surprising due to the common literal interpretation of the name of this instructional strategy as discussed earlier.Other features, however, were only mentioned by about 60% or fewer faculty.This indicates that important ideas from education research have generally been poorly communicated to faculty and gives credence to the hypotheses that faculty often modify PI due to limited knowledge about proper implementation.However, there were clearly times where a faculty member knew of the suggested implementation and intentionally chose to modify their use.Because a discussion of this topic is so extensive and illuminating we have chosen to present this analysis in a separate paper [26].We reiterate that all of the interviewees indicated in the survey, and confirmed at the beginning of the interview, that they knew about PI.
Faculty may have literal (and broad) interpretations of the names of RBISs.Thus, it is important to be sensitive to how the names of RBISs are used in informal conversations with faculty as well as in more formal dissemination efforts.Communication is hindered when the same term means different things to different people.This finding also calls into question the extensive use of survey methods to determine the use of RBIS by faculty.This is common practice in research and evaluation studies where the impact of dissemination efforts or professional development efforts is measured through faculty knowledge about or use of RBIS is self-reported based on a researcher-provided list of RBIS.Unless survey respondents are asked (during the survey or in subsequent interviews) to describe the meaning that they ascribe to these RBIS names, survey results can be misleading and of limited utility.
Our results also suggest that efforts to promote RBIS are less successful at communicating about certain aspects of these innovations than about other aspects.There are many possible explanations for this finding: (i) change agents may selectively emphasize or forefront particular aspects of their innovations in their talks and writing, (ii) informal communication networks which share information about RBIS may selectively share particular aspects of these innovations, and (iii) faculty hearing about the RBIS may selectively attend to particular aspects of these innovations based on their interests and prior knowledge about teaching and learning.In all cases, we suspect that curriculum developers need to be more clear and explicit about articulating the important features of their innovations and justifying the importance of those features.
Additionally, change agents should consider a wider range of how ideas are communicated.Just as a traveler would have better results by learning the local language than in trying to speak louder or slower, change agents are likely to have better results by developing a more robust change model than to focus on intensifying the same strategy.Typically, dissemination has been based on single and brief encounters and unidirectional communication from change agent to faculty.Reconsidering this model is a potential area of progress.
Summary.-Faculty are often unaware of, or have misunderstandings of, essential components of a RBIS.Faculty (like students) make sense of new information through their existing ideas.Change agents should put more effort into understanding how faculty are perceiving the dissemination messages related to a RBIS and adjust their communication methods accordingly.
Research question 4.-How do faculty report learning about Peer Instruction?
Respondents were asked "Do you recall how you first learned about Peer Instruction?"Twenty-four of the thirtyfive interviewees recalled a "first" exposure to PI. Faculty responses are represented in Table V.We note that what faculty recall as their first exposure may not accurately represent their first exposure, but may represent, instead, FIG.Percentage of all by degree of awareness of each PI feature.Aware equals aware of and reflective about a PI feature, Somewhat Aware equals somewhat aware of and somewhat reflective about a PI feature, Not Aware means explicitly admits that they were not aware of the PI feature, Not Mentioned means the PI feature did not come up in the interview.the first experience with PI that is salient or memorable to them.The categories were developed from the preliminary analysis of the data and, as with all of the analyses presented here, the final categorization was done independently by at least two researchers and any disagreements were resolved via discussion.The thirteen exposure categories are as follows: • Informal discussions with nonlocal colleague: Exposure to Peer Instruction via informal interaction (discussion, email exchange, etc.) with an instructor who teaches at a different institution.Exposure to Peer Instruction via collaborative work with faculty in a school of education or PER group (either while in graduate school, at a previous institution, or at their current institution).• Other exposure: Exposure to Peer Instruction via some way that does not fit any of the other categories.Of the twenty-four participants who could recall a first exposure, more than half (58%) said that they first learned about Peer Instruction through informal discussions with a colleague.The second most common response was that they first heard about PI through a formal talk (such as a colloquium) or workshop (25%).Only 8% (N ¼ 2) claimed to have first heard about PI through reading (both citing Mazur's Peer Instruction book).Later, we will see that reading is not inherently a less memorable mechanism of exposure (since many faculty report learning about PI through this mechanism).Thus, we can conclude that informal discussions with colleagues are the most memorable first exposures and that reading about PI is not a common form of first exposure.This is a very important finding as most dissemination efforts focus on impacting faculty by producing materials to read (i.e., Mazur's Peer Instruction book).
We were also interested in finding out all avenues through which faculty may have encountered Peer Instruction.In addition to asking about first exposure, we also asked faculty how they subsequently learned more about PI.Thrity-two of the 35 faculty interviewed about PI could recall some aspects of how they were exposed to PI.Table VI provides the associated percentage of participants that were exposed to PI through each avenue.
We see informal discussion with their colleagues, reading, and formal presentations are all important ways that faculty learn about Peer Instruction.Of these, only informal discussions were a highly prevalent first exposure (Table V).This suggests that written materials and formal presentations may be sought out by faculty following initial exposure to PI through an informal discussion.It should be noted that although these results indicate that the most common way that faculty learn about PI is through informal discussions, this does not mean that the majority of knowledge about PI comes from informal discussions.A workshop, for example, could impact faculty knowledge of PI more than an informal discussion in the hallway.Our findings suggest that social interactions are an important way that faculty are exposed to research-based instructional strategies.These informal exposures then can lead faculty to seek out more information thorough formal structures (i.e., read a book or attend a workshop).
Summary.-Change agents tend to focus dissemination on the development of written materials and on giving talks and workshops.While these do appear to influence faculty, the most important first exposures appear to be informal discussions.It appears that these informal discussions provide faculty with much of their initial knowledge, and that faculty turn toward written materials and presentations to learn more if they develop sufficient knowledge and interest through these informal interactions.As will be discussed further in the conclusions of this paper, this is a major finding and represents both a shortcoming of the standard dissemination-based model of change and offers hope for the design of a more effective model.
Research question 5.-Is there a relationship between avenue of exposure to Peer Instruction and the extent to which an instructor uses Peer Instruction?
We found that faculty were exposed to Peer Instruction in different ways and that they implemented PI differently.To see if there is a relationship between these two things, we compared respondents' implementation group to their avenue of exposure.Results are reported in Fig. 7 and Table VII.
Overall, we did not see large differences between the groups.A few potentially interesting patterns are highlighted and discussed below.Because of our small sample sizes and large number of categories it is not meaningful to perform statistical comparisons.We therefore offer these observations not as definitive answers but rather to stimulate thinking for future research into this important area.
• Nonusers were the least likely to recall a "first" exposure (only 50% of nonusers compared to 71% of high users and 78% of mixed users recalled their first exposure).• Nonusers may be more likely to cite reading as their first exposure as compared to high and mixed users.It is particularly interesting that none of the high users cite reading as their first exposure but they were more likely than another group to have read about PI at some point in their learning process.• In-depth experience with PI through work with a school of education, a PER group, and (or) as a graduate student appeared to be highly impactful exposure experiences.Even though we had only a  small number of interviewees (N ¼ 5) who had been exposed to PI through these avenues, all of these were mixed or high users.Similarly, all instructors who were exposed to PI as a graduate student (N ¼ 4) are now mixed or high users.• It appears that high users (71%) are more likely to have been exposed to PI through discussions with a local colleague than mixed users (44%) and nonusers (30%).The reverse trend is also true.Mixed users (39%) and nonusers (30%) are more likely to been exposed to PI through discussions with a nonlocal colleague than high users (14%).These trends offer two potential insights for those interested in promoting use of RBISs.First, they highlight the overall importance of informal social interactions in the change process.While much of the change efforts of the STEM education research community are focused on producing written materials (i.e., books, websites, journal articles, etc.), these materials may not by the primary way faculty learn about new instructional strategies.And, most importantly, written materials do not appear to be effective as initial motivators for faculty to learn about a RBIS.This, of course, does not imply that written materials are not an important source of exposure; just that written materials do not appear to be a compelling first exposure.
Second, these trends suggest the potential impact of graduate school experiences and (or) collaborations with those in educational research.A number of universities are making efforts to revise their introductory level courses to establish instructional techniques that have a significant research-based component; i.e., replace recitation sections with tutorials such as those produced by the University of Washington [36,37].Our interviews of faculty suggested that when graduate students are involved in these reform efforts it impacts their own teaching later on.Although we do not have specific examples in this study, this finding suggests that programs designed to expose graduate students to innovative instructional strategies (e.g., CIRTL [38]) may have a strong impact.In a similar way, faculty who have become involved in a collaboration with an education research group are very likely to be users of PI.While it may seem obvious that these experiences would impact faculty, such experiences are not often thought of as mechanisms to create instructional change.
Our findings indicate that they could be and that this might be a useful area to develop.Efforts to revise teaching in a way that includes graduate students, as well as support for local Physics Education Groups may be able to significantly increase the spread of RBIS in physics.
Summary.-Our data were not conclusively able to answer the question of how avenue of exposure impacts implementation.However, they point toward informal discussions and working on an educational research or curriculum development project (while as a graduate student or as a faculty member) as potentially important and underdeveloped and understudied mechanisms for promoting instructional change.

V. CONCLUSIONS
In this section, we focus on three important conclusions of this study for those interested in promoting use of research-based instructional strategies (RBIS).Although this study was done in the context of a single instructional strategy (Peer Instruction) within a single discipline (physics), we expect that these conclusions are more widely relevant.
A. There are communication gaps in current dissemination efforts While dissemination efforts appear to be reasonably successful at increasing faculty awareness of RBISs such as Peer Instruction, there are still communication gaps.Faculty who implement a RBIS may not be fully aware of the specifics of the intended use of that innovation.For example, as we saw above, faculty were often unaware of many of the critical components of PI.Additionally, it should be noted that faculty were often unaware they were unaware.This presents a significant issue as faculty who implement an innovation without adequate awareness of the recommended use are likely to encounter difficulties and (or) not see the promised success of the innovation.In addition to the subsequent loss of potential learning gains for their students, it is also possible that the instructor will decide that the innovation is not effective and discontinue use.They may potentially even lose faith in other innovations promoted by the education research community and become less likely to try other innovations in the future.
Those involved in dissemination efforts should pay attention to communication issues and to addressing common misunderstandings about innovations.It should be noted that in the case of Peer Instruction, Mazur's book [8] does outline in great detail how to implement PI.The communication gap in this case was not a lack of the message being delivered clearly, it was a failure of the message being heard.This suggests that just delivering a correct and complete message is not sufficient to create learning in the intended recipient.Note that this particular challenge is not unlike the challenges facing physics instructors who attempt to have their students learn physics concepts.Similar to good teaching, change strategies that involve active learning can be highly productive [11].It is also likely that an important reason behind the communication difficulties in this case is that significant communication about PI happened through informal conversations among faculty.It is reasonable to think that as information was passed from person to person, essential information about how to implement PI was lost.

B. Faculty modify innovations and need more guidance to do so effectively
As we noted above and also in previous reports [3,39], faculty rarely use a research based instructional strategy "as is."They almost always use it in ways different from the recommendations of the developer.In some cases faculty are aware that they are not using the RBIS as recommended and in other cases faculty are not aware of the recommended use.We do not view modifications in general as a problem.Faculty teach in a wide variety of settings with a wide variety of students, local structures, course expectations, etc.It is unrealistic, and we believe unproductive to expect faculty not to modify a RBIS to fit their teaching situation.However, as found in this study, faculty often modify in ways that may be less likely to lead to effective outcomes.Of the self-described users of Peer Instruction, only about one-quarter (4=15 ¼ 27%) use all nine features of Peer Instruction and only about one-half (7=15 ¼ 47%) use seven or more of the nine features.For the majority of faculty in our study (the mixed users), certain aspects of PI were rarely used.For example, even though many faculty reported being aware of the suggestion to have students commit to an answer as an individual before discussing in their groups, only 20% of faculty in the mixed user group reported consistently using this feature of PI.
Faculty may modify an innovation for a variety of reasons.In the case of students committing to an individual answer, it is likely that this might be dropped because this step takes time and faculty generally feel they have too little class time to cover the course material as it is.The education research community does not give faculty appropriate guidance to make an informed decision about this issue.How much less effective is PI with this component dropped?For an instructor trying to balance innovation use with the use of the precious resource of class time, this becomes an essential question and one which the research community does not currently provide much guidance on.
In order for the education research community to do a better job at supporting faculty through modifications, we need to do more research to understand the modifications faculty are likely to make, and the impact of the modifications on the effectiveness of the innovation.Currently it is common for a curriculum or pedagogy to be considered complete after being developed, and shown to be effective, at one institution and then made available to the general population of faculty.This approach leaves faculty on their own to figure out how to adapt the innovation to their unique situation, which may lead to a loss in the effectiveness of the innovation (or an improvement in the innovation).Peer instruction has been around for a long time and has become relatively widely known.Such research into the impact of importance of specific features is only beginning to emerge; e.g., see Refs.[40][41][42].
We encourage the education research community to pursue more research on secondary implementations and critical features of RBIS use.What modifications do faculty typically make and what are the reasons for these modifications?To what extent do typical modifications diminish or improve the effectiveness of the innovation?What support do faculty need to work though implementation difficulties and achieve success?

C. Informal, social interactions among colleagues are a key mechanism of communication about reforms
While most dissemination efforts focus on communicating with faculty though formal mechanisms, such as journal articles, websites, books, workshops, etc., the results of this study suggest that a significant way that faculty learn about teaching innovations is through conversations with colleagues.Studies of engineering faculty similarly point to the importance of these informal mechanisms [43,44].This represents a mismatch between the expectations of those interested in promoting the use of RBISs and the reality of how change happens.While informal interactions are problematic in some ways (e.g., as mentioned above essential information may not be communicated consistently) they can also be very influential as evidenced by the reports of the faculty in this study.It is important for researchers and curriculum developers to acknowledge and embrace this characteristic of social communication channels and develop strategies to leverage these social interactions in productive ways.This is an area where more work and research is clearly needed.Possible avenues to pursue to harness the powerful impact of social interactions include promoting and supporting more opportunities for faculty to come together over an extended period of time to learn and support each other in a structured environment, e.g., Refs.[45][46][47][48].Additionally, more efforts could be made to support and encourage online community development such as listserves and interactive websites.In all cases, we emphasize that it is not enough to try these approaches to faculty development.For knowledge of effective faculty development to improve, these activities must be approached from a research perspective so that the knowledge base can improve over time.

VI. RECOMMENDATIONS
In this section we make four recommendations that come from this work for those who are interested in effective and sustained educational transformation.
(1) Effective change strategies will acknowledge the highly social nature of the change process.Change agents should not view the widely impactful social interactions as a hindrance but rather embrace them and search for ways to leverage them while providing structures to minimize the communication gaps that will occur as information travels informally.In addition to maintaining the current avenues of formal communication such as workshops, print materials, and websites, change agents could also support and encourage communities of faculty to support change (i.e., facilitated faculty learning communities, the development of online communities, etc).(2) Faculty will modify ideas.Simply providing faculty with general directions and curricular materials decreases the likelihood of successful modifications.More research is needed to guide faculty in deciding when and how to modify a RBIS to fit ideas into their unique environment.Additionally, faculty are likely to benefit from longer term support than provided by a workshop or journal article.As faculty struggle to implement ideas they would benefit from more ongoing support to meet them where they are in their own learning process.Frequently change agents use an intuitive rather than a research-based model for change, e.g., Refs.[11,25], and then wonder why they did not have a more significant impact.As our findings show, intuitions are not always accurate (for example, the importance of social connections is often underestimated).The research on how to create impactful and sustained change is still in its very early stages and there are many opportunities to use research to inform practice.

VII. FINAL WORDS
In summary, we found that (i) faculty self reports on surveys can be poor measures of whether or not they use a particular instructional innovation, (ii) faculty frequently use innovations in ways that are inconsistent with the recommendations of educational research (due to either a lack of knowledge or a personal decision to modify), and (iii) faculty learn about innovations in ways that are neither expected nor supported by the commonly used disseminating curriculum and pedagogy change strategy.Each of these findings can help change agents improve their efforts to promote the use of research based instructional strategies.
In the biggest of pictures, we offer that the most important implication of our work is that the standard model of education change currently used in higher education STEM fields is fundamentally flawed.Most change efforts currently follow the Disseminating Curriculum and Pedagogy change model.This model is characterized by the development of innovations by experts (i.e., physics education researchers) followed by the dissemination of the innovations to mainstream faculty.Our findings, presented in this paper, show the failure of this model in two key ways.First, the dissemination model assumes faculty will implement an innovation as is, and does not account for the reality of faculty modifications.Second, the dissemination model assumes that faculty primarily learn about innovations through formal means (such as written materials, talks, and workshops) failing to account for the reality of the significant impact of more informal social interactions among colleagues on teaching practices.
Education researchers often criticize faculty for not taking a research-based approach to their teaching (i.e., relying on the lecture method to teach because it intuitively makes sense that having an expert tell students what they need to know is efficient and effective).However, we have generally failed to use a research-based approach to our change efforts (i.e., relying on the dissemination model of change because it intuitively makes sense that having an expert develop curriculum and tell faculty how to use it is efficient and effective).While the dissemination model has been effective in some ways, it is an incomplete model.We encourage more research into how effective and sustained change actually occurs and the subsequent development of a research-based model for educational transformation.

Adapts:
Instructor adapts instruction to student responses on in-class tasks Answers not graded: Students are not graded on in-class tasks Commit to answer: Individual students have a dedicated time to think about in-class tasks and commit to answers independently before discussing with classmates.Conceptual tasks: Uses conceptual questions for in-class tasks Tasks draw on student ideas: In-class tasks draw on common student prior ideas or common student difficulties Multiple choice questions: In-class tasks have discrete answer options Questions interspersed: In-class tasks are interspersed throughout class period Students discuss: Students discuss their ideas about in-class tasks with their peers Vote after discussion: Students commit to an individual answer after peer discussion D. Analysis procedures Our analysis and coding of the interview transcripts was guided by the following analytical questions: (a) How did faculty learn about PI?(b) How is PI defined by faculty?(c) What aspects of PI are faculty aware of and reflective about?(d) What aspects of PI did faculty use in past implementations of PI? (e) What aspects of PI do faculty currently use?(f) What changes did faculty make in their implementation of PI over time?(g) What reasons do faculty give for initially trying PI (or aspects of PI)? (h) What reasons do faculty give for not trying PI

FIG. 3 .
FIG. 3. The number of PI features, out of nine, reportedly used versus the number of interviewees.Interviewees are grouped based on their self-reported survey responses as a PI user, PI former user, or PI nonuser.An arrow indicates the average number of PI features used by each group.Researcher-assigned mixed user category is also shown.

FIG. 7 .
FIG. 7. Percentage of faculty reporting to have been exposed to Peer Instruction through each avenue.Hashed bars indicate the percentage of people who recalled that avenue as their first exposure out of the subset of people who could remember their first exposure.

( 3 )
Good teaching practices also apply to faculty professional development.Faculty are people.Change agents should look for ways to incorporate what we know about how students learn into their efforts to support faculty learning about RBIS.We have highlighted above the importance of social interactions, and the need to address faculty modification.Note that these "barriers" to change can be reframed in teaching terms as change agents meeting faculty where they are rather than structuring change strategies around the agenda of the change agent.Thinking more about what we know about learning in general is likely to highlight other areas where change strategies can be improved.(4) More research is needed.Just as research into how students learn has greatly improved learning outcomes, specific and focused research on how change happens can greatly improve the change process.

TABLE I .
Descriptive statistics of full-time permanent faculty survey respondents: Interview participants were sampled from the first three groups (PI users, Former PI users, and Knowledgeable non-PI users).

TABLE II .
Institution type and gender of faculty interviewed in each Peer Instruction user category.

TABLE III .
Nine features of Peer Instruction targeted in this study.

TABLE IV .
Researcher-assigned PI user groups.

TABLE V .
Percentage of faculty stating each avenue of exposure as their first exposure.
Exposure to Peer Instruction via reading that does not fit in one of the other categories.• Work with School of Education or PER group:

TABLE VI .
Percentage of faculty citing each avenue of exposure as a way that they learned about Peer Instruction.

TABLE VII .
Percentage of faculty within each implementation group citing each avenue as a way that they have been exposed to Peer Instruction.% High users (N ¼ 7) % Mixed users (N ¼ 18) % Nonusers (N ¼ 10)