Designing for sustained adoption : A model of developing educational innovations for successful propagation

Raina Khatri, Charles Henderson, Renée Cole, Jeffrey E. Froyd, Debra Friedrichsen, and Courtney Stanford Department of Physics and the Mallinson Institute for Science Education, Western Michigan University, Kalamazoo, Michigan 49008, USA Department of Chemistry, University of Iowa, Iowa City, Iowa 52242, USA Texas A&M University, College Station, Texas 77843, USA School of Chemical, Biological, and Environmental Engineering, Oregon State University, Corvallis, Oregon 97331, USA (Received 30 November 2014; published 22 February 2016)


I. INTRODUCTION
Physics education research as a field has been highly prolific in developing new instructional strategies and teaching materials [1][2][3].Implementation of these strategies and materials has repeatedly been shown to result in significant positive impacts on students [4].
The reality faced by physics education researchers, however, is that developing good materials and documenting their effectiveness is not sufficient to promote widespread adoption.New strategies and materials are often presented by the developers at conferences and published in academic journals [5].Research shows that while many instructors are aware of new teaching strategies and materials, the use of these strategies lags substantially behind awareness [6].Additionally, when instructors try a new strategy or material, they often discontinue use [7] or make modifications to the original work that can potentially render the strategy ineffective [8].
The challenge faced by education researchers, then, is how to better propagate their new instructional strategies.In this paper we purposefully use the term "propagation" to draw attention to the difference between propagation and the more commonly used term "dissemination."Propagation puts the focus on the users of a new instructional strategy.Propagation has occurred when others use the new instructional strategy or material.Propagation activities are the ways that developers (and others) seek to create propagation.Dissemination, on the other hand, puts the focus on the developer.Dissemination has occurred when the developer tells others about a new instructional strategy, often via one-way communication mechanisms [9].Telling people about new ideas (i.e., dissemination) is rarely sufficient to get others to use a new teaching strategy or material [10].Thus, while dissemination is an important propagation activity, it is only one part of a successful propagation plan.In order to improve the situation, we need a better model of designing for successful propagation that can help education developers construct propagation plans that lead to sustained adoption of education innovations.This paper describes the first two phases of a three-phase study to develop such a model.

II. LITERATURE REVIEW
Here we briefly summarize some of the key pieces of knowledge from the literature about effective development and dissemination.We will do this based on three recent syntheses that draw on a large number of primary sources.More specific connections between the results of this study and particular findings in the literature will be made when relevant in the results sections.
One piece of relevant literature situates this study in the wider landscape of research on change in higher education [11].Henderson, Beach, and Finkelstein (2011) conducted a review of N ¼ 191 journal articles regarding change strategies in higher education.They identified three communities of researchers interested in improving undergraduate science, technology, engineering, and mathematics (STEM) instruction: STEM education researchers, faculty development researchers, and higher education researchers.Each of these communities had a preferred change strategy, with STEM education researchers focusing on the development and dissemination of curricula and pedagogy.The research presented in this paper is situated within the "disseminating curricula and pedagogy" change strategy in which education researchers develop new teaching materials and then inform individual instructors about these new materials.While other change strategies are also relevant in STEM education, the development and dissemination strategy is the prevailing way of thinking about change in the STEM education research community and thus it is important to consider what successful practice looks like within this change strategy.
There have been two recent syntheses that focus on how to minimize the gap between research and practice [12,13].Fixsen et al. (2005) reviewed N ¼ 743 articles about program implementation outcomes in different fields (e.g., psychology, engineering, social services, and justice.) The review generated a model of stages of implementation for programs and highlights the importance of change agents understanding the context of local implementation and engaging in frequent communication between program leaders and on-site staff for better fidelity of implementation.This emphasis on the context of local implementation and communication between program leadership and others is echoed in Hinton et al. (2011).This guide for education researchers who want to disseminate their work more effectively is based on a review of funded education development projects in Australia.It describes strategies researchers can use to communicate with potential adopters, and advises project teams to begin dissemination efforts at the beginning of their projects.Both sources are also concerned with the sustainability of new programs once funding for them is over.The agreed-upon points from these syntheses are threefold: (i) the context in which a new innovation is being implemented matters, (ii) communication between developers and potential adopters is necessary for implementation, and (iii) it is difficult to develop a product that results in sustained use after the funding period is over.These sources provide some useful elaboration of the barriers to successful propagation and point to some of the key features of the system that should be considered when thinking about propagation.

III. RESEARCH DESIGN
While much is known about the process of adoption of innovations in general [10] and common barriers to propagation [12,13], less is understood about the practices of education developers that lead to sustained adoption of innovations within the specific context of funded projects in undergraduate STEM instruction.The goal of this work is to develop a better understanding of the practices that lead to successful adoption.The outcome of the work is a model (i.e., a "theory") of designing for successful propagation within the context of higher STEM education.
Grounded theory guides the overall design of this multiphase study, visualized in Fig. 1.In a grounded theory study, the researchers begin with the data (not theoretical suppositions) and the theory is developed to fit the data [14].Grounded theory studies typically begin with an open-ended research question (like our goal above) which can become more focused as the study progresses [15].While people often think about a grounded theory study as being built on approximately 20 interviews with individual informants, grounded theory studies can and frequently do have multiple sources of data beyond interviews [16].
The core ideas of grounded theory that we employ here are as follows: • Theoretical sampling.This means targeting specific sources of data which will best address the research goal, rather than aiming for a generalizable sample [17].• The constant comparative method.Using this method, new data are interpreted as they are collected.
Interpretation of past data can change as new insights are reached [18].• Creative intuition.In a grounded theory study it is appropriate to not merely document codes but elevate those codes to theoretical constructs with the input of creative insight [16].In keeping with the grounded theory approach, we build an emerging theory from the raw data in several steps, with the end result being knowledge propositions.Our design is similar to a study designed to identify reasons for the overrepresentation of minority students in special education programs in large school districts [19].This study consisted of three phases: (i) describing the overall program process with data from a district, (ii) purposefully identifying 12 schools within the district to describe their program referral process in a more specific context, and (iii) in-depth case studies of 12 students to understand issues that had arisen in the data.Their analysis began with open coding and led into the creation of interrelated categories, carried over throughout the phases.Our study is similar in the phases of data collection and the overarching analysis scheme across the phases.
Each phase of our overall study, illustrated in Fig. 1, has a subgoal and involves gathering specific data to achieve that subgoal.The goal of the first phase was to develop an understanding of typical propagation practice of education developers.We used data from a survey of N ¼ 1284 National Science Foundation (NSF) principal investigators (PIs) and focus group data from eight disciplinary groups of NSF program directors.The goal of the second phase was to develop an understanding of successful propagation practice through detailed study of three instructional strategies that have been well propagated: Peer-Led Team Learning [20], the PhET Interactive Simulations [21], and Peer Instruction [22].The third phase will further build upon the tentative model resulting from the first two phases using data from interviews with a purposeful sampling of PIs of additional successfully propagated instructional strategies, as well as interviews with PIs of typical education development projects at the conclusion of their funding period.
Similar to many qualitative research traditions, the goal of grounded theory is to systematically develop emergent ideas beginning with the raw data.We began data analysis with open coding.Codes were grouped under categories, and categories were explained with propositions (which could interrelate), and finally a model was developed to combine the propositions.
The overall analysis plan is depicted in Fig. 2 below.
A detailed description of the first two phases and their respective methods and results are provided in the following sections.

IV. PHASE 1
We started our model development process by examining the current propagation practices of education developers.This allows for comparison between typical and successful practice, and it allows the final model to reflect the good parts (if any) that exist in current practice.Data for phase 1 came from a web survey of 1284 NSF principal investigators (PIs) and focus group interviews with 30 NSF  program directors.These groups of key informants and comparison between their responses allowed us to construct a baseline of typical propagation practice in funded NSF projects.

A. Phase 1 methodology 1. NSF principal investigators
One data source for phase 1 was a survey of N ¼ 1284 NSF Course, Curriculum, and Laboratory Improvement (CCLI) PIs.The survey aimed to capture the dissemination practices of CCLI PIs and was developed and implemented by Tront and colleagues [5].The survey was sent to N ∼ 2400 CCLI award winners (the predecessor to the Transforming Undergraduate Education in STEM, or TUES, program) and resulted in 1284 usable responses.With permission, we analyzed responses to three questions from the survey that related to PI beliefs about propagation of educational innovations (Fig. 3).
Analysis of questions 7, 14, and 15 began with a round of open coding to capture the essence of the responses, followed by several iterations of coding to produce categories with more descriptive subcodes.These categories were used to synthesize propositions.Many PIs used the space in "other, please specify" in question 14 and used 15 to provide further information on barriers, so the open ended responses from questions 14 and 15 were combined in the analysis.Note that the Likert scale options of question 14 provided ideas on barriers to the PIs, and they used the "other" space to go on further beyond the listed barriers.

NSF program directors
The second data source used in phase 1 was informal focus group conversations with the NSF TUES program directors [23].The 8 focus groups with 30 total program directors (all approximately 2 h long) were conducted by us and took place at NSF.Each group consisted of program directors who worked in a particular TUES discipline (biology, chemistry, computer science, engineering, geoscience, interdisciplinary, physics, and mathematics).We asked the program directors for their perspectives on what leads to successful propagation of TUES projects and what problems occur during propagation of TUES projects.We took detailed notes of the discussions.In the analysis we developed codes relating to propagation for each disciplinary group.Similar codes expressed by the program director groups were combined into categories and written in language that best represented the ideas expressed by the groups.These emerging categories were also revised through discussions within the research team and compiled in a two-page summary document.Program directors were given the opportunity to comment on this summary document.These categories contributed, like the categories from the PI survey analysis, to propositions.

NSF principal investigators
What does successful dissemination mean to PIs?-The analysis of question 7, "What does successful dissemination mean to you?" allowed us to understand how PIs view the propagation process.PIs largely responded to question 7 in terms of providing a dissemination activity that they plan to carry out, the results of using such an activity (in terms of numbers as the question suggested), or they provided both an activity and how many users they would ideally reach.The percentage breakdown and example responses are given in Table I.
We note that the phrasing of question 7 likely primed respondents in terms of the number of users, which is why 55% provided a number of users.Each category is discussed below.i. Dissemination activities Dissemination activities mentioned by PIs who gave them as part of their response were categorized using an emergent coding scheme.Six main codes emerged: publications (28%), presentations (20%), share with colleagues (12%), workshops (17%), website (10%), and textbooks (10%).Among the 241 PIs who mentioned a dissemination strategy, only 7 (2.9%) mentioned a dissemination activity that did not fit into one of the six previously mentioned.These methods included a white paper, producing a DVD, printing a brochure, mentoring students to pass on their work, and directly distributing a report to thousands of departments in the country and to policy makers.The emphasis on publications and conferences is also seen the Tront et al. analysis of the survey [5].
ii. Result of using the dissemination activities PIs who mentioned adoption (often in terms of numbers) as successful dissemination targeted two groups: external and internal adopters.Across disciplines, 71% of PIs indicated that it would be a success if others used the innovation outside of their home institution.Another 20% would consider dissemination to be a success if they influenced faculty or administration at their own institution.The remaining PIs (8.5%) in this category thought of adoption as stages involving both internal and external adoption, first at their home institution and then beyond.
iii.Other definitions of dissemination In spite of the priming from the question to give a number of adopters, many respondents provided "other" definitions of dissemination (12% as listed in Table I, or 150 respondents total).These definitions mainly fell into three categories: (i) informing other colleagues or instructors that the innovation exists, (ii) to bring about change in their discipline or science education as a whole, and (iii) to make the innovation freely available to other instructors.Some respondents who provided "other" definitions (6%) made a distinction between dissemination and adoption, seeing them as different terms.This is consistent with our definition of propagation as discussed earlier.Overall, however, the survey wording conflated the ideas of dissemination and adoption and most PIs went along with that conflation.
iv. "Have not disseminated yet" This is a small category but we draw attention to it (2% of responses in Table I).PIs with this response often said that they were at the beginning of their project and had not yet began to think about dissemination.
What barriers to dissemination are PIs concerned about?-In our analysis of questions 14 or 5 about the barriers PIs experienced, PIs overwhelmingly cited lack of some kind of resource, most commonly time, as a barrier to disseminating their work.Other resources they lacked were funding and personnel.There was also demand from PIs for more resources from the NSF (besides funding), such as a special conference for the dissemination of TUES projects, a marketing team for TUES projects, and help assessing the innovations for efficacy.

NSF program directors
Three categories were developed from the program director focus groups: propagation activities used by PIs that are not effective, propagation activities used by PIs that are effective, and barriers to propagation.Each focus group had both discipline-specific and general codes within each of the three categories.The discipline-specific codes largely focused on disciplinary barriers, such as the lack of centralized professional societies in biology and for interdisciplinary projects, and the size of engineering as a discipline.Here we focus on the three categories and the general codes within each category.• The nature of some disciplines was seen as not conducive to propagation.For example, the large size and larger number of subdisciplines within engineering and biology make propagation difficult.The place-based nature of geoscience meant that many grants are developed for a particular location and may not be appropriate for propagation to a different geographic area.

C. Phase 1 conclusions
The PI comments provided several insights.First, PIs want to institutionalize their innovation and disseminate to a broader audience outside their institution.Second, PIs think about dissemination mostly in terms of impersonal, "mass-media" mechanisms such as publishing papers or giving conference talks.Third, PIs separate development of the project and dissemination into distinct project phases, one of which (development) is most important and should occur within the grant funding period, and the other (dissemination) is less important and can appropriately occur after the grant funding period.The results illustrated what much of the literature and anecdotal experience in the field has long suggested: education researchers do not consider dissemination until near the end of a project when much of the work has already been completed, and they rely on traditional academic channels (i.e., talks and papers) to inform others about their work [23].The goal of this phase of research was to understand typical propagation practices.From the results, we suggest the following propositions: • Proposition 1: Principal investigators focus most of their attention on product development.Dissemination occurs after development, if at all.• Proposition 2: Principal investigators think of dissemination in terms of impersonal, "mass-media" mechanisms such as publishing papers or giving conference talks.
The program directors' input bridged typical and successful practice through their firsthand knowledge of typical grant outcomes along with their experience with a few grants that had been successful in creating large-scale change.They noted that many typical PIs submitted grants on their own, with no collaborations, and that grants with multiple collaborating institutions had a greater chance for institutionalization at those multiple sites.
The program directors had the same basic framework for propagation (development followed by dissemination) as expressed by the PIs, but strongly emphasized collaborations and hands-on approaches to disseminating.This mirrors what is known in change literature about opportunities for the developers to get feedback and beta testing [24] and opportunities for adopters to engage with an idea before being persuaded to try it and implement it [10].Research also indicates that instructors want to be involved in the development process of new strategies [25] and that propagation should begin at the start of a project through interactivity between instructors and developers [26].From the program directors, we suggest two more propositions: • Proposition 3: Program directors think about education development projects as a development phase followed by a dissemination phase.• Proposition 4: Program directors strongly emphasize the importance of interactivity in both development and dissemination.

V. PHASE 2
The goal of phase 1 of this project was to understand typical dissemination practices of education developers.We now turn to phase 2 with the goal of understanding successful dissemination practices by examining in detail three projects that have been widely implemented.

A. Phase 2 methodology
In order to develop a model of designing for successful propagation, it is important to study instances of successful propagation.Practices leading to successful propagation will be particularly useful when contrasted against typical practice.In phase 2 we studied three successfully propagated instructional strategies within college physics and chemistry.The sections below detail the selection of cases, the data collection, and the analysis for this phase.

Purposeful selection of cases: three well-propagated instructional strategies that vary in type of change
Identifying well-propagated instructional strategies.-Thethree instructional strategies were selected from a pool of branded well-propagated instructional strategies (N ¼ 19) intended for undergraduate STEM instruction [27].This list of well-propagated instructional strategies was developed by the authors through an email survey of 39 experts from seven STEM disciplines.The details of the process to identify experts and of the survey are described in a forthcoming article [28].Results from the expert survey were corroborated using other evidence of propagation, such as the number of nondeveloper sites using the strategy, publications about the strategy by nondeveloper instructors, spread to other disciplines outside the originating discipline, and visible signs of a network of users such as an active online presence or physical conferences devoted to the strategy.The list of well-propagated instructional strategies is available at our project website [27].
Selecting strategies that vary in change required to adopt.-Thethree instructional strategies used in this study were selected from the larger list to represent different degrees of cooperation (e.g., between individuals or departments), resources (e.g., money, institutional space), and change to teaching practices required for implementation.These measures (described for each instructional strategy in Table II below) were developed by the project team to explicitly discuss features of different educational innovations [29].
The degree of change required by adopters is thought to be an important variable in the likelihood of successful implementation (a "smaller" change to typical teaching practice might be easier to implement than a "larger" change to practice) [9].This range in degree (from no change to moderate) was desired in building a model applicable to a wide variety of instructional strategies.We wanted to see if innovations with different degrees of change use similar or different strategies to varying success in their own contexts.The instructional strategies were also selected based on their widespread adoption in and applicability to a wide variety of STEM disciplines.In the following sections we briefly describe each of the three well-propagated strategies that we studied.
Descriptions of selected cases.-Peer-LedTeam Learning (PLTL) is an instructional strategy that retains lecture and replaces recitation with weekly "workshops" facilitated by a peer leader (an undergraduate student who has done well in the course) [30].PLTL resources offer guidelines about what kind of student makes a good leader, weekly training of peer leaders, types of materials to use in workshops, appropriate faculty roles in developing materials and training leaders, and suggestions on small variations to adapt the program to different institutional settings.Critical components for successful implementation of PLTL include keeping the workshops to 6-8 students, making the workshop integral to the course, and ensuring that the program is supported by the department and institution [20].The program first started in 1991 as an idea of "collaborative-learning groups to improve student success at the City College of New York (CCNY)" [31].The idea began under an initial NSF grant and has impacted over 100 institutions [31].
PhET Interactive Simulations are highly flexible, freely available simulations developed to depict physical systems which allow students to alter variables and view the results.Simulations have been developed for a large number of physics and astronomy concepts (with more recently added simulations in chemistry, geoscience, biology, and mathematics) [32].Each simulation presents a particular physical system in which many things can be changed, measured, and explored.The simulations include multiple representations and provide immediate, dynamic feedback in response to user interactions.PhET simulations can serve a variety of instructional uses (e.g., classroom demonstrations, student labs, student homework).PhET began in 2002, founded by Nobel Prize winner Carl Wieman, and the team continues making new simulations today.PhET simulations are widely used and have resulted in over 200 million downloads [33].
Peer Instruction is a pedagogy that modifies a typical lecture course [22].In a Peer Instruction class, the instructor delivers a brief lecture and then poses a multiple-choice conceptual question to the class about the topic.Students have an opportunity to think and respond individually, often using "clickers."If the students' answers are varied, they have an opportunity to speak to each other (hence "Peer Instruction") to discuss their answers and convince each other of the correct answer.They then respond to the question again, and if the topic is better understood on this round of answers, the instructor moves on to the next topic.Peer Instruction has been shown to improve student learning in a variety of instructional contexts [34].It has been heavily championed by its developer, Mazur, and his research group at Harvard.Its influence is far reaching and documented in literature [1,35,36], including adoption in other disciplines besides physics (for example computer science, where other researchers have been promoting it [37]).

Data collection
To conduct in-depth investigations of the three chosen strategies, we used qualitative methods [14,18,[38][39][40].As discussed above, prior research has identified each of the three instructional strategies as being very successfully propagated.Thus, we did not seek to document the extent of propagation, but rather document the actions of the developers that led to this successful propagation.
Multiple data sources allow the researcher opportunities for triangulation, or checking one source of data against another in a chain of evidence for claims.The sources of data for the three strategies are summarized in Table III.
PLTL already had a detailed study of its dissemination, carried out and recorded in a 2008 book by Varma-Nelson and Gafney [20].This study is the product of ten years of interviews and site evaluation data.This provides the story of PLTL from its inception to institutionalization at more than one hundred other departments, in different disciplines and types of institutions.We interviewed the authors of the book for more detail on the development of the PLTL project since the research by Varma-Nelson and Gafney mostly focused on the results of implementation.One of the book authors, Varma-Nelson, was also one of the original developers of PLTL.Other data sources included the PLTL website [45] and the PLTL guidebook [30].For PhET, we conducted interviews (N ¼ 4) with members of the PhET team, including current and past faculty researchers involved with the project and a programmer involved for many years.Other sources of data included grant proposals, academic publications, presentations, YouTube videos, press releases, and other publically available information.The interviews provided an initial sense of the project in the first round of coding, and evidence from the document data was used to corroborate points made in the interviews.
The methodology used for the Peer Instruction study was similar to that of the PhET study, although there was a difference in the available document data.The PhET team was able to send grant proposals for the project, which helped outline early events.Peer Instruction, being a decade older than PhET, began in the era of paper grant proposals which were no longer accessible.However, there were hundreds of presentations made by the Mazur group that are freely available online.This data source was important in developing an understanding of the history of Peer Instruction.The sources of data were, in order of relevance, interviews with members of the Peer Instruction team (N ¼ 4), PowerPoint documents for workshop and colloquium presentations delivered over a fifteen year period (N ¼ 20), the Peer Instruction User's Manual [22], videos of talks available on YouTube, academic publications about Peer Instruction, and press releases.
Interviews were used as a primary data source in all three cases.They ranged from 30 to 90 minutes, were conducted over the phone or in-person and recorded, then transcribed verbatim.Interviews were semistructured, using a standard protocol across interviews and cases (questions included asking participants what they believed to be key events in the project's timeline and their personal involvement) but allowing for follow-up questions during the interview.Other documents were used as secondary data sources to confirm connections made from the interviews.Finally, the interviewees were asked to read over early write ups and make comments to ensure the findings represented their views and were factually accurate.When questions arose from other sources of data, interviewees were contacted via email to clarify.

Analysis
Analysis was an iterative process with multiple rounds of coding.Initial narrative analyses of these data sets are described elsewhere [52,53].This narrative provided context during the aggregate analysis phase, which aided the researchers in making connections.We note that the narratives showed there were commonalities across the three separate cases, even though they represent different degrees of change-all three were multiyear, multigrant, with distinct similar key events.
The analysis presented here for this grounded theory study focuses on propagation activities through a more detailed documentation of the actions and beliefs of the project teams.This delineation of documenting what the interviewees did and what their thoughts on those actions were is a strategy from grounded theory methodology, in which "process coding" is used to identify action words [39].The coding process began by identifying all of the actions and beliefs in each of the interview transcripts and selected documents.Actions and beliefs were coded separately, which meant they could overlap (and could subsequently be examined for co-occurrence).To be coded as an action, a segment of text needed an action clause or verb such as "we did that for a while…" or "we ran workshops there."To be coded as a belief, segments needed a clause such as "that was important because…" Doing this thorough coding of every action allowed us to characterize all the activities described, and coding the beliefs separately allowed us to draw attention to the actions associated with beliefs.Once the actions were identified from the interview transcripts (and, in the case of PLTL, published documents) they were triangulated with the secondary data sources.
Codes were developed to capture the range of actions and beliefs.Terms from the phase I codes, categories, and propositions were used when possible (e.g., "Dissemination -giving colloquia" or "Development-interviewing PLTL website [45], report [46], guidebook [30] Newsletters (N ¼ 10), website [47], "Look and Feel" document [48] Book chapter [49], websites [50,51] students").If an action or belief did not fit into an existing code, new codes were formed to capture the idea from the text.Two researchers coded several documents separately and discussed differences in coding decisions, which led to more descriptive and consistent codes.

B. Phase 2 results
The following results describe the categories developed in the coding process with codes, their definitions, and quotes to illustrate an instance of the codes.Figure 4 provides an overview of the coding system.

Overview of results
Four categories of actions emerged from the coding process (Table IV).The first two categories (development, dissemination) were carried over from phase 1.Two additional categories (funding, support) emerged when actions did not fit into the phase 1 categories.
The following sections present the results of phase 2 for the primary categories: development, dissemination, support, and funding.Within the primary categories we discuss the most prominent codes and make connections with the results from our characterization of typical practice from phase 1 of the study.Also reported are the number of times interviewees expressed a belief that the action contributed to the success of the project.Separating beliefs from actions aided analysis in drawing attention to those actions that interviewees felt were important, and conversely also allowed for more impartial analysis of the actions themselves by filtering out the beliefs and viewing the actions alone.Discussion of belief codes will be presented along with the relevant action codes.

Development
Development activities across interviews generally had heavy overlap with dissemination activities, in the form of secondary implementations.This is in contrast to the solitary development (one site only or limited sites) • The PLTL team used NSF grants to establish two consortia of institutions using PLTL.With the pool of data from these institutions, and the aid of the external evaluator, the team developed a list of "Six Critical Components" for a successful implementation of PLTL [31].
• The PhET team spent several months of developing simulations (around ten), and then used this experience to identify basic tenets of design elements [32].This includes the "look and feel" [48] of a simulation, the iterative development process (learning goals, first design, student testing and interviews, classroom use, redesigns), and the expertise required (content, programming, and education experts.)Interviews with students proved to be invaluable in interface design [32].The phenomenon of identifying formal design principles after a testing period is also seen to some extent in the development of Peer Instruction, as illustrated by the following quote: "And what we were trying to get at there was a sense of what was the range of ways in which people implemented Peer Instruction and […]-were they effective?And basically the broad conclusions we drew were that first of all it was really important that people have students both think about the questions, commit to When people had implemented and finished the year, we would even give them our slides.
[…] And we would get lots of invitations and instead of going all the time we would find whoever was in the region and say "here are the slides, go talk."(PLTL team member 1) answers on their own, and have students discuss them.There were some people who were running Peer Instruction in a sort of mode where it was sort of a quiz mode, and it was not about learning from the questions but it was sort of quizzing students with the questions, and that did not seem to be successful."(Peer Instruction team member 2).
In this quote, the team learned from other implementations what practices worked less well, leading to the identification of practices that were more successful.This mirrors the formation of critical components in PLTL in which less successful implementations were compared with successful ones: "We had not yet thought of the critical elements.We were just saying, 'What is it that is making some programs successful and others not?' and we kept looking at the group size.And there were places where they were putting in 20 students with one peer leader and that is never going to work."(PLTL team member 1).
Comparison of implementations enabled PLTL to identify reasons some implementations were successful, such as having smaller groups versus larger groups.Group size then became one of the critical components to successfully implement PLTL.
For all three cases, testing the new innovation with students and in other settings allowed the project teams to identify design principles for an ideal implementation.
Hiring personnel and changing roles.-Intertwinedwith testing, feedback, and design principles is the staff involved with doing so (31 mentions).Notably, the use of an external evaluator for PLTL and different expertise in PhET contributed to identifying design principles.Other staff, especially prominent in the case of Peer Instruction, is included in the form of graduate or postdoctoral researchers carrying out additional education research within the project (24 mentions).Often, team members take on new roles, especially leadership roles (16 mentions.)We note that in typical practice, the project may not exist long enough for roles to change.In these successful multiyear projects, though, roles often did change.This may contribute to project sustainability.
The role of research.-Researchactivities associated with the same grants or research group were coded separately from testing and feedback activities.Research is distinct from the other program activities because the purpose of education research is to discover new knowledge in science education.Testing and feedback activities for the innovation can overlap with this goal but are usually more focused on the specific context of the product rather than on generalizable knowledge.Research activities could take the form of testing specific aspects of the innovation in different contexts, or using the innovation in connection with another research question (such as how the innovation impacts certain groups of students).Three interviewees linked research activities to the success of the project, believing it helped validate the project in the eyes of potential adopters.

Dissemination
The codes given for the dissemination category in Table VI are described in the sections below.
Engaging in dissemination activities.-Projectteams engaged in a wide variety of dissemination activitiesthese are reported in a separate table of codes below in Table VII.
The propagation activities used by the teams of successfully propagated projects overlap with the propagation activities listed by PIs of typical projects-PIs also mentioned publications, talks, websites, and workshops as ways to disseminate.The difference between typical and successful practice appears to be the magnitude of those activities.
(i) Giving presentations and talks All three cases mentioned talks and presentations as main modes of dissemination.Participants from the Peer Instruction team believed presentations contributed to the successful propagation of the project.
• Peer Instruction was disseminated directly to many physics faculty through departmental colloquia.The importance of these talks in dissemination was expressed by multiple interviewees.The Peer Instruction team has given over 600 talks, many of them as invited colloquia or featured conference presentations.• The PhET Simulations also benefitted from invited talks: Carl Wieman was continually invited to speak about his research and used those talks as opportunities to discuss PhET.• PLTL encouraged faculty who had implemented it to publish and present on their experience.This allowed newcomers to the program to have a stake in it, and their publications and presentations could contribute to their careers as well as the spread of PLTL.This was the "creating scholarship and leadership" step in their model for dissemination [31].
We note that while the activity of "giving talks" is shared by typical and successful practice, there are differences in the prestige and the number of talks being given.Featured talks by Wieman reached a large number of people due to his prestigious status as a Nobel laureate.The colloquia given by the Peer Instruction team are also different dissemination vehicles than a typical conference talk for two reasons: they allow the speaker more space to tell a compelling story, and they engage the whole department at one time.These points about story helping to create change and using a department as a unit of change are both seen in the literature [54,55].The new scholars and speakers of PLTL may not have had access to prestigious venues but there were many people talking about it besides the original founding team.
(ii) Running workshops All three teams ran workshops, with varied results.
• The PLTL team ran numerous workshops-ranging from short, introductory workshops to multiday, in-depth workshops-but found that other instructors were not adopting PLTL after these experiences, despite the evidence of effectiveness and the involvement of peer leaders (undergraduate students) at the workshops to demonstrate the program.They found that people kept returning to talks and workshops without having put PLTL into place-this led to the creation of their own dissemination framework [31], in which workshops were necessary to impart information but not sufficient to promote use.• The PhET team gave workshops both locally to high school teachers and nationally to faculty, but found them to not be an efficient way to reach large numbers of people, even though they helped the individuals who attended.• The Peer Instruction team has been involved with the Physics and Astronomy New Faculty Workshop since its beginning, which helped spread awareness of Peer Instruction.Says Mazur, "Also, the New Faculty Workshop was probably responsible for triggering many invitations.Because the junior faculty would hear me at the New Faculty Workshop, get excited at my talk, and then invite me to come to their home institution." In typical practice and in the views of the NSF program directors, workshops were viewed as the best possible dissemination method.The three cases here show that they work well for training people (in the case of PhET) and getting people interested in learning more (PLTL and Peer Instruction.)However, none of the three teams were entirely satisfied with workshops alone as their main dissemination vehicle.
(iii) Hosting a website and publishing articles These methods of dissemination, mentioned by all three project teams, aligned with the methods used by PIs in phase 1.Some project team members spent considerable time preparing manuscripts attached to the research activities of the projects.PLTL was notable in encouraging secondary implementers to write about using the strategy, again as part of "encouraging scholarship and leadership."(iv) Other methods of dissemination All three teams used several creative methods of dissemination that were not mentioned by PIs or PDs in phase 1.
• The PLTL team created usable materials, such as a guidebook, and ran a "mini-grant" program.Interested institutions applied for small grants from this program to begin PLTL programs at their own institutions, which required money to pay the peer leaders.Part of the application was getting institutional or departmental support to match the amount of the grant, and agreement to submit progress reports.This involvement of the department is mirrored in literature which suggests the department is a key unit of change [55].• The PhET team found exhibit hall booths an effective way to target new audiences through selected conferences.Other dissemination methods included an online presence through Twitter, Facebook, YouTube, a blog, and a newsletter.Currently they are building a website of support materials for teachers which also will have a community component.• The Peer Instruction team produced a user's manual book [56], a DVD, and is currently building training modules for users online.Like PhET, they have an online presence through a blog and community website [57].
The common thread between these extra methods is creating supporting materials.These support materials went beyond the main support identified by typical PIs, which was to write a textbook.This is discussed below in the section describing the support category.
Spread into new audiences and interaction with publishers.-Projectteams frequently mentioned new audiences as key events in their projects.These included international audiences through visits, spreading into different disciplines, different grade levels, or translating existing products for greater access.
Each project team had interactions with publishers, with varied results.
• PhET actively reached out to target communities by having an exhibit booth at different conferences.
"..We ran exhibitor booths and we would tell people about PhET, and that really got out to a lot of people… we got to a point where we never ran into anybody who never heard of PhET, everybody had heard of PhET and was already using PhET, so we weren't getting any new people."-PhETteam member 1.
• PhET also pushed into a broader audience through the inclusion of the PhET simulations in major textbooks, regardless of publisher, due to the decision to make the license for the simulations attribution only.This means publishers can print and use the simulations for free as long as they give credit.PhET also spread through the content addressed by the simulations as it branched from introductory physics concepts, to more advanced physics concepts, and eventually into other disciplines, beginning with chemistry [58].• Peer Instruction benefitted from the involvement of the publisher, who freely distributed the Peer Instruction User's Guide to interested faculty in the first years of the book.
Creating philosophy of dissemination.-All three teams, but particularly PLTL and PhET, held beliefs regarding how dissemination and change work and acted on those beliefs, either by writing them down (much like the list of design principles of the product itself mentioned above), or building and presenting the product in alignment with their beliefs.
• The PLTL team explicitly created a four-step strategy for dissemination.They found that people kept returning to talks and workshops without having put PLTL into place.Thinking about the problem, they developed their own dissemination strategy with four steps [31] The goal of "Successful implementation" was to put PLTL into place at a new institution.Finally, "developing scholarship and new leadership" encouraged faculty to publish and present on their experience implementing PLTL.• The PhET team intentionally makes design decisions about the simulations with the goal to keep them accessible and adaptable.
Honing presentation.-ThePLTL and Peer Instruction teams discussed ways in which they changed their dissemination materials, such as how they delivered talks and workshops, over time as they did these activities.For Peer Instruction, this was mostly how talks were delivered, for example, building in some discussion of barriers: "My talk initially had focused mostly on the "why," not on the "how."It became clear that my "why" was very compelling.I convinced a lot of people of the need to teach interactively.But then, if I did just that, they were left with so many questions that there was always a need for a follow-up.So now, I started to build that in." (Peer Instruction team member 1).
The PLTL workshops changed once the Six Critical Components existed to incorporate them as key points required for implementation.

Support
Support activities were mentioned less frequently than development and dissemination activities, and interviewees tended not to express strong beliefs about their efficacy.Like the categories of development and dissemination, support activities can overlap with other activities.The codes from Table VIII are described more fully in the sections below.
Creating teacher support or resources.-Asmentioned above in the dissemination section, all three project teams created some kind of teacher support or materials, especially ready-made materials.• As mentioned above, the PLTL team developed many supporting materials for interested instructors, such as the guidebook and other short guides to implementation.The guidebook [30] includes details about how to run the program, how to select and train leaders for the program, and it provides example problems formats for group work.The book includes testimonials from peer leaders and instructors on the positive impacts of PLTL.• The PhET website already has numerous activities for teachers surrounding individual simulations (which teachers can upload and contribute to), but the team is also making a new website for teachers, using their body of research on how to best use the simulations with students.This website (in production) also has a goal of building a community of teachers.• The Peer Instruction team has a community website with some materials explaining what Peer Instruction is and tips for successful implementation.A separate group of computer science educators have created a support website for Peer Instruction in a computer science setting, with advice for implementation and slides of material for specific courses [50].We note that some of these activities (particularly the original Peer Instruction website) were viewed by the team members as activities that were not very successful, or could be successful with more funding and manpower to foster a community of instructors adding their own resources.
Addressing barriers to adoption.-Thisaction was mostly undertaken by the PLTL team.They learned from interacting with interested potential adopters that the program seemed too challenging to put into place, as illustrated by the following quote: "…and faculty from other places would say, you know, it sounds good, but it also sounds like the invasion of Normandy, and it's just too much, especially in the case where you know, they might have 400 freshmen in an introductory chemistry class."(PLTL team member 2).
To address these barriers they developed their own dissemination plan (involving direct assistance to implement the program at a new institution) and the mini-grant program, as one of the biggest barriers mentioned by potential adopters was the funds required to pay the peer leaders.
Direct training and site visits.-Oneactivity undertaken by the PLTL and Peer Instruction teams was going to a site directly or allowing outside instructors to come to their institution and learn about the innovation.
• The Peer Instruction team would travel frequently to other institutions for colloquia or occasionally more in-depth interactions.One interview described hosting instructors from another institution at Harvard so they could learn to do it directly from them, then flying out to visit them later to assist in the implementation.• The PLTL team includes "successful implementation" as step three in their dissemination plan, or to directly aid new implementers to start their own programs.
Maintaining product over time.-Thisactivity was mainly seen in the PhET interviews.The team was concerned the pace of technological developments might make simulations unusable, and were looking ahead to different platforms.This was a different activity than other support mechanisms seen in the other two cases, specific to the computer-based nature of PhET.

Funding
The funding category (Table IX) is slightly different from the others.While the others have occasional overlap, funding was integral to every activity undertaken by the teams.Below is a table (Table X) that presents the top action codes overlapping with "uses for funding."Funding was used in development (through paying secondary development sites, and in research efforts, and personnel), dissemination (funds for a website, workshops, travel), and support (funds for travel, publishing materials).
We note that the teams were interested in creating new mechanisms for funding besides grants to continue the projects.

C. Phase 2 conclusions
The three cases provided an in-depth look at the activities undertaken by project teams behind well-propagated strategies.The goal of this phase was to understand successful propagation practice.Four additional propositions were developed from these cases.
• Proposition 5: In successful projects, testing and feedback are used to identify the design components.
Although PhET and PLTL are very different types of innovations, they shared a common pilot testing stage in their projects that led to explicitly stating the design principles necessary for successful use.This suggests the transferability of these findings to other innovations regardless of how easy or difficult they are to implement.
• Proposition 6: Successful projects engage in traditional mass-media dissemination.In contrast to typical projects they do traditional dissemination at a large scale and also have more interactive and creative dissemination mechanisms.
The three project teams also aligned with the typical PIs regarding the types of dissemination activities undertaken, but found ways of making talks, in particular, more effective for their innovation.The ways each team gave talks about their projects were unique (PhET benefitted from large-scale invited talks, Peer Instruction from hundreds of departmental colloquia, and PLTL from creating new leadership to give more talks about the project.)One point that aligns with change literature is the impact of an opinion leader [10], which PhET and Peer Instruction both had.Further, they came up with new ways to disseminate, such as producing a DVD, having an exhibit booth, or using a large dissemination grant to run a minigrant program.
• Proposition 7: Successful projects realize that users need support to be successful.All projects studied continue to struggle to identify ways to best support users.
Through their experiences of interacting with possible adopters, the teams of all projects began to realize that additional support was needed in order for adopters to be successful.Each team discussed efforts to develop support mechanisms.Many of these mechanisms were not highly successful.
• Proposition 8: Successful projects received continuous funding over an extended period of time.
Finally, funding was involved in every category and used in a variety of ways.Funding was involved with development, dissemination, and support activities.This finding is consistent with the view of the program directors that many of the initiatives they recalled as having an impact received funding (over ten years or more) over a series of related grants.

VI. CONCLUSIONS
We synthesized the propositions from phases 1 and 2 (summarized in Table XI) in the form of a model for effective propagation activities (Fig. 5), which educational developers can apply to create a strong transferrable product that is likely to propagate.As discussed in this paper, the model is derived from a survey of principal investigators, focus group interviews with NSF program directors, and context-specific examples of well-propagated instructional strategies in physics and chemistry.Although we expect the general structure of the model to remain, phase 3 of the study will use interviews from more PIs of successful innovations in more disciplines and contexts and may result in further elaboration of the model to provide context-specific advice for education developers within each stage.

A. Description of the model
The model of designing for successful propagation resulting from this study has three core propagation activities: interactive development, interactive dissemination, and support of adopters.Each of these stages is interconnected with the other stages and each stage relies on funding.The following sections use examples from phases 1 and 2 to describe each of the three stages.

Interactive development
Often, a new teaching strategy is developed in a single context.Developers may discuss the strategy with others in their department or even have a local collaborator.However, there is little or no interaction with potential adopters at other institutions.Phase 2 propositions 5.In successful projects, testing and feedback are used to identify the design components.6. Successful projects engage in traditional mass-media dissemination.In contrast to typical projects, they do traditional dissemination at a large scale and also have more interactive and creative dissemination mechanisms.7. Successful projects realize that users need support to be successful.All projects continue to struggle to identify ways to best support users.8. Successful projects received continuous funding over an extended period of time.
Instead, this model suggests that successful developers create and refine a new instructional strategy with collaborators and (or) potential adopters.Interactive development involves getting feedback from the beginning, possibly a trial period collecting data from other implementations to strengthen the innovation, and articulating product design principles.With other people involved from the start, there are more stakeholders in the new innovation and also more opportunities for feedback to strengthen the product.
For example, the PhET Simulations accomplished interactive development through student interviews at their own institution and hiring staff in different areas, such as interface design, content expertise, and pedagogy expertise.As the program went on, collaborations were formed with other science content personnel (biology, chemistry, etc.) and other institutions.Peer Instruction, too, used student interviews to inform questions for use in class along with graduate students, post docs, and early collaborations with another institution.For both of these strategies interactive development was accomplished primarily at their home institutions through the involvement of many people with different perspectives.This is in contrast to PLTL, which involved over ten institutions in interactive development resulting in having a solid format that worked across settings.

Interactive dissemination
Typical dissemination practice uses academic channels, such as journal articles and conference presentations, almost exclusively.These activities, at best, raise awareness but generally do not result in wide propagation [3,7].
Interactive dissemination means engaging the target audience, through immersive workshops and personal connections.This aligns with ideas in the change literature that emphasize the role of interpersonal communication channels [26] and the necessity of thinking about change as a process in which potential adopters require different communication messages at different times [59].Offering an interactive multiday workshop is an example of interactive dissemination.
Peer-Led Team Learning offered workshops to impart information about the program.Interested instructors attended workshops but encountered barriers implementing the program at their own institution.Recognizing this problem, the team conducted a "mini-grant" program with funds from a large NSF dissemination grant, offering financial support to institutions who wrote proposals that showed they had institutional matching of funds if they received the mini-grant.The mini-grant program stimulated significant interaction between the developers and potential users.
Peer Instruction and the PhET Simulations both benefitted from members of the project team giving numerous invited talks, but also ran workshops.Peer Instruction was spread through sheer quantity of departmental colloquia given as well as involvement in national workshops.Colloquia generated opportunities for personal interactions with people in many institutions.
The PhET simulations team held local workshops for teachers and has open resources to use simulations that instructors can add to.Although these activities were not regarded as successful in spreading PhET by the team, they do show efforts to create a community of teachers using PhET.The team did think of the exhibit hall booths as being successful in letting instructors at targeted conferences know that PhET exists, and doing so through a brief personal interaction at those communities again speaks to a level of interactivity in the team's dissemination efforts.

Supporting adopters
Support is less well understood in the context of higher education than the previous two stages.Education developers do not typically support potential adopters after initial adoption.However, this was seen as a problem by interviewees associated with the successful projects and is also reflected in the change literature [7].The model suggests that support is necessary to provide assistance to adopters as they attempt to use the innovation.An example of support is people being available to consult after a potential adopter starts using an innovation.
The Peer Instruction team offers direct support to potential users, and in recent years has shifted talks about the pedagogy to include more discussion of the barriers instructors may face putting it into place.Both PLTL and Peer Instruction developed user's guides to give adopters more details and advice on how to implement the strategy; the PhET Simulations, too, offer lesson plans for specific simulations.The PhET Simulations place emphasis on accessibility, as seen in their licensing decision (which allows textbooks to use them free) and in their efforts to keep the simulations running on current technology.More research is needed to identify a broader range of productive support mechanisms.

B. Future work
The third phase of this study will use purposeful sampling to further test and potentially expand the model.As an example of how we will use purposeful sampling, consider the potential role of prestigious project leadership in successful propagation.
Two of the three cases discussed above, PhET and Peer Instruction, had the benefit of prestigious leadership (PhET was started by a Nobel laureate, Peer Instruction by a successful Harvard physics professor).Change literature suggests that prestige can be a factor in decisions to adopt a new innovation [10].In our study we found that, in these cases, frequent well-attended talks, colloquia, and workshops were key methods of informing the physics community that these innovations existed, and prestige may have allowed for access to these platforms.However, it is also clear that prestige is not necessary for successful propagation: the case of PLTL shows that nonprestigious leadership can propagate a new innovation successfully.
To unpack the role of prestige, whether it allowed for greater access to dissemination mechanisms or itself was necessary for others to adopt innovations, the next phase of research will seek confirming and disconfirming cases.We will investigate (i) typical PIs at the end of their funding period to see if they undertake similar dissemination activities as those mentioned here and (ii) cases of other successful (but less prestigious) PIs to further add and compare to the current model.

C. Implications for education developers
Although we plan to further test and refine the model of successful propagation in phase 3, we feel that there is sufficient strength in the core aspects of the model to allow us to draw three broad implications for education developers.
(i) Implication 1 Education developers typically seek to develop a polished product before getting feedback.Instead, early feedback on incomplete products from potential users is crucial to building a strong product based on explicit design principles.
Developing an innovation with potential adopters (interactive development) increases the likelihood a stronger product.This matches a point from the literature: Many physics instructors are genuinely interested in changing their teaching practices and using novel materials [60].Researchers often expect instructors to adopt finished products without engaging those instructors in the development, and then blame the instructors for making changes in the strategy during implementation [25].Our findings suggest that enlisting interested potential adopters early in the development of new teaching strategies and materials is key in building a strong product with explicit design principles.
(ii) Implication 2 Typical education developers rely on communication channels such as journal articles, conference presentations, and websites for dissemination.Successful developers also do these things but, in addition, find ways to interact with potential adopters and allow potential adopters to interact with their product.
Academic articles and presentations are necessary components of being an academic, and they provide an authoritative source for others to cite.Having research and evidence of student improvement is probably necessary but certainly not sufficient to convince others to adopt an innovation.This is exemplified by workshop attendees of PLTL who needed additional support to actually start PLTL at their institutions.There are other ways of disseminating information, and our model suggests that the more interactive, the more effective.Some suggestions from the three successful strategies: running workshops, using personal connections with other instructors, buying an exhibit booth at conferences, and using invited colloquia to convey a motivational story, with plenty of time for interactions afterwards.(iii) Implication 3 Typical education developers do not plan for supporting adopters during implementation.Successful developers learn about barriers to use through their interactions with people who have tried their product.Successful developers realize that adopters need support during implementation.
Research shows that while instructors are interested in using new instructional strategies, and often try new strategies, they also often discontinue use [7].Supporting adopters can help assuage the problem of discontinuation, as seen in the three cases we studied.In each case the team continues efforts to work with adopters long after the development period.Peer Instruction and PLTL are both over twenty years old and their teams continue to promote them.Support can take place by providing ready-made materials that other instructors can modify, rather than have to develop from scratch (e.g., PhET teacher activities, PLTL guidebook, Peer Instruction User's Guide).It can also take a more direct form of spending time or money on assisting new implementers.
Interactive development, interactive dissemination, and support of adopters are not typical practice of educational developers, yet appear necessary for successful propagation.The model of designing for successful propagation emerging from this work provides a guide for education developers to construct propagation plans likely to lead to sustained adoption of education innovations.

FIG. 1 .
FIG. 1. Overview of the larger study, carried out over five years.This paper describes phases 1 and 2, which have resulted in a tentative model of educational development for successful propagation.

FIG. 5 .
FIG. 5. Tentative model of designing for successful propagation resulting from phases 1 and 2 of this study.The model has three core propagation activities to be undertaken by the project team: interactive development, interactive dissemination, and support of adopters.The model emphasizes both the roles of three feedback loops that interconnect different stages and the role of funding for each phase.

TABLE I .
[23]ple PI responses to question 7 and coding, adapted from Ref.[23].Barriersto propagation.-Programdirectors identified several general barriers to propagation.Three barriers were most prominent in the codes: • Many innovations do not have sufficient evidence of efficacy to convince STEM instructors to adopt them.Program directors felt that projects should pay more attention to collecting evidence of efficacy.For example, in the biology and engineering discussions, program directors believed faculty want to see demonstrated effectiveness, and see efficacy as a reason to change practice.The mathematics program directors believed more beta sites would help with adoption, because then more than one site is involved and there is more evidence of effectiveness.• Many disciplines do not have professional societies that promote innovative teaching.Some disciplines (e.g., physics) have centralized, influential professional societies that aid in dissemination, but this should occur in more disciplines.
ness.However, workshops require money and skilled facilitators, both of which are in short supply.•Collaborating institutions.Program directors believed projects involving multiple institutions are more likely to be successfully propagated.Collaborating institutions provide beta sites to assess the innovation during development and also increase the number of faculty with a stake in the work, which helps sustain and propagate the innovation.• Extended funding period.Many of the most successfully propagated projects received funding over an extended period (often 10 years or more) through a series of grants.They mentioned projects that moved from type 1 to type 2 grants, allowing for more dissemination efforts.However, they also acknowledged a tension within NSF between providing additional funding for existing projects and providing funding for new projects.• Professional societies.Program directors believed professional societies can play key roles in propagation, by promoting specific innovations or, as in the case of physics, hosting programs such as the New Faculty Workshop to expose many potential adopters to innovations.

TABLE II .
Descriptions of the types of change required of potential adopters to adopt the three cases discussed.

TABLE III .
Summary of data sources used in the studies of well-propagated strategies.

TABLE IV .
Categories used in coding interviews from the three well-propagated strategies.Each main category contains subcodes.
4IG.4.Overview of coding system hierarchy, with the four main categories on top and frequently mentioned codes associated with each category.observed in the typical practice of PIs.The codes listed in Table V are described below.

TABLE V .
Codes and example quotes for the development category.The counts represent the number of times an action was coded and the number of times the subcode belief "contribution to success" associated with that action was coded.

TABLE VII .
Frequently used subcodes and example quotes for the dissemination strategies category.

TABLE VI .
Frequently used codes and example quotes for the dissemination category.
: 1. Stimulating interest, 2. creating a deeper understanding, 3. successful implementation, 4. developing scholarship and new leadership."Stimulating interest" is accomplished through publications and presentations."Creating a deeper understanding" can be accomplished through workshops, as the goal is to educate newcomers in the details of the program.

TABLE VIII .
Frequently used codes and example quotes for the support category.

TABLE IX .
Frequently used codes and example quotes for the funding category.9

TABLE X .
Codes co-occurring with the code "uses for funding."

TABLE XI .
Summary of knowledge propositions based on results of phases one and two of the study.PIs focus most of their attention on product development.Dissemination occurs after development, if at all. 2. PIs think of dissemination in terms of impersonal, "mass-media" mechanisms such as publishing papers or giving conference talks.3. Program directors think about education development projects as a development phase followed by a dissemination phase.4. Program directors strongly emphasize the importance of interactivity in both development and dissemination.