Design and application of a framework for examining the beliefs and practices of physics teaching assistants

[This paper is part of the Focused Collection on Preparing and Supporting University Physics Educators.] We present a newly validated and refined framework, TA-PIVOT (TA Practices In and Views Of Teaching), for examining how physics TAs talk about and how they engage in physics teaching. This work builds upon and extends prior efforts to characterize instructors’ beliefs and practices by examining both domains in parallel. We present the comprehensive framework (developed from a study of 31 total TAs) and demonstrate its utility in analyzing both interviews and classroom video observations for a sample of eight TAs. We also discuss how this framework may be used to examine variation in beliefs and practices, track the development of beliefs over time, and inform TA preparation.


I. INTRODUCTION
National calls for science education reform have led to widespread changes to physics instruction at the introductory level [1,2].As we modify courses to promote students' active engagement in the learning process and support the development of scientific reasoning [3] and mastery of conceptual physics content [4], we must attend to how instructors in these environments conceptualize and enact their own roles in supporting these goals [5].One category of educational environments common to introductory physics courses is the smaller, discussion-based sections that are intended to complement lecture and are primarily led by graduate Teaching Assistants (TAs).In recent years these sections have become more student centered through the adoption of research-based curricula, such as the Tutorials in Introductory Physics [6].Because graduate TAs are partly responsible for establishing classroom norms for student participation in these transformed environments, the instructional beliefs and practices of TAs have become the subject of increasing attention in physics education research [7][8][9].
We choose to focus on graduate TAs for a number of reasons.First, TAs impact student learning-there is documented correlation between TAs' practices, such as the use of Socratic dialogue, and their students' scores on conceptual postassessments [9,10].Second, as graduate students, TAs are in the process of reconciling their overlapping roles as both teacher and student.They are being exposed to (and expected to participate in) multiple types of educational environments; these instructional strategies they are expected to support as TAs may be vastly different from those they experienced (and continue to experience) as students.This overlap in roles as student and teacher affords a rich opportunity to examine how TAs make sense of teaching and learning at a critical time in their careers.
Finally, we must also recognize that the graduate TAs of today are the faculty of tomorrow, who will be teaching courses, designing and implementing curricula, and working with future graduate and undergraduate students [11].In the absence of formal pedagogical training, the primary opportunity for TAs to develop and refine teaching beliefs is through their experiences in the classroom and any accompanying weekly preparation sessions.In many cases, physicists are not required to engage in pedagogical training opportunities after leaving grad school.Therefore, the potential for long-term impact on the beliefs of future faculty encourages us to examine how TA experiences shape the ways in which physicists talk about and carry out their roles as instructors.
Here we build on and extend preliminary work [12] to provide a framework for us as researchers to link the domains of stated beliefs (how TAs talk about instruction) and practices (how they act in instructional environments), which we call TA-PIVOT (Teaching Assistants' Practices In & Views Of Teaching).In prior work, we demonstrated the potential utility of this framework by applying it to example data without going into significant detail of the coding rubric [13].In this work, we present the entire framework, describe its development, and give specific examples of how it is applied.

II. PRIOR RESEARCH A. Nature of instructor beliefs
The ways in which teachers think about what goes on in the classroom and how they conceptualize their roles in working with students are the subjects of broad attention in education research, spanning preservice K-12 teachers to university faculty [14][15][16][17][18][19].Numerous studies have investigated how teachers develop an understanding of the nature of teaching, as well as how such an understanding relates to their instructional practices.As noted by others [14,15], the overall coherence of these studies is inhibited by a general lack of agreement and consistent use of terminology such as beliefs, conceptions, perspectives, attitudes, and values (just to name a few).However, efforts to synthesize prior studies have yielded useful sets of assumptions upon which subsequent studies may be based [14,16].In particular, "beliefs" are generally understood to be tacitly held [17,18], have some connective structure [19], and appear critical to how people define, interpret, and make decisions regarding their behaviors in social settings [16].In this study, we have chosen to employ the construct of beliefs, and we use it to mean tacit, personally held values and assumptions that shape how teachers interpret classroom situations.In other words, beliefs serve as a lens through which TAs view classroom situations.
In considering instructors' beliefs, we emphasize their contextual nature.Just as student learning of physics must be considered within nested levels of particular contexts [20,21], we must also consider how beliefs are supported, constrained, and shaped by local context.Hoyles [22] describes all beliefs as being situated as a result of being constructed within particular contexts, and Pajares [14] similarly emphasizes the context-specific nature of beliefs.This context dependence will become particularly important in comparing beliefs and practices or considering the "transfer" of beliefs between classroom environments, since different contextual factors may tend to elicit different beliefs.

B. Relationship to practices
A natural complement to how teachers think about their own teaching is how they enact their roles as teachers in the classroom.Numerous studies have indicated that instructors' beliefs about teaching influence their classroom practices [23], although the exact relationship between beliefs and practices is complex and not a "one-to-one" mapping [24].This finding underscores the danger of ascribing "enacted belief" labels to classroom practices, as the practitioner and researcher may not agree on the belief reflected by a particular practice.
Furthermore, the relationship between beliefs and practices is mediated by local constraints, and such constraints may weaken the coordination between them [23].(For clarity, we will use the term coordination when comparing beliefs with practices, and alignment when comparing beliefs or practices with the curricular approach.)One model for this relationship is that instructors' beliefs and practices must be considered as being situated within local context, which mediates the relationship between them.Prior studies have indicated that faculty operating within similar instructional settings negotiate situational constraints differently, and their ability to do so is reflective of the sophistication of their overall teaching model [5].
In investigating the connection between beliefs and practices, we are building upon the approach of recent studies of math and physics TAs that employ a combination of video observations and interviews to provide a more complete picture of their beliefs and practices [24,25].
It is tempting to use teachers' classroom practices as a proxy for their beliefs, and vice versa.However, in the words of Kane et al. [26], considering only teachers' talk amounts to "telling half the story" and limits our ability to investigate the nature and contextual dependence of instructors' pedagogical beliefs.Instead, we must consider both how teachers describe their views of teaching and enact practices in educational settings.In other words, talking to TAs and asking them to describe what matters to them in enacting their role as teachers allows us as researchers to more completely understand the decision-making process that TAs undergo in the classroom.

C. Development of instruments
Extensive research has been conducted examining the beliefs and practices of pre-and inservice K-12 teachers and university faculty.To this end, numerous instruments have been designed for documenting instructors' use of particular practices, for example, the Reformed Teaching Observation Protocol (RTOP) [27], Teaching Dimensions Observation Protocol (TDOP) [28], and Classroom Observation Protocol for Undergraduate STEM (COPUS) [29].Likewise, frameworks exist to characterize instructors' views on the nature of teaching, usually through the use of interviews and surveys [30][31][32].Previous research on university academics' teaching has tended to focus on either interview or observational data alone, one as a proxy for the other.In particular, there is not an existing framework that spans the contexts of beliefs and practices in order to examine similar constructs within each domain.

A. Motivation
Our interest in outlining a framework for bridging beliefs and practices is built upon a number of objectives.First, such a framework would afford greater insight into the nature of beliefs and practices as situated within particular educational environments.Second, it would allow us to uncover and examine instances of consistency and inconsistency between beliefs and practices, and allow us to consider the factors contributing to these instances.Additionally, examining the nature of coordination between beliefs and practices allows us to assess the bounds of proxy; that is, we may be able to tell to what extent and under what conditions measured beliefs may be substituted for practices.As noted earlier, a particular tendency in the literature is to ascribe beliefs to particular practices and vice versa; a greater understanding of coordination would in turn streamline future data collection in these settings by providing an empirical basis for the substitution of survey or interview responses for direct observation of practices.
On the more applied side, the creation of such a framework is a critical step in documenting and transforming TA preparation and professional development.There is a recognized need for investigating the effectiveness of existing preparation methods in helping TAs develop and refine productive pedagogical beliefs and practices [33].Previous research has demonstrated that key factors in the ability of physics faculty to persist in using research-based course transformations include the following: (a) an awareness of course transformations, (b) a belief in the effectiveness of course transformations, and (c) an awareness of barriers or constraints to the implementation of course transformations [34].Furthermore, instructors' ability to negotiate such barriers appears to be grounded in their broader teaching perspective [5].
The research questions that guided the development of this framework were the following: (1) Can we design an instrument that is able to distinguish stated beliefs and practices for physics TAs? (2) Do we observe instances of coordination and/or discoordination between TAs' beliefs and practices?(3) Can we observe shifts in individual TAs' stated beliefs over the course of a semester?In this paper, we focus primarily on the first two questions, and will demonstrate the potential of this framework to address the issue of dynamics posed by the third.

B. Methods
The development of the TA-PIVOT framework originally began with preliminary end-of-semester written surveys designed to elicit TAs' definitions of teaching, as well as their self-reported reflections on how their ideas about teaching changed during the semester.Responses to these surveys tended to be brief, but nonetheless provided emergent themes that informed the design of a subsequent interview protocol.
Drawing from other interview approaches [8,25], we then designed an interview protocol that would allow TAs to elaborate on their responses and, in particular, describe specific teaching instances in greater detail.The interviews (with participant consent) were approximately 40 min in length and semistructured in that the interviewer could ask to clarify or follow up on specific points at his discretion.
The protocol also included open-ended questions to identify broad themes (e.g., "What is teaching, to you?"), as well as opportunities to reflect on specific classroom events.The complete interview protocol is listed in Appendix A.
In parallel with the interviews, we collected classroom recordings using fixed cameras trained on tables of about four students, in a manner similar to the fixed-camera observational studies conducted by the University of Maryland [35].Students and TAs were informed beforehand when recordings were taking place, and consent was obtained in accordance with campus human subjects protocols.Researchers were not present in the room during recording.We then identified and extracted video clips in which the TA was present at the table (which we will refer to as "episodes"), which were then coded as described below.
We conducted two rounds of data collection, first with three TAs and then with an additional five TAs in a subsequent semester.In the first (pilot) round, we videotaped TAs during their week 7 tutorial sections and conducted postsemester interviews with each one.In the second (expanded) round, we videotaped five weeks of the TAs' interaction with students (week 4, forces; week 7, work and changes in kinetic energy; week 9, conservation of momentum; week 12, conservation of angular momentum; and week 14, buoyancy).These recordings were conducted using the same fixed-camera methods described previously.Additionally, in order to characterize TAs' beliefs throughout the semester, we conducted pre-, mid-, and postsemester interviews using the same protocol.
The first interviews took place after classes had started, so the TAs already had at least one week of teaching to draw upon.Furthermore, these interviews took place after each of the participants had completed the presemester teaching orientation.Hence, we do not claim that the first interviews reflect the TAs' initial "state" before entering the physics program.It was logistically difficult to arrange interviews with TAs before the orientation, and we were ultimately more interested in examining the ongoing impact of teaching in a transformed learning environment than the effect of a single workshop.

C. Setting and participants
Our complete data inventory (spanning multiple semesters) includes 13 TAs who participated in both interviews and videotapings; an additional 18 TAs were interviewed but not videotaped.All of the videotaped TAs served in tutorial sections of the first-semester calculusbased mechanics course (Physics 1) at the University of Colorado (CU), described in detail in Finkelstein and Pollock [36].TAs in this course lead 1 hour weekly tutorial sections of about 30 students seated in small groups.The TAs facilitate group discussion and engage the students in conversation about their responses and reasoning.
In this paper we focus on 8 first-year TAs teaching Physics 1 tutorials (Table I).Three of these TAs, Sarah, Daniel, and Keith (all names are pseudonyms), were pilot study participants serving their second semester as TAs, and none of them had formal teaching experience before their first experiences as TAs the previous fall.In the previous semester, Sarah had also taught Physics 1 (Mechanics) whereas Daniel and Keith had taught Physics 2 (Electricity and Magnetism), the subsequent course in the series, also using tutorials.
The remaining five TAs were part of the expanded follow-up study and thus participated in additional interviews and videotapings.Darren and Joanna had both served as TAs as undergraduates, with Joanna having taught in a SCALE-UP [37] setting with the UW tutorials.Terrence served as a learning assistant as an undergraduate, although he decided not to participate in the accompanying pedagogy course because was busy preparing for the GRE and writing an undergraduate thesis.Oliver was an international graduate student who originally attended CU as part of an undergraduate research program, and was then accepted to the graduate program.In addition to serving as a college physics tutor, Caleb had also served as a K-8 reading tutor in a foreign country during college.
Over the course of the study, limited interviews and videotapings were also conducted with TAs in laboratory settings of other algebra-based introductory courses for comparative purposes and to test the reliability of the framework across classroom settings.We also collected supplementary data including interviews and videotape of undergraduate learning assistants who worked alongside TAs in tutorial settings [38], early surveys that informed the development of the interview protocol, and field notes from TA preparation sessions.These additional data informed the study but are not reported on here.

Coding of both interview and video transcripts began
with initial open coding to identify recurring themes, patterns of speech, and behaviors [39].The unit of coding analysis tended to be individual statements in interviews, and turns of talk in observed practices.Codes were assigned until both researchers were satisfied that no emergent themes could be identified in the collected data; the assigned codes were then sorted into broader categories that could be compared against existing theoretical models in the literature.Statements could receive multiple codes if applicable, as will be illustrated in the next section.The final coding scheme constitutes the finest grain size of the framework described below.
We used a coding software package (NVivo) to analyze both the interview and video data according to the developed rubric (Tables III, IV, and V in Appendix B).Coding was performed by a single researcher.In order to assess reliability of the coding schema before proceeding with coding of the full data set, we recruited another researcher to receive 20 min of training before coding two 40 min TA interview excerpts.The researchers initially agreed on 83% of coded statements, which rose to 95% following a thorough discussion of the rubric and subsequent clarification of codes.A similar process was carried out for video coding: A second researcher was trained in the coding procedure and then given a 3.5 min segment of video to code.After initial coding, the raters agreed on 78% of codes.Following discussion and revision, the level of agreement rose to 97%.

IV. RESULTS: FRAMEWORK
The TA-PIVOT framework has three overall dimensions (agency, goal, and assessment), along with subdimensions for each area (shown in Table II).These dimensions are interconnected and may serve as anchors for an instructor's overall perspective of teaching and learning.Each subdimension is the result of iterative open coding along with consultation of existing literature.
This paper is not meant to serve as an exhaustive case study, but rather as a demonstration of the utility of the framework.In order to introduce the framework and the associated coding schema, we will start by considering representative quotations drawn from TA interviews.We then follow with illustrative vignettes to examine the same TAs' instructional practices, along with summaries of their overall observed practices across a week of episodes (31 total videos).In Sec.IV, we will extend this framework to examine the overall consistency of beliefs and practices, highlight variation among an expanded set of 8 TAs, and consider shifts in beliefs over the course of a semester.

A. Agency
Agency is defined in the literature as the socioculturally mediated capacity to act [40].In examining agency, we are concerned with who is intended to be active and influential in the learning process.We recognize that in the environment under consideration, both TAs and students are constrained in their possible modes of participation, and thus our definitions of agency are therefore subject to local context.For example, students and TAs are not free to choose the content of the tutorial they work on each week, nor whether or not they use tutorials at all.However, we argue that within this classroom setting there is sufficient latitude for instructors to establish norms for participation through their use of discourse and tools.In doing so, we build on prior work that demonstrates how instructors' implementation of Peer Instruction [41] influences students' perception of classroom norms [5].
Generally speaking, we use this dimension to examine whose contributions are valued and given attention during a classroom interaction.For example, in an interview, one TA stated that his role should be to "step in and tell them how I would do it," which would be coded as teacher agency because it places the clear value on TAs' ideas over the students'.In contrast, a statement that "students should work collaboratively and the TA should not interfere" was coded as student agency because it downplays the TA's role while emphasizing students' collective action.Statements that were coded as mixed involved mutual TA-student action or did not give a clear indication whose contributions were preferred.
In describing her approach to teaching during an interview, Sarah places an emphasis on supporting groups in working together to complete the tutorial.Her role as a teacher is "facilitating [the students'] discussion" and she describes how she is careful not to interrupt an ongoing student discussion.She does note that there are times when the teacher should explain or provide examples, but says that it is best when such ideas are student generated.The following quote reflects Sarah's interest in listening to student ideas: I think the approach of listening to [the students'] conversation was good because it lets them kind of put out a whole bunch of different ideas, different ways of thinking it, instead of just supposing your way of thinking about it immediately.[Sarah post, 00:10:39] In contrast, Daniel and Keith's beliefs reflect a strong emphasis on their own agency in a student's learning process.Daniel describes how students should initially work together in struggling through challenging problems, but his role is then to "tell them how I think about it, so they can see how my mind works."Furthermore, Daniel believes it is important for the TA to know the material "cold" because if he needs to stop and think in front of the students, then they will stop paying attention.Daniel describes himself as "not interested" in hearing students' incorrect responses, preferring to step in with a correct explanation: Similarly to Daniel, Keith states that a primary role of the TA is answering student questions.He expresses surprise that some of the learning assistants he worked with were not always able to answer student questions: I guess, initially I was expecting [the LAs] to be able to answer all the questions the students had, and then I realized, well, maybe that's not their job, thats the TA's job.[Keith post, 00:37:20] Figure 1 compares the coding results for each TA's beliefs along the dimension of agency.We present absolute counts of codes for agency during the interview session.Using Fisher's exact test, Sarah's distribution of codes are statistically different from both Daniel's (p < 0.01) and Keith's (p < 0.01); Daniel and Keith's distributions are not statistically different from one another.

B. Goal of instruction
Goals and intended outcomes are a widely recognized component of instructor beliefs [42].This dimension corresponds to the intended outcome of the learning process, and represents a condensed version of the taxonomy of the cognitive domain drawn from Anderson [43].Within this dimension, we separate instructors' contentbased goals from their epistemological stance toward answer making or sense making [5].
Although the tutorials themselves are conceptually focused, other forms of knowledge are emphasized in various other course components.For instance, weekly homework sets assigned through the CAPA online homework system [44] emphasize traditional physics problem solving with numerical answers and no prompt for explanation.TAs may choose to integrate these other cognitive levels within the tutorial setting, or keep them separate.
Keith's goals for his students appear to be a focal component of his beliefs about teaching, as he spent a greater fraction of his interview time than other TAs describing what he wants students to learn about physics.
Here is how he responds to a question about what he finds most and least useful about the UW tutorials: Sometimes I feel like the way [the tutorials] introduce the equations, or prompted the students to write down equations or start doing math was not the best way of doing it… [The students] just weren't in the mode where they were kind of thinking about these physics equations, and it was always hard to get people to recall.You know, I'd be like, "This makes sense because, well, what is momentum equal to, in terms of mass and velocity?"They'd be like, "Ummm," they'd have a hard time thinking about that kind of stuff.I guess I feel like having more problem solving and mathematics kind of informs the intuition and vice versa, it's good to have this kind of back-and-forth way of thinking about it.[Keith post, 00:23 :42] In this quote we see a sampling of the type of understanding Keith wants students to develop.First, there is an explicit focus on factual knowledge, in that he wants students to be able to recall formulas when they need them.In addition, he expresses that activities that focus on problem solving support the development of physical intuition in a "back-and-forth" way.We coded this as a combination of procedural and conceptual knowledge.
Compared with Keith, Daniel spent less time describing the types of physics knowledge he expects students to use and learn in the classroom, but he expressed confidence in what he did say (verbal stress indicated): For instance, I'll set up a Newton's laws problem where you have forces and I'll say, "This is the way you always do it, you always set up the free-body diagram, then you always write down each component of the Newton's second law, and then once you've got that all written down, the next step is to try and combine it in ways…" [Daniel post, 00:16 :35] Here we see a clear focus on procedural knowledge, but in a more fixed, rote manner indicative of factual knowledge.Effectively, Daniel is emphasizing a process for students to internalize rather than adaptable strategies that students can reconstruct.
Sarah also emphasizes students' problem solving abilities, but discusses it in the context of connecting concepts learned in the tutorial setting to solving the types of traditional physics problems that appear on the weekly homework assignments: … just because [students] have to do that for the exams and things, or they just read a problem and they're like, "I know about these concepts, but I have no idea how to connect it to a problem, or what to look for in this problem that tells me what concepts I need."And some of that might be they need to get used to doing that kind of problem and it will come as they take more physics classes.I think that was difficult for some students.
[Sarah post, 00:26 :47] In this way, Sarah appears similar to Keith in valuing procedural and conceptual knowledge.Sarah also found ways to reconcile physics concepts with students experiences outside the classroom, such as in this example: Like when we did pressure this semester, one thing that was helpful was, for understanding that it only depends on depth, is so if you dive down to the bottom of a swimming pool, you know, it really hurts your ears, and it doesn't depend on what the sides of the pool looked like, or how wide the pool is, it's just how deep you go.So that was very helpful for people.[Sarah post, 00:37 :12] Here, using prior physical experience to understand a feature of pressure was coded as valuing sense making.
Figures 2 and 3 show the distribution of knowledge and espistemological stance codes for the Sarah, Daniel, and Keith.Procedural knowledge was the only knowledge subdimension mentioned by all three pilot TAs, mostly having to do with students' abilities to solve traditional computational physics problems.This sentiment was common among many of the interviewed TAs; they generally support the development of students' problem solving skills, although there are differences in opinion on what role the tutorial should play in that process.Keith expresses that there should be more traditional problem solving during the tutorial sections, going as far as to suggest that part of the recitation time be dedicated to showing examples of how to set up and solve mathematical physics problems.Sarah thinks the tutorials serve a role that is complementary to traditional problem solving.

C. Assessment
Assessment is a critical component of teaching practice.In this dimension, we examine how an instructor gathers information about a teaching situation, and to what end this information is to be used.For example, an instructor may assess students' understanding by reading what they have written in their tutorial book, by gauging their response to challenge questions, or by observing their manner of participation in a group.The assessment dimension also includes the ways in which TAs use questioning to understand classroom situations, and also encompasses broader perspectives on the role and nature of assessment in instruction.
Our coding subdimensions for the assessment dimension identify particular features of formative and summative assessment, including the intended purpose of assessment and the form it takes relative to instructional practice.In applying the coding rubric, it can be difficult to identify these forms of assessment based on observed practices alone.Instead, we identify features of assessment that can be more readily observed, and these can be mapped onto commonly identified components of formative and summative assessment.
The form subdimension can be somewhat constrained by the TAs'assigned role.For example, in the tutorial environments under consideration, TAs are discouraged from giving quizzes or discussing homework, so we are only able to observe a subset possible practices around assessment.For this reason, we will merge the form and purpose subdimensions of assessment for the subset of TAs presented here.
In this quote, Keith describes a particular way that he assesses student understanding: [The students] had a hard time understanding what the new words, their meaning in physics class meant, as in understanding that things like "velocity" and "acceleration" were very precise terms that meant something.And that they're supposed to be used in certain contexts.… they would get confused about speed and velocity because they wouldn't get in the habit of using physics terms.[Keith post, 00:07 :34] To Keith, students' use of informal language is an indicator of a lack of understanding.This is interesting framing on Keith's part because he is conducting assessment based on students' forms of participation in the social setting (situated).The language he uses, however, is indicative of a more summative perspective on assessment, as he does not describe how he might build upon students' everyday vocabulary to help them move toward using physics terminology.This is coded as evaluative assessment, rather than formative.
Sarah also describes how she uses questioning in particular ways: Or also, sometimes we came up with interesting questions to ask people who finished early, which was kind of fun.Or questions you can ask to try to deepen intuition of, or gauge their understanding of it without just asking the answer to the question in the tutorial.[Sarah post, 00:40:56] In Sarah's view, questioning can serve as a means to "gauge understanding" (evaluative) but also to "deepen intuition" (formative).Likewise, Sarah also described in her interview how part of her role is "asking questions to answer questions," which also reflects her belief in using questions to drive discussion in addition to evaluating student understanding.
A summary of assessment coding is shown in Fig. 4. Compared to other teaching components, the TAs appear less explicit in describing their beliefs about assessment in the classroom.In the absence of ongoing pedagogical training, these TAs appear less accustomed to terms (and concepts) such as "formative" or "summative" in describing their own assessment practices.In fact, these terms were not mentioned in interviews with any of the 13 TAs who were videotaped.We note that Sarah tends the most toward emphasizing more formative types of assessment, although both the amount and variation are not as great as in other dimensions.
By looking across these dimensions holistically, we can paint a broader picture and infer a more recognizable (if not stable) model of the TAs' teaching philosophies.We described previously how the relative focus of each TA on these dimensions provides insight as to their overall models of teaching.For instance, Keith's focus on content goals (using interview talk time and number of codes as an indicator) are reminiscent of the teaching model that Kuhs and Ball [45] describe as content focused with an emphasis on performance.In this model, student mastery of mathematical rules and procedures is emphasized, with the instructor serving primarily to demonstrate and model problem solving tools and methods.
While we do not argue here for a robust and consistent model of beliefs, we can use these dimensions to more carefully examine TA practices, thereby allowing us to consider TAs' described and enacted models of teaching side by side.In the next section, we examine TA practices through the lens of the TA-PIVOT framework.

D. Application to practices
To illustrate the application of the TA-PIVOT framework to TAs' instructional practices, we use it to analyze several classroom episodes.In practice it is useful to examine complete episodes of TA practice and then unpack the observed practices in that episode across each of the three dimensions.We first provide examples of how this is done, and then present the coding results for TA practices.
We begin by considering one particular topic area covered by each of the TAs during week 7 of Physics 1, after which we will summarize the video coding results across all observed episodes captured from videotape data during that week.The following vignettes are drawn from the "Work and changes in kinetic energy" tutorial [6].In this tutorial, students are presented with a scenario in which a force F 0 is applied to two blocks, C and D (m C < m D ), as they move between two lines on a frictionless table.In the question under consideration, students are asked to agree or disagree with a student who claims that block C will have a greater kinetic energy than block D at the time they each cross the finish line.On the next page, the tutorial prompts the students to check the consistency of their reasoning with the work-energy theorem.
We find this question particularly interesting to focus on because it provides a classic example of the UW tutorial "elicit-confront-resolve" model, in which students are presented with a question that is known to commonly elicit incorrect ideas; then prompted to consider an alternative explanation; and finally guided toward resolving the inconsistency [46].Very often these types of questions present a dilemma for TAs to "troubleshoot" student reasoning on the spot or allow students room to persist with incorrect ideas knowing they will be prompted to resolve it on their own.ideas to students.In once instance, a group interrupts her in the middle of an explanation to say, "We think we get it." With respect to the TAs' goals, there is a clear emphasis on factual, recall-based questioning among the TAs.In fact, Daniel relies exclusively on testing students' factual knowledge during week 7, whereas Sarah and Keith demonstrate an additional but relatively more limited focus on conceptual and procedural understanding.This is somewhat surprising due to the conceptual focus of the tutorials, and encourages us to think about the TAs' reliance on this lower-order cognitive domain.Additionally, Sarah's practices tend to emphasize students' ability to interpret and make sense of physics content, whereas Keith's practices focus on the correct, scientifically accepted answer.
We also observe clear differences among the TAs' preferred forms of assessment.Daniel relies exclusively on student responses to his checking-style questions, as well as their written answers, to gauge understanding.Sarah and Keith are more mixed in attending to students' forms of participation in the class, as well as direct questioning, to inform their practice.In the case of Daniel and Keith, this questioning tends to take a more closed initiate-response-evaluate form [47] that constrains student responses along a predetermined path.Although Sarah also engages in this style of questioning, we also observe her using questions to open the discussion further in a more formative manner.

A. Comparing beliefs and practices
In order to compare beliefs and practices coding side by side, in Fig. 6 we present coding results for all eight TA participants.Beliefs coding results are drawn from postsemester interviews, whereas practices codes are aggregated from all observed episodes.Here we use alignment with the tutorial approach as a rough measure of cognitive demand in ways that reflect both our view of the tutorials themselves, and also how they are incorporated in our course setting.We should also stress that we differentiate coordination of beliefs and practices from inflexibility.We view sophisticated teacher knowledge as inherently fluid and adaptive, and we therefore do not associate "sophistication" with any particular component of these dimensions.Rather, we expect more sophisticated instructional practices to be flexibly applied and informed by local contexts and student feedback.
Daniel and Keith's observed practices related to agency appear relatively well aligned with their stated beliefs, both of which emphasize teacher agency.However, they are inconsistent with the intended approach of the tutorial, which is to allow students the opportunity to externalize their own reasoning and resolve disagreements and inconsistencies amongst themselves.In comparison, Sarah is aligned with the tutorials in her stated beliefs but not as well in her practices.
Examples like Sarah's illustrate how challenging it can be to consider and apply pedagogical intentions "in the moment." Compared to the interview setting, all of the TAs had a greater tendency to emphasize Factual knowledge in the classroom.We also observe indications of consistency between beliefs and practices in TAs' epistemological stances.For instance, Daniel was coded as only answer making in the interview setting, and he had a relatively higher fraction of this code in practice; the same was true for Sarah along sense making.In this instance, Sarah is coordinated with the curriculum in both her beliefs and practices, which may indicate a productive foothold that could be used to help her align other aspects of her instructional practices with her beliefs.
Our earlier observation regarding the TAs' lack of focus on assessment in their beliefs appears to carry over to their practices as well, where we see that questioning is not a focal component of TA practices.When it is, it is primarily used to gauge student understanding rather than allow an opportunity to expand the discussion.This result is surprising to us, given the critical role of assessmentparticularly formative assessment-in instruction.This finding motivates us to reflect on current TA preparation methods and consider how discussions of the assessment accompanied by efforts to incorporate them into ongoing practice would be productive in increasing TA attention to beliefs and practices surrounding these constructs.
Furthermore, we find that TAs in these environments may exhibit surface-level support or buy in to transformed curricula while exhibiting beliefs and practices that appear contrary to the pedagogical intent of that curriculum.Daniel is one example of a self-described supporter of the UW tutorials, having used them as an undergraduate physics student in the same environment that he is now a TA.This observation is consistent with Dancy and Henderson's finding that physics faculty who self ascribe as users of a research-based curriculum may not necessarily implement them as designed [48], and when faculty modify researchbased curricula, it tends to be in a way that emphasizes more traditional forms of teaching [49].Opportunities to teach using transformed curricula afford a chance for graduate teachers to become deeply engaged with the aims of research-based instructional strategies that will serve them in future teaching; however, as Daniel's example demonstrates, this engagement does not happen on its own.We notice that the TAs' "centers of gravity" appear to be toward the left side of the chart, indicating a general tendency toward antialignment with the curricular approach.Looking across the dimensions, the TAs' practices appear to be "stratified" relative to one another.That is, the TAs' relative positions along each dimension appear to be coupled (for instance, compare Darren's points with Caleb's).Furthermore, the TAs appeared more mixed around the constructs of agency and knowledge than the other dimensions.Darren in particular demonstrated a consistently teacher-centered approach to his instruction, primarily characterized by his direct answers to student questions with relatively infrequent prompts for student collaboration or discussion.Although no TA was consistently student centered in his or her approach, Caleb and Joanna's demonstrated the widest range of agency practice codes.

B. Coordination between beliefs and practices
As illustrated in Fig. 6, we observed instances of TAs whose stated beliefs were aligned with curricular goals (but not aligned with their practices), and TAs whose stated beliefs aligned with their practices (but not the curricular goals).In order to better visualize the degree of coordination between TA beliefs and practices, we can translate the Fig. 6 data into a two-dimensional format, as shown in Fig. 7.In this figure we have plotted aggregate coded practices against post-semester coded beliefs.The coordinates of each marker correspond to the positions of the diamonds in Fig. 6.
In this representation, a TA's location on the chart indicates the degree of coordination between his or her beliefs and practices.Points along the diagonal indicate coordination between stated beliefs and enacted practices, with points further from the origin being more aligned with the curriculum.Points that are above the diagonal indicate that a TA's stated beliefs are more aligned with the curriculum than his or her practices, and vice versa.
A few trends are immediately apparent from Fig. 7. Overall, there is a general trend away from alignment with the curricular approach (i.e., toward teacher agency, answer making, and evaluative assessment).We note that Darren tended to have the largest differences between coded beliefs and practices.On the other hand, Terrence demonstrated a relatively small difference between coded beliefs and practices, although each of these tended to be antialigned with the tutorial approach.We may therefore wonder whether TAs such as Darren are aware of the difference between their described and enacted practices, and whether they would recognize a disconnect upon viewing excerpts of their own teaching.

C. Dynamics in beliefs
As many TAs are first-time teachers, we are interested in what impact these teaching experiences (particularly in transformed environments), may have on their stated beliefs about teaching.In Fig. 8 we present both preand postsemester interview coding results using the same two-dimensional representation to illustrate pre-post semester shifts in the TAs' coded beliefs.Only the five follow-up TAs are included in this figure, as presemester interviews were not conducted with the pilot TAs.In this representation, the x-y position indicates the pre-post shift (if any) in the TAs' coded beliefs.Points along the diagonal are indicative of no pre-post shift in beliefs, whereas points above or below the diagonal indicate shifts that are either aligned or antialigned with the curricular approach.Furthermore, the distance between the point and the diagonal indicates the relative size of the shift.
For the most part, although they expressed differing beliefs from one another in the interview setting, the TAs did not appreciably change their beliefs over the course of the semester.Terrence appeared to undergo large shifts along epistemological stance and assessment, but these may be exaggerated due to the small number of overall codes compared to other dimensions.The TA who appeared to undergo the smallest shifts was Darren, who was also notably consistent in using a specific metaphor ("translating information") to describe his teaching during interviews.
Although it is not apparent from the aggregate practices shown in Fig. 6, we did observe preliminary indications of differences in TAs' observed practices across different tutorials.However, we did not observe an overall progression toward more or less alignment with the tutorial over time.Rather, the individual tutorials tended to support and/or constrain TA practices on a week-by-week basis, which warrants further investigation.

VI. DISCUSSION AND CONCLUSIONS
We conclude by returning to the motivating questions presented in Sec.III: (1) Can we design an instrument that is able to distinguish stated beliefs and practices of physics TAs?
We have constructed a framework for comparing TA beliefs and practices along three primary dimensions, and demonstrated its utility in distinguishing among TAs within each of these dimensions.Figure 6 showed how the full set of 8 TAs compared in their post-semester coded beliefs along the TA-PIVOT dimensions.Furthermore, we were able to replicate our initial observation from pilot data that some TAs tended to be more aligned with certain aspects of the tutorial approach than others in their stated beliefs.
(2) Do we observe instances of coordination and/or discoordination between between beliefs and practices?
The expanded study uncovered additional instances of discoordination between TAs' beliefs and practices.Furthermore, the degree of coordination between TAs' beliefs and practices tended to vary across the TA-PIVOT dimensions.For instance, most of the TAs were coordinated in their beliefs and practices regarding knowledge goals, but not their epistemological stance.Some TAs, like Darren, exhibited a more consistent level of discoordination across dimensions.As we described previously, perceived barriers to implementation are one explanation for why instructors' observed practices may not coordinate with their stated beliefs.Talking to these instructors and presenting them with examples of their own practices affords a rich opportunity to approach the question of why they chose to engage in certain practices but not others.At the end of the post-semester interview during the expanded study, the five TAs commented on segments of video drawn from the video data inventory.In this case the TA video reflections provided additional information regarding the underpinning motives for these differences in approach.A number of preparation models include video consultations to scaffold practitioners' ability to reflect upon instructional practices, and we envision the TA-PIVOT framework serving to enhance these efforts (for instance, by allowing TAs to self-code).
(3) Can we observe shifts in TAs' beliefs over the course of a semester?
We were able to distinguish shifts in stated beliefs for the TAs as measured by the TA-PIVOT framework, although the size of the shifts were small compared to their overall span of coded statements.Hence it appears that TAs tended to persist in particular beliefs, despite teaching in a research-based classroom environment that is structured to promote student-centered instruction.In other words, it does not appear sufficient to simply place TAs in instructional environments that are aligned with student-centered pedagogy in order to impact their beliefs about teaching.
This result is consistent with prior research on the development of teacher beliefs, which suggests that preservice teachers' beliefs change over a time span of several years.Furthermore, four of the five TAs in the expanded study population entered with some form of teaching experience, some in environments very similar to the CU tutorial model.It is possible that these prior experiences served to engage these TAs in refining beliefs about teaching before their arrival at CU, which may have reduced the "novel" impact of teaching in a transformed environment.
Although TAs' practices did not appear to undergo consistent overall shifts over the course of the semester, in analyzing the data we observed indications of week-byweek differences, suggesting that certain TAs tended to utilize different strategies for different tutorials.We hypothesize that the structure and/or content of the tutorials may be a factor in these differences in approach, and the TA-PIVOT framework will be useful in future work examining the subtle parameters that influence TAs' choice of strategy.
In conclusion, we have developed a framework for the analysis of instructor beliefs and practices along three dimensions, which is grounded in both existing theory and emergent themes in the data drawn from interviews and observations of TAs.This TA-PIVOT framework serves to connect how TAs talk about their own teaching to the ways that they engage students in the classroom.In the process of constructing and applying the TA-PIVOT framework, we have observed (i) variation among TAs in their beliefs and practices, (ii) apparent instances of inconsistency between TAs' beliefs and practices, and (iii) no significant shifts in particular beliefs over time.These observations provide an opportunity to examine TAs' awareness and perception of how their classroom practices relate to their beliefs, and provide us as researchers the opportunity to investigate the nature and sources of tension between beliefs and practices.For instance, we would like to have TAs self-code their own practices as both a novel way to further test the framework and a targeted reflective opportunity for themselves.
We have designed the TA-PIVOT framework to be an adaptable tool that can be applied to a wide array of classroom settings.Now that we have established a baseline for utility of the framework in one research context, future work will allow us to further refine the coding schema and explore the framework's potential application to examining and improving professional development for TAs.

FIG. 2 .
FIG. 2. Coded instances of beliefs along the knowledge dimension.

1 [
FIG. 4. Coded instances of beliefs along the assessment dimension.

FIG. 6 .
FIG. 6. Side-by-side comparison of coding results for (a) beliefs and (b) practices for all 8 TAs.Percentages along the top indicate percentage of assigned codes for each TA.Percentages along bottom (in italics) correspond to the position of the diamond markers, indicating each TA's median code and overall degree of curricular alignment.Positive values indicate alignment with the tutorial approach, whereas negative indicates antialignment.

FIG. 7 .
FIG. 7. Two-dimensional representation of coordination between TA beliefs and practices.For each coding dimension, the marker position indicates the degree of alignment of post-semester beliefs (horizontal) and aggregate practices (vertical).Pilot study TAs have been included for comparison.In this representation, points on the diagonal represent well-coordinated beliefs and practices.(See guide at right.)

FIG. 8 .
FIG. 8. Two-dimensional representation of pre-post semester shifts in TA beliefs.For each coding dimension, the marker position indicates the degree of alignment between the curriculum and pre-semester beliefs (horizontal) or post-semester beliefs (vertical).In this representation, off-diagonal points represent shifts.(See guide at right.)The three pilot TAs are not included, as pre-semester interviews were not conducted with them.

TABLE I .
Study participants, along with their prior teaching experience and incoming familiarity with tutorials.(All names are pseudonyms.)Starred participants were part of the initial pilot study that included limited classroom observation and only postsemester interviews.

TABLE II .
The TA-PIVOT framework, with broad dimensions in bold in the far left column.A more detailed description with the coding rubric is provided in Appendix B. would prefer to see that they got to the wrong track and then get them to the right spot, and then [have] me tell them the way I would think about it, correctly, rather than hear them like, talk about and go around in circles about stuff and try to follow where they were going.I would prefer to just get to the point where the confusion set in and tell them how I did it, and then hear them go, "Ohhh… okay."[Danielpost, 00:32:16] I

TABLE IV
[I] try to ask those sort of questions that like, that can be used more generally, like reasoning like that… you can use in all free-body diagrams and like, a lot of the problems in mechanics in general."Confirming result "I tried to just get through it fast.Instead of having to stop and explain too much stuff to the LAs."This is a better way of thinking of it, you shouldn't necessarily just memorize equations.But it's a lot more helpful if you try to figure out where the formulas come from."Sometimes it was pretty quick just because they pretty much got it and then I just wanted them to continue so they could discuss it among themselves… because it seemed like it was going well, and it wasn't really going to help anything if I just asked them to repeat it more.So then I would just move on."