Student Epistemological Framing on Paper-Based Assessments

Assessments are usually thought of as ways for instructors to get information from students. In this work, we flip this perspective and explore how assessments communicate information to students. Specifically, we consider how assessments may provide information about what faculty and/or researchers think it means to know and do physics, i.e. their epistemologies. Using data from students completing assessment questions during one-on-one think aloud interviews, we explore how assessment features impact (or did not impact) student engagement with the assessment problems. We analyze video recordings and transcripts to infer the epistemological framings and resources students use while completing introductory-level physics problems. Students' framings tended to be fairly stable, but when shifts did occur - they were triggered by a shift in resource, which can be activated by assessment feature. This work extends existing work on epistemological framing into the realm of assessment and allows us to consider the effects of assessments on our students' understanding of physics teaching and learning.


I. INTRODUCTION
As physics educators at undergraduate institutions, we are all aware of the importance of assessment in our classes. Individual assessments help us -and our students -understand whether they have learned the content and skills we have painstakingly taught them. Additionally, we assume that how a student performs on assessments throughout the semester helps us evaluate and track student progress. Students in introductory physics frequently take multiple, high-stakes assessments (or exams) each semester along with weekly homework assignments and lab reports. Assessment occurs often and in a variety of ways.
Consider a common assessment item on introductory physics exams -a numerical problem that requires calculation to solve. Imagine a question such as Diego is standing on a scale in an elevator and the elevator starts to accelerate upwards at 3m/s 2 . If Diego weighs 71 kg on his scale at home, how much will the scale read while the elevator is accelerating?
We suspect this type of question is familiar to many of our readers. We suspect the following solution is also familiar. F = ma F gravity = ma elevator mg = ma elevator (71)(9.8) = m(3) 231kg Units match! As instructors, we might find ourselves horrified by this student's solution. It is definitely wrong! They may have matched units but they entirely forgot the normal force. Many of us can relate to seeing such solutions on our exams and being confused that the students did not learn what we taught them.
In these moments, our default is often to become frustrated -either with them for not learning enough content to solve the problem, or with ourselves for not teaching it well enough. That is, we attribute their lack of performance either to them or to our teaching. In this paper, we suggest that there may be a mechanism other than poor content knowledge that accounts for what looks like "poor" performance.
Specifically, if we look closer, we can understand what the student is doing here as a type of pattern matching, or numerical plug 'n' chug using formulas she is familiar with [1]). She is solving the problem using the symbols and mathematical formalisms we use in class and teach students to use. She has found the force of gravity and appropriately substituted in the acceleration of the ele-vator. In that way, she is solving the problem in a way that looks like other problems she has done.
Viewing student performance through this lens assumes that students are working to make sense of the assessment in the terms they think we want. The logic is something like: "I have seen my teacher use this formula in problems like this, so I will do the same thing here." This perspective on student engagement in assessment removes the assumption that students are being foolish when they do things that appear outlandish to experts. Instead, it assumes they are trying to do what we have asked them -even taught them -to do.
In this paper, we explore this perspective in order to move away from our traditionally deficit-oriented model of assessment. Specifically, using the theoretical machinery of epistemological framing and resources, we seek to understand student performance as a reasonable -rather than misguided -interpretation of what assessment writers have asked them to do. We draw on data from thinkaloud interviews with undergraduate students in introductory physics to (a) explore how students frame assessment, (b) unpack the knowledge resources that underlie those frames, and (c) document stabilities and dynamics in resources and frames within the context of assessment items. Specifically, we describe how assessments may send students messages about what types of knowledge and knowledge production activities are appropriate to use in the assessment context. We discuss implications for these findings for educational research and assessment practice.

A. Epistemology
Within psychology and education, researchers have long been interested in how people understand the nature of knowledge and learning [2]. These understandings -referred to as a persons epistemology -involve their knowledge and beliefs about the nature of knowledge itself and how knowledge is built and evaluated. Since the 1950s, scholars have proposed a variety of dimensions of epistemology. For example, epistemological knowledge includes knowledge about the goal or aims of knowledge construction [3], how knowledge is structured [4], how knowledge is justified [5], and the appropriate activities to use in constructing knowledge [6]. Scholars have explored both personal epistemologies -how people view their own knowledge and learning -and scientific epistemologies -how professional scientists construct knowledge and learning [7].
Much of the work understanding epistemology has taken place in educational settings (See Hofer & Pintrich, 1997 for a review). The focus on educational settings arises because of the particular importance of epistemology for learning. Specifically, there is the assumption that "epistemological premises are a part of and an influ-ence on the cognitive processes of thinking and learning [8]. In his cognitive model of learning physics specifically, Redish (2004) calls epistemology a control structure in that it "interact[s] strongly with (and often controls) the [knowledge] resources students have for creating knowledge (p. 30) [9]. That is, a persons epistemology can impact the way they engage in learning by dictating the kinds of knowledge they use and the ways they use it.
Of particular importance for us in this work is the finding that what has often been labeled learning "difficulties [10] [11] [12] may instead "stem in part from [...] epistemology" [13]. For example, Lising and Elby present the case of a student Jan, who, despite possessing all the knowledge and skills needed to make sense of a physics tutorial, does not do so because her epistemology -her understanding of what knowledge and knowledge building she is supposed to do in physics class -"gets in the way" (p. 381). Epistemology mediates content knowledge in introductory physics.
Despite substantial interest in epistemology in physics learning since Hammer's initial introduction of it [14], we do not yet know of any who have directly examined epistemology in the context of assessment in undergraduate physics. The work has been constrained mostly to classroom or classroom-like contexts. In our work, we explore the hypothesis that the same mechanism at work in the case of Jan -the same "epistemological effect" [13])might be at play in assessments. Specifically, we explore what epistemologies are active when students use their content knowledge to complete assessments.

B. Assessment Design
What types of assessments do our undergraduate students typically engage in? Assessment is a central pillar of our current education system and is often divided into two types: formative and summative. Here, we will use the definition of formative assessments as "intended to provide feedback to the system to inform next steps for learning" and summative assessments as measures "of individual achievement" [15]. Both of these definitions focus on assessments as ways to get information about what students know. In this paper, we focus on summative assessments. Beyond these two definitions, though, current theories of summative assessment design focus explicitly on how we get information about students' current understanding.
The Assessment Triangle, described in Knowing What Students Know, has been central to assessment development for nearly twenty years. It focuses on three interconnected models for developing assessments: cognition, interpretation, and observation [15]. Taken together, these three models (aka, the Assessment Triangle) are designed to treat summative assessment as an evidentiary argument, focused on designing tasks that allow the instructor/researcher to gather evidence in order to make claims about student knowledge.  [20] and the BEAR Assessment System (BAS) [21] also focus on obtaining evidence to support claims of student knowledge. Both of these approaches have been highlighted as promising ways to assess the Next Generation Science Standards [22,23].
The Standards for Educational and Psychological Testing states, "Test development is the process of producing a measure of some aspect of an individual's knowledge..." [24]. In physics in particular, Adams and Wieman have argued that the development of concept inventories in PER typically follows the steps outlined by this document [25].
In each of these approaches to designing assessments, the focus is on obtaining information about student knowledge and little to no attention is paid to the messages that these assessments send to students. Assessments developed in these ways are assumed to be measurement instruments to get data about the knowledge of students in the same way a thermometer is a measurement instrument to get data about the temperature of a water bath; that is, that the measurement does not affect the system (or affects it minimally). Given this state of assessment design theories, we suggest that instructors and other individuals designing tests for physics courses probably do not think about the messages being sent to students either.
Within the assessment culture of undergraduate physics, most of the assessments are focused exclusively on evaluating student content knowledge. Further, many do so in a way that tacitly neglects students' understanding of either physics generally or the assessment in particular. There are a few assessments that attempt to direcly measure student epistemologies. For example, the Redish, Saul, and Steinberg declare, "[W]e describe the Maryland Physics Expectations survey; a 34-item Likertscale agreedisagree survey that probes student attitudes, beliefs, and assumptions about physics" [36] and Adams et al start their abstract, "The Colorado Learning Attitudes about Science Survey (CLASS) is a new instrument designed to measure student beliefs about physics and about learning physics [37]." However, we are not aware of any assessments that are designed to study or explicitly understand the connection between student epistemology and assessment of content knowledge.

III. THEORETICAL FRAMEWORK
As described above, researchers interested in epistemology have not yet examined the ways in which student understandings of knowledge and learning are present in their engagement in assessments. Similarly, researchers interested in assessment have overlooked the ways in which assessments are understood from the perspective of knowledge and learning. Here, we turn our analytic attention to bridging the gap between those two literatures. In what follows, we describe our theoretical stance on epistemology and its implications for our research on assessments.

A. Epistemological Framing
Understanding the ways in which epistemology interacts with student engagement in assessment requires a precise conceptualization of epistemology. When it was first conceptualized, epistemology was understood as a set of categories that students adopt and then progress through over the course of their lifetime. These categories applied to all aspects of their learning at any given age. For example, Perry described students as either "Dualist" or "Relativistic" in all contexts of their lives; they are not "Dualists" in math and "Relativists" in science [2]. However, research shows significant "flexibility and variability in student reasoning" in K-16 science and physics classrooms [38]. As such, rather than a stable model of epistemology, we adopt a model of epistemology rooted in the sociolinguistic and anthropological construct of framing. [39] [40] Framing is a person's sense of "What is it that is going on here?" In recent work in physics education, scholars describe epistemological framing [41] as a students' answer to "How should I approach knowledge?" [38] Epistemological framing, then, is the tacit stance students take toward learning-based activities [42].
A key feature of framing in general -and epistemological framing in particular -is that it is contextual and dynamic rather than stable across time and place [43]. For example, when a student enters science class learning electrostatics, they likely think very differently about knowledge than when they are in a discussion with their friends about what pizza to order. Even more, they also likely think differently about learning during a portion of science class that is a lecture versus small group discussion. [44] Even within small group discussion, students' sense of what knowledge and knowledge building activities should be used can shift dramatically [41]. This contextuality means that framing is highly dynamic. Be-cause people shift their understandings of knowledge in different contexts, framing must also change over the time scale of contextual change (hours, minutes, and seconds) rather than over the scale of a lifetime.
Existing research on framing in undergraduate physics education suggests that students adopt a variety of frames when engaged in learning physics. For example, several scholars have explored the sensemaking frame in which students reason about physical phenomena by constructing an explanation and then filling in a gap in that explanation [38,42,45].
Bing and Redish [46] identify four common epistemological framings students adopt during physics problem solving: Calculation, physical mapping, invoking authority, and math consistency. In their work exploring framing in quantum mechanics, Modir, Thompson, and Sayre (2017) describe a set of frames that differ along two dimensions -whether students draw on mathematics or physics, and whether they are engaged in algorithmic or conceptual thinking [47]. Each of these framings involves different -though not necessarily better or worse -understandings of knowledge and knowledge construction in physics class.
In addition to identifying framings that are prevalent in physics, scholars have also focused on whether and how students move between framings. Hammer and his colleagues describe transitions between mathematical manipulation and intuitive sensemaking that are both short-lived and lasting [38]. Bing and Redish identified their four frames by explicitly looking for and unpacking shifts [46] and have suggested that the frequency and fluency at moving between framings is part of becoming and expert [48]. This work highlights the dynamic nature of epistemological framing.
The theory of epistemological framing suggests the need to refine our question of interest even further. Specifically, this framework suggests that students may not adopt a single epistemology during assessment. Instead, they may transition between multiple framings. As such, we now ask: How do student epistemological framings influence their engagement in assessment tasks?

B. Epistemological Resources
As we have described, existing scholarship within physics education highlights the dynamics of framing. To examine and document those dynamics, much of the literature has focused on identifying observable shifts in behaviors. Scherr and Hammer (2009) pioneered focus on behaviors with their careful analysis of students' verbal, non-verbal, and para-verbal behaviors in small group interactions [41]. They describe how "different behavioral clusters are evidence ofand in dynamic interaction withstudent epistemologies" (p. 148). This finding led to an explosion of work in PER that identified behavioral clusters and their associated framings.
For example, consider more closely the work of Modir, Thompson, and Sayre who identified four epistemological framings in upper level student problem solving [47].
To identify framings, they describe how they "reflected on... episodes [of student activity], seeking to answer whats going on? for each of them... we sought to capture changes in students discussion or behavior that might indicate a shift in the students problem solving processes" (p. 020108). Here, they focus on the behaviors associated with their framing. This focus on behavior makes sense and has given the field substantial traction in identifying moments when epistemological framing shapes student engagement and learning. However, a feature of the theory of epistemological framing that is commonly left out of the literature in physics education is that framings are local collections of epistemological resources [38] [49]. Specifically,our model of epistemology -grounded in the work of Hammer and Elby [6] -assumes that rather than being unitary entities that stably exist as a coherent unit, framings are more like networks of many smaller elements that are all activated together in context [38].
The small elements that make up epistemological frames -or what Rosenberg, Hammer, and Phelancall "multiple local coherences" -are epistemological resources [49]. These resources are finer-grained elements of knowledge are assembled anew in each context to create a person's in-the-moment epistemology. This finergrained structure affords contextuality in a person's epistemology.
A number of potential epistemological resources have been proposed that govern student behavior in knowledge-production contexts. Some deal with the nature of knowledge itself: the form of the knowledge product [50] or the source of the knowledge [6]. Others deal with how knowledge is produced/constructed: knowledge production activities [44] or the goals of those activities [3].
In our work, we hypothesize that understanding the specific resources that make up the framings will give us more insight into how and why students engage in learning in particular ways than merely identifying their epistemological framings as a whole. This assumption follows the work of Rosenberg, Hammer, and Phelan (2006) and Russ and Luna (2013) who each go an analytic level below framings to the level of resources [44,49]. Specifically, We apply this assumption to the context of assessments and ask: How are the epistemological resources that make up student epistemological framings evident in their (potentially dynamic) engagement in assessment tasks?

IV. RESEARCH QUESTIONS
Our everyday experience as physics instructors leads us to question why students engage with assessments in the way they do. Further, our underlying asset-oriented perspective toward student learning leads us to reject the existing deficit-based explanations in the literature [51].
As such, we began this research with the question: Why might students reasonably engage with assessments in the way that they do?
As a result of our theoretical framework, we refine and extend our question.
1. What epistemological resources make up the framings students adopt during assessments?
2. How do these epistemological resources influence their dynamic engagement in assessment tasks?
Specifically, this works seeks to understand whether existing analyses and assumptions of how students dynamically bring different forms of knowledge into physics learning can be extended into the realm of assessment.

A. Participants
In this article, we analyze data collected as part of a different study on how to assess scientific practices in introductory physics courses [52]. The participants in this study were predominantly engineering majors enrolled in the first or second semester of a calculus-based, introductory level physics course. The interviews were conducted near the end of the semester. Students volunteered to take part in the study and were compensated for their time (equivalent to $20). No preference was given to their performance in the course. Pseudonyms are used throughout the paper. The study collected data from two groups of ten students, with each group of ten receiving a separate exam. We analyzed data from all twenty students.

B. Data Collection
The data was collected as individual think-aloud interviews designed to replicate the context of a summative assessment. To do so, interviewers printed each list of questions as a formatted test, had subjects write their names at the top, answering only clarifying questions, and telling students to complete the questions as a test with no permission needed to move on to another question. Each interview took each student 45 to 90 minutes to complete. The students had unlimited time to complete the questions.
The questions on the tests were developed using the 3D-LAP to elicit evidence of students abilities to engage in the scientific practices [53]. One of the exams focused on the scientific practice of Using Mathematics and the other focused on Developing and Using Models. Students solved physics problems on a variety of Physics 1 subjects. The exam included both selected and constructed response items. Only the constructed response items were used in this analysis

C. Data Selection
From the recorded interviews, there were two sets of 10 students. One student from each group of 10 did not have usable audio, resulting in 9 recorded interviews for each problem set. The first assessment given to the first group of 9 students featured 3 constructed response questions. The second assessment given to the second group of 9 students featured 5 constructed response questions. This yields a total of 72 instances of students answering a constructed response question. Of these 72 responses, 5 were not used. Reasons for not using a response included the student not attempting that specific problem or stating that they have seen the problem before and know the answer (and, thus, do not engage in the problem solving process. Below is an excerpt from Mark.
Mark: Alright, so we had this exact problem on our test last year, after doing it.
Because we were specifically interested in how students engaged with the features of this specific assessment question, we decided not to use his response or similar responses where students do not engage in problem solving.
The examples we use throughout the paper come from specific assessment questions we will refer to as the Ferris Wheel problem and the Car problem, shown in Figures 3 and VI C, respectively. We also transcribed the audio from what we will refer to as the Gravitron problem, which we chose as an example of students ardently staying in their problem solving strategy. The full Gravitron question is shown in Figure 1. Although we largely reference those problems in the cases we present, all constructed response questions were analyzed via the process described in section D.

D. Data Analysis
Of the remaining 67 instances, all were coded for epistemological frames. A sample of 20 instances were transcribed and coded for epistemological resources. After coding each full sentence expressed by the subject, we then compared when frame shifts occurred to when shifts in individual resource shifts occurred. In this section, we describe how we coded the data and offer an example of coding one instance.

Coding Epistemological Frames
For this particular project, we opted to use epistemological frames that had already been identified in the literature. In particular, We wanted to use frames that could be identified by looking at behaviors, a process originally described by Scherr and Hammer [41]. We chose the frames described in Chari et al. [54]. We selected this set of frames and behaviors primarily because they were already identified and because they mapped well onto our data. Chari et al. used their frames to analyze group problem solving interviews, but the language used to define the frames does not limit them to group or individual work. Below are the characteristics and behaviors associated with each fram as described in Chart et al. (2017) [54].
Conceptual Physics: Students and instructors are in this frame when they discuss physics scenarios and phenomenon, about properties of physics quantities related to the task at hand. They may also exploit the symmetry of a physical system by investigating related concepts.
Algorithmic Physics: In the Algorithmic Physics frame, students recall physics equations or apply physics knowledge to re-arrange known equations using math. Students may also derive expressions for specific cases from a general physics equation or validate an expression via dimensional analysis.
Algorithmic Math: This frame refers to performing mathematical computation by following wellestablished protocols without questioning the validity of those protocols e.g. solving an equation or computing an integral.
Conceptual Math: Students are in this frame when they exploit properties of mathematical constructs to quickly obtain a result without diving into algorithmic manipulation e.g. noticing that all the odd terms in a sun are equal to zero.
Using these frames and definitions, we analyzed the recorded video interviews using a top-down coding scheme to identify frames and noted each time a student switched from one set of behaviors to another and identified this as a frame shift.

Coding Epistemological Resources
Next, we segmented transcripts by complete sentences and identified epistemological resources through bottom up analysis of students' behaviors and statements as defined in Table I.
First,as we were specifically interested in what causes a frame shift, we transcribed the three examples we found of a clear frame shift. All responses from two questions where almost all students solved the problem in the same frame were coded for resources to investigate what prompted students to chose this frame.
We then coded all of transcribed attempts for epistemological resources. Each full sentence of the transcribed interview was sectioned and we inferred epistemic resources according to definitions given in Table I. We grouped by each full sentence because that was as large as we could go while still determining a resource shift and as small as we could go while still having the context to determine a resource.
In order to code the resources, We used categories defined in Hammer and Elby's 2002 paper On the form of a personal epistemology [6]. We specifically focused on Nature of Knowledge, Source of Knowledge, Epistemic Activity, and Epistemic Source because we found clear indicators of these in the words and actions of the students.
The individual resources we used also came from Hammer and Elby (top-down coding). In addition to repurposing codes from the literature on epistemology, we also generated codes from the data in an emergent fashion (bottom-up coding) [55]. These codes are Equation Sheet as a Knowledge Source; and both Number and Reasoning as Epistemic Forms. The working definitions and an example from the data for each resource are listed in Table  I. Only resources that appeared in the data are listed.
We did not limit Epistemic Form to the answer type requested by the assessment question. Instead, we decided there could be different epistemic forms possible for one question and for different portions of the problem-solving process. For example, suppose a multiple choice question asks students to find an acceleration, given a mass and net force, and lists four different values for acceleration. Students could either plug values into Newtons Second Law and choose the acceleration closest to their answer, or they could reason through the options listed and pick the most probable value. We believe these two students, although their ultimate answers were a letter, had a difference in Epistemic Form because they were working towards different ending conditions, a number or reasoning, that they later translated into a letter.

Inter-rater Reliability
The first author coded all 67 transcripts for their frames and frame shifts. The second author again coded a subset (8) of the transcripts. The two coders agreed on all codes (100%) perfect agreement.
Additionally, the first author coded each full sentence of each the 67 transcripts for each of the four dimensions of epistemological resources. To check the reliability of this coding, the second author coded a 10% subset of the transcripts (7). We then calculated percent agreement across the doubly-coded transcripts for each dimension. The percent agreement for each epistemological dimension (aka coding category) was greater than or equal to 80%. Specifically, agreement was 81%, 84%, 81%, and 98% for Knowledge Type, Knowledge Source, Epistemic Activity, and Epistemic Form, respectively. This is considered acceptable agreement for qualitative coding in research [56].

Resource Type
Resource Definitions Example from Data Knowledge Type If you asked the student how they know this, they will say...
Okay so centripetal force is going to be equal to rw squared  Alright, so this force has to keep them from falling down

Epistemic Form
The student believes the answer to their question will be indicated by...
Okay, we need to find Alpha Reasoning ...an application of conceptual knowledge.
It doesn't tell you the mass. So, I guess because this is a conceptual question.

Example Analysis
Here we include an example of how this analysis is done in practice. Figure 1, or "The Gravitron Problem" features a problem that gives students the radius, coefficient of friction, and angular velocity of a Gravitron ride and asks them to determine if the ride is safe for riders within a range of masses. Spoiler alert: The safety of the rider is not dependent on their mass. Below is a transcript of Erica beginning to solve the problem: Erica: Okay, so Im going to get the rotational ones (picks up equation sheet, starts copying equations onto paper). Um, so you just want to find like your minimum 25 kilograms and your maximum 160 kilograms and then any answer you get between that is that going to, um, stay on the wall. (Looks at problem) Um, 3 radians a second, so Im just going to write that up here (writing on paper) radians per second (typing into calculator) Let me get that number 60 seconds for one minute and its 2 pi radians for one revolution (types into calculator) or something like that.
First, we used behaviors to determine the Epistemic Frame each student was working in. Erica starts the problem by rearranging known physics equations that she can use to input given values. She believes the output of her function will tell her whether some people will slide off the Gravitron. For these reasons, we determined that Erica began the problem in an Algorithmic Physics frame.
Second, we grouped coherent thoughts and coded each thought using the four categories of Epistemic Resources defined in Table I. Erica's first full sentence is "Okay, so I'm going to get the rotational ones." We coded her Type of Knowledge as Propagated because the knowledge of the physics equations is communicated directly to Erica through the equation sheet. We coded her source of knowledge as her equation sheet. We coded her Epistemic Activity as Accumulating because Erica is gathering knowledge from an external source. Erica's epistemic source is ambiguous here because she could be intending to use the equation to consider how the different quantities given affect each other and reason her way to an answer. We used additional context from later in the problem to determine that Erica intended to use the equation to input her given values and output a number that would indicate which riders would be safe, so we coded her Epistemic Form as Number.
We repeated this process across all 20 solution attempts that we transcribed.

A. Different Frames are Made up of Different Sets of Resources
Existing literature examining student epistemologies describes how frames are made up of resources (cite Resources, Framing and Transfer). Specifically, frames are "local coherences" of resources (Cite Rock Cycle); groups of several different resources tend to co-occur and change together. For example, Russ and Luna (2013) identified distinct sets of resources one teacher drew on when she engaged in behaviors from different framings. In one frame she engaged in particular epistemological activities with particular epistemological goals and in another frame she used different activities and goals.
Our independent coding of frames by behaviors and resources by utterance allows us to look for local coherences of resources within and across frames identified by other researchers (cite Ellie paper here). That is, we can engage in an analysis similar to that done by Russ and Luna (2013) and Rosenberg, Hammer, and Phelan (2006). Although this result is not theoretically "new" (by definition frames are made up of resources), we begin with it here for two reasons. First, other researchers in PER have yet to unpack frames in terms of their underlying resources. So this result demonstrates the feasibility of that work. And second, knowing the resources underlying the frames for assessment is essential for arguments we will make later in the results section.
We begin by looking at the framings themselves. Of the 67 responses used, three students' behaviors indicated they started in one frame and transitioned to another. These students are featured as case studies in Sections C and D. Two students began in an Algorithmic Physics frame and transitioned to a Conceptual Physics frame, and one student started a problem in a Conceptual Physics frame and transitioned to an Algorithmic Physics frame. This yields a total of 70 frames observed. Table II shows the resources we identified in each of the four frames. Looking across the table We see empirical evidence for our As expected from our theoretical perspective. Specifically, within each frame, we identified a subset of resources that are used. Across frames, different combinations of resources are used. Particular frames are associated with particular groups of resources different from other frames. The introductory students in this study almost always began problems in an Algorithmic Physics frame. Looking across the 67 total initial frame instances, 54 students started in the Algorithmic Physics frame, 11 students started in the Conceptual Physics frame, one student started in an Algorithmic Math frame, and one student started in a Conceptual Math frame, as shown in Table  II. For some questions, beginning and staying in the Algorithmic frame is an effective way to solve the problem. An example of this would be the Car problem that asked students to verify odometer readings at several times for a car. Students were given the cars acceleration and initial velocity. See Figure  Lisa finishes by indicating on her paper that the feature is working correctly. It was determined that Lisa was working in the Algorithmic Physics frame because Lisa first accumulates information given in the question and equation sheet, then manipulates the physics equation to isolate the desired variable, then inputs her given values into her new equation and obtains a number that she interprets as an answer.
In this case, the Algorithmic Physics frame serves Lisa well, and she has no need to transition. Other questions are more easily solved in a conceptual frame. Ideally, students could switch fluidly when they get stuck on a problem in their current frame, as experts do [48]. However, our data shows a fairly stable initial Algorithmic Physics frame, meaning that students seldom switch out of their initial algorithmic physics frame after beginning the problem in this frame. We found the Algorithmic Physics frame implements a specific set of resources.
The question that showcases this best from the problem set is the Gravitron problem, where students were given the radius, coefficient of friction, and angular velocity of a Gravitron ride. They were asked to determine if the ride was safe for riders of a range of masses. See Figure 1 for the full problem. Notice students were specifically asked to show their reasoning using diagrams, equations, and words. The problem may have been solved easily in a Conceptual Physics frame by noticing that the force of friction and the force of gravity should be equal so as not to let the rider slide down, which results in mass cancelling out. The resulting equation shows that gravity will overcome the force of friction and all riders will fall down. Here is a quote from Amanda as she works to complete the Gravitron problem: Amanda: So, right now Im thinking about what to do for the Torque, to try to find or use some of the kinematic equations. Im trying to remember how to do that. Find the alpha, but So I know there will be a friction force (Draws or writes on paper) which is just going to be mg mu, and then and that will keep you put at the wall minimum. So then, (Writes on paper) you can find torques for the minimum and the maximum mass. (Types into calculator, writes on paper) Okay, so now we have two torques. Hmm. Im not sure where to go from there.
Amanda begins the problem by attempting to rearrange given rotational kinematics equations such that she could plug her given values into an equation and get an answer. For those reasons, we determined she was in the Algorithmic Physics frame.
Of the nine students who attempted the Gravitron problem, we observed all students beginning the prob-lem in the Algorithmic Physics frame as indicated by their listed behaviors.
Across problems, our data shows that students often started in and did not shift from the Algorithmic Physics frame. Analysis of students' statements in the Gravitron problem shows that students did not attempt another frame even when making little or no progress in their current frame. We provide possible reasons for this in the Discussion.

C. Kicking a Resource Can Shift a Frame
While students seldom shifted frames, we did observe three instances, all caused by a change in a single epistemic resource. This was the result of two main influences, interviewer intervention and assessment feature.
Our first example of this comes from a student named Luke working on the Car problem ( Figure VI C). Below is the transcript of Luke beginning the problem: Luke: Okay, so the positions are these make sense. So, as its moving, its accelerating at 2.2 meters per second every second. Its accelerating. So 5.2 Since its not accelerating during this time. Well, I guess I proved here that the difference in between is one second, so I guess that is the correct position. I feel like there is a simpler way of doing this that Im overlooking. Um (re-reads the problem silently) This segment spans two minutes. Luke begins the problem by thinking conceptually about acceleration and the quantities he was given. Notice how he does not start any mathematical calculation or manipulation, but rather expects to obtain an answer by reasoning through the problem. The interviewer, noticing Luke was on the wrong path, intervenes: Interviewer: When you find the positions, you can compare. The question is giving position. It is asking for validation.
Luke: Oh, I think I assumed the wrong thing when I did it this way. Because I assumed that Well, I guess So I assumed that yeah. So, assuming that how fast its going, starting at that velocity, I found that it would take one second to get from this point to this point, which verifies that this is correct. So then going to this point, I guess I would have to do this same thing again. Thats just a lot of math.
This segment spanned one minute. After the intervention by the administrator, Luke realizes that he can solve the problem by substituting his givens into a known physics equation, which is quite the relief. Initially, Luke attempts to solve the problem by contemplating the properties of the quantity acceleration he was given in the question. After the intervention, Luke finishes the question by substituting the given quantities into a physics equation on his equation sheet. We interpret this, first, to be a shift from a Conceptual Physics frame to an Algorithmic Physics frame after the intervention by the interviewer.
A closer look at the resources Luke uses reveals a shift in Epistemic Form at that moment. Before the intervention, Luke believes he can reason his way to an answer to the Yes or No question of whether the odometer is functioning properly on the car. We interpret his Epistemic Form at that moment to be Reasoning. During the intervention, Luke realizes that he can easily calculate a number that will reveal the answer to the Yes/No question. We argue that the intervention at the level of Epistemic Form is what caused Luke to shift from a Conceptual to an Algorithmic Physics frame, and thus a new set of resources.

D. Assessment Features Can Shift a Resource
Outside of intervention, instructors may be able to influence their students' resource use, and thus their framing, through features of an assessment question. To show this, we have examples of students Lisa and Jack completing a problem where they were asked to determine where a rider feels heaviest and lightest on a Ferris Wheel. See Figure 3 for full problem. Students could solve the question in a Conceptual Physics frame quickly by thinking about where Normal Force points at different location on the Ferris Wheel. Notice that the question does not give the students any quantities, and the variables given in terms of letters are constant. As you will read, this prevents students from solving in the pervasive Algorithmic Physics frame. A total of nine students attempted this problem. Seven attempted the problem in what we determined to be a Conceptual Physics frame. Although we cannot conclude what made the students treat the Ferris Wheel problem differently, some admitted to having seen the problem before, so it is possible they already knew it could be solved conceptually. The remaining two students, Jack and Lisa, treated the problem very differently. First, let us look at how Lisa solves the Ferris Wheel question. Below is the transcript of Lisa starting the problem: Lisa: Um (picks up equation sheet) Wheres centripetal force? (Writes down centripetal force equation) Hm (looks at equation sheet) not given any numbers. Its just weird to me because it seems like nothing is changing. I mean, this is just going to be D over 2. The mass of the person isnt changing. Angular velocity is not changing. equations from her equation sheet and plug in known quantities that she can translate into a position on the Ferris Wheel as her final answer. For these reasons, we determined that Lisa starts in the Algorithmic Physics frame. She becomes frustrated when she realizes that this method will not yield an answer for her. After pausing, she changes her approach: Lisa: All Im thinking about is that your potential energy is going to be the highest when youre at the top, lowest at the bottom, so I think those are going to be the points at which youre going to be feeling heaviest, and those are the points I dont know why Im thinking this but I just keep thinking of a clock and a pendulum swinging or anyone being on a swing and your the points where you feel like youre accelerating the fastest are the ones where youre crossing this vertical axis. But I couldnt tell you why.
After realizing that she cannot answer the problem by manipulating given physics equations, Lisa changes her approach by contemplating her physical experience of being on a swing and applying her knowledge of potential energy. For these reasons, we determined that Lisa switches to the Conceptual Physics frame. Initially, Lisa believes that she can solve mathematically for an expression which she can translate into a position on a Ferris Wheel as her final answer, which we interpret as having a Number as Lisa's epistemic form. Not providing numbers or varying quantities is a feature of the assessment that blocks Lisa's initial Epistemic form, thus forcing her to try something new to finish the question. Next, Lisa believes she can determine the answer by reasoning through it, so we interpret her epistemic form to be Reasoning. We believe this is a clear case of an assessment feature shifting an epistemic resource, thus shifting Lisa's epistemic frame. We saw a very similar pattern in how Jack completed the problem as well. Below is the transcript of Jack starting the problem: Because Jack attempted to solve the problem by plugging values into known physics equations, we conclude that he was initially in an Algorithmic Physics frame. Because Jack thought plugging values into the angular velocity equation should give him an answer he could translate into a position on a Ferris Wheel, we interpret his initial epistemic form to be a Number. Jack hits a roadblock when he realizes he cannot solve the problem this way, and decides to change his strategy: Jack: So, I guess because this is a conceptual equation I am going to guess, for that, um, okay, so the greatest force you would be feeling (points upward) since youre going up, that means you would feel the lightest at the top, and the heaviest at the bottom. Because, like, youre going, accelerating faster at the bottom, and your mass is the same, so like the force would be greater at the bottom than it would be at the top. Because youre accelerating faster at the bottom than at the top. So, yeah, thats my answer.
Jack finishes the problem by applying what he knows about the physics phenomenon of force. We determined Jack finished the problem in a Conceptual Physics frame. Jack also switched from trying to obtain a mathematical expression as his answer to trying to reason his way to an answer, so we determined his final epistemic form to be Reasoning. As Jack expresses, the assessment question not including numbers causes him to change what he is doing. We interpret this as the assessment feature shifting his epistemic form, which shifts his epistemic frame.

VII. DISCUSSION
Our first claim is that Epistemological Frames (as identified by behaviors) can be defined by the Epistemological Resources used. Our analysis is consistent with much of the other work on problem solving suggests that we can make sense of students' problem solving work during assessment in much the same way we can make sense of their reasoning in non-assessment situations. When students engage in problem solving during an assessment, they draw on a range of epistemological resources to do so. In Rosenberg et al.'s paper, Multiple Epistemological Coherences in an Eight Grade Discussion of the Rock Cycle [49], researchers noticed groups of epistemological resources that consistently appeared together when students engaged in specific problem solving strategies that they called "epistemological coherences." Rosenberg et al. described that each group of resources was stable and reinforced the others. In our interpretation of this data, we understand their "epistemological coherences" to be epistemological frames as defined by student behaviors.
Additionally, defining frames by epistemological resources makes some intuitive sense. "Causal storytelling as an activity would not make sense in the Algorithmic Math frame where students are computing and executing well established protocol. However, one could imagine that students displaying the behavior associated with the Algorithmic Math frame would engage in Accumulating as they gather given information for their computation. "Number as an epistemological form would not make sense in a Conceptual Physics frame where students are thinking about physics phenomenon and reasoning their way to an answer, but "Reasoning" certainly would make sense.
Defining Frames in terms of resources also changes the way we think about their relationship to one another. This could provide a useful tool to researchers. For any given coherent thought given by a student, it may prove easier at times to determine three to four epistemological resources that can be matched with a particular frame than attempt to interpret sometimes ambiguous student behavior. Defining frames in terms of both resources and behaviors gives researchers on small teams an efficient way to check their frame coding for accuracy and consistency.
Our second claim is that novice physics students seldom shift frames. Our results agree with previous findings. As discussed in the Literature Review, Bing & Redish found that one marker of expertise in physics is the ability to switch between frames as necessary [48]. Ideally, students could shift frames when one fails to help them complete the task at hand. Students did not shift frames when their current frame proved unhelpful, such as in the specific example of the Gravitron. Rather than believing students perform poorly at switching frames, we believe students may be consciously choosing to begin and stay in the Algorithmic Physics frame. This could be due to some Physics textbooks which specifically call for students to solve physics problems algorithmically [57][58][59]. This could also be due to the course instructors who sometimes teach students an algorithm for solving physics problems. [60] [61] Our third and fourth claims are that Epistemological Frame shifts happen on the level of an individual Epistemological Resource and that assessment features can prompt that resource shift.
Our data implies that researchers could focus on how to shift resources as the means to the desired ends of frame shifting. We did not observe simply telling students to solve the problem conceptually or algorithmically to be effective in shifting frames. Notice that in the Gravitron problem (Figure 1 students are explicitly asked to show their reasoning through words, but this failed to shift students towards reasoning in a Conceptual Physics frame. The frame shifts we did observe happened due to a shift at the level of a single resource. Therefore, we argue that trying to shift an individual epistemological resource may be an effective way to shift a students epistemological frame.
Frame shifting as the result of a shift in a single epistemic resource is consistent with the findings of Rosenberg et al. Students began completing their worksheet about the Rock Cycle by trying to accumulate as much information as possible from the external sources of information presented to them. After hearing them struggle, their teacher intervened by telling the students to "start with what you know." Their teacher suggested the students shift their source of knowledge, and the way the students engaged with their assignment changed. Rosenberg et al. noted the underlying meaning of her statement which was that students have many ways of engaging with assignments. More importantly, Phelen, the teacher, implied that the epistemological coherence students work in can be prompted to change by a single phrase when she asked students to "start with what [they] know."

A. Future Research
Future work will explore the ability to do the reverse and make assessment questions that prompt resource shifts. A list of recommendations for prompting resource and frame shifts in assessment writing will be created sent to instructors for field testing. This list will be revised based on results from the field testing and published for instructor use.
Additionally, future work will expand upon claims 3 and 4 to explore if a single resource shift is a necessary or only a sufficient condition for frame shifts.

VIII. CONCLUSION
In this work we explored the relationship between epistemological frames and epistemological resources during student engagement in assessment.
Students in this study typically began problems in the Algorithmic Physics frame by writing down their givens, determining what they are solving for, and searching through their equation sheet to see what they can manipulate to return a value. To those of us who have taught in introductory physics courses this finding is not surprising. In fact, it implies that the Algorithmic Frame is well-rehearsed in the students' physics courses, either in lecture, homework, assessments, or some blend of the three.
While students had a fairly stable initial frame, we found that influences like intervention from authority (i.e. the interviewer) and assessment features can be effective at shifting an individual epistemic resource, which shifted the students' epistemic frame. This finding suggests that although the Algorithmic Physics frame is "sticky", student epistemological framing during assessment can be fluid in much the same way as it is during other classroom activities. [62] These findings suggest a need to attend even more closely to assessment than we already do. We often think of assessment as occurring after instruction -something "added on" at the end of instruction for us as instructors to get information about our students. But this work highlights that students are also getting information from assessments -information that might impact their understandings of how they should engage in physics learning. IF that is the case, then we as instructors and researchers need to pay more attention to the kinds of messages we might be sending with our assessments to be sure they align with our goals for the course. Specifically, we must pay attention that we are cuing specific epistemological resources that will be productive for students. Additionally, we need to be intentional about how we do that cuing so that it leads to substantive reframings.
Further, in this work, we used the perspectives of epistemological framing and resources to make sense of student engagement in assessment. The fact that this analysis "worked" at all is in and of itself an important result. It was possible that the cognitive mechanisms that have been used so productively to understand learning just would not have been useful in this different context. However, we found that framing analysis is consistent with student behavior during assessments. Additionally, the fact that our findings align with other work on problem solving outside of the assessment context is also noteworthy. Again, it was possible that what students did during assessment would be entirely discontinuous with their engagement in learning. Thus, our findings that students do rely on problem-solving-like framings and their associated resources during assessment are noteworthy. Specifically, it encourages us to re-examine our own assumptions about whether and how assessment is different from other classroom activities -both for ourselves and for our students.