Investigating undergraduate students ’ views about the process of experimental physics

Students sometimes learn about a model of the “ scientific method ” that is linear and clear cut. While this approach may have pedagogical advantages, it does not reflect how science is often done in practice. The Experimental Modeling Framework (EMF) describes the complex and iterative process of experimentation in the domain of physics, including comparing predictions with experimental data and enacting revisions to models and apparatus. We conducted interviews with 10 undergraduate students who had just completed an advanced physics lab course in order to investigate their views about the EMF as a depiction of the process experimental physics. We report the results of a thematic analysis that investigates students ’ views about the EMF and explores the extent to which students identified iteration as an important aspect of experimental physics. Generally, the students in this study found the EMF to be reflective of the process of experimental physics. They identified several distinct differences between the EMF and a traditional depiction of a linear scientific method (SM), including the lack of iteration in the SM, as well as the importance of asking questions and reporting results, which is absent from the EMF. Additionally, student discussions of the fundamentally iterative nature of science were most likely to occur during direct comparisons of the EMF and SM. We discuss implications of this study for both research and instruction. We suggest that, in a lab course where iteration is a goal, explicit in-class discussions and comparisons of models of the process of experimentation could be beneficial for students ’ epistemological development. DOI: 10.1103/PhysRevPhysEducRes.18.020146


I. INTRODUCTION
In the undergraduate physics curriculum, one goal of many lab courses is for students to learn about the process of experimental physics [1][2][3][4][5][6][7][8][9]. Often, students have an understanding that the process of science is rigid, linear, and clear cut-you start with a question, construct a hypothesis, conduct an experiment to test that hypothesis, and then find out if your hypothesis was correct or not [10]. This idea about the nature of science is pervasive in our culture, and not unique to the discipline of physics [10][11][12][13]. While there are likely many benefits to introducing the methods of science in this way, the assumption that the scientific process is linear can be misleading. When students arrive in our undergraduate courses, we may need to help them overcome this rigid notion of science and help them to see, through participating in authentic scientific practices, that the process of science is iterative and nonlinear.
We have anecdotal evidence from our experience teaching physics lecture and lab courses that students have to work against this ingrained notion of a linear and rigid scientific method in order to understand that iteration and troubleshooting are an integral part of the construction of scientific knowledge, and that there is no one "correct" method that defines science [10,14,15]. For example, many years ago the second author was teaching an advanced laboratory course and there was a pair of senior physics majors working on a project in which they were using a Michelson interferometer to measure the wavelength of a HeNe laser. The apparatus was old and the students encountered issues when trying to count fringes on the interference pattern. Unprompted by the instructor, these students engaged in a thorough and sophisticated revision process in which they took the apparatus apart, cleaned and oiled it, and then constructed a system including a photodiode and an oscilloscope to more optimally count the fringes. Impressed by their initiative and development of experimental skills over the course of the project, the instructor asked these two students at the end of the course what their plans were after graduation. One of the students replied, "We always thought we wanted to go on and do something in experimental physics, but this course has taught us that we're not cut out for it." Further prompting revealed that the students felt that because they had not started from the beginning of the experiment and marched through a methodical series of steps until completing the objective, that they had failed. They saw the iteration and troubleshooting that they engaged in as an indicator that they were not cut out for experimental physics.
In our teaching, we have seen students internalize the idea of a linear and rigid scientific method. Over several semesters of a middle-division modern physics lecture course in which students have to read and discuss a chapter on the nature of science [16], the first author has seen many students comment on the tension between the idea of a linear scientific process and the more complex scientific process they are now beginning to experience and learn about in their undergraduate physics or science courses. For example, one student wrote in an online discussion about the reading, "It is interesting how the author mentions that science does not follow a rigid process and backs that up with examples, but yet from elementary to high school we are taught that it is a rigid, step-by-step process that is always followed and necessary." The data we present in this paper will provide further examples of students recognizing the traditional linear depiction of science as something that is intimately familiar to them. In the discussion of the reading, another modern physics student wrote, "when we're taught about science and science experiments it all spans from "the scientific method." I never stopped to think that using essentially a cook book recipe to solve problems isn't the way most of the brilliant ideas people have…come up with [were derived]." As in our anecdote of the two advanced lab students who perceived their iteration around their experiment as a personal failure, we think, at the undergraduate level, this notion of science as a linear and rigid process can be detrimental to students' learning and to their identity development in the domain of science. Scholars of science education have been writing for decades about how it is misleading to portray science only as a set of discrete processes or skills [10,13,14,[17][18][19]. This distilled outline of scientific thinking, in one form or another, is still taught to many students today despite the reality that science in practice is not a rigid set of rules to follow and there is no one correct scientific method within or across science disciplines [10,12,13].
The manner in which science is portrayed has consequences for aspiring scientists' learning, public understanding of science, and the relationship between science and society [17]. We have the opportunity (and responsibility) in our lab courses to portray the practice of experimental physics in a way that is true to the myriad ways that physicists actually engage in experimentation, and to attend to our students' views of, and experiences with, experimental physics.
As part of a broader research project, we partnered with instructors of advanced physics lab classes to investigate implementation of, and student learning during, studentdesigned multiweek lab projects. One of the instructors' goals for the projects was to get their students to be iterative in their experimental work. That is, they wanted students to experience experimental physics as a dynamic process in which you have to continually refine your questions, models, and/or apparatus rather than a linear process of walking through a series of predefined steps. Motivated by these instructors' goals around iteration, and the pervasiveness of (and potential harm caused by) the myth of a linear scientific method, we conducted a study to investigate advanced lab students' views about models of the process of experimental physics. Understanding students' views at the culmination of an advanced lab course with open-ended projects can help us to better understand what informs students' perceptions of the nature of science and how to support that development in our lab courses. This study is not an evaluation of particular lab courses or pedagogical approaches, but rather is an investigation of students' views situated in a particular context.
In this paper, we report on interviews with advanced lab students in which we have students reflect on the Experimental Modeling Framework (EMF) as a model of the process of experimental physics [20,21]. Students discuss the ways in which the EMF does and does not reflect their understanding of experimental physics and compare it with a more traditional linear depiction of the scientific method (SM). Through a coding analysis of the interviews, we explore students' views about the process of experimental physics, with particular attention to their discussions of revision and iteration. We present the results of this analysis along with implications for both research and instruction. In the following section, we review relevant literature in physics education research about epistemologies of experimental physics, to which this work contributes.

A. Student epistemologies of experimental physics
Students' epistemologies of physics (i.e., their beliefs about the nature of physics knowledge and what it means to learn and know physics) [22] can influence their reasoning [23], conceptual learning [25,26], course performance [27], and interest [28]. As such, supporting students' epistemological development is a major area of research within physics education [24,29], including, specifically, research on student learning in laboratory courses [30].
Studies of students' views about the nature of doing and learning experimental physics have been primarily quantitative studies using assessment instruments such as the Colorado Learning Attitudes about Science Survey for Experimental Physics (E-CLASS) [30][31][32]. These studies show that courses that use traditional guided labs tend to see negative shifts on the E-CLASS (i.e., students exhibit less expertlike beliefs after a semester of instruction), whereas courses that use research-based pedagogical approaches [e.g., Investigative Science Learning Environment (ISLE) [33], Modeling Instruction [34]] see no shift or slight positive shifts [35]. Of particular relevance to our present study, these studies also show that lab courses that include one or more weeks of open-ended activities see small positive shifts on the E-CLASS pre-to postinstruction [36].
In a qualitative analysis of open-ended prompts added to the E-CLASS, Hu et al.'s findings speak to the benefits of open-ended lab activities for students' epistemological development [37]. They found that many students agreed that the primary purpose of physics experiments was to confirm previously known results, but also agreed that physics experiments contributed to the growth of scientific knowledge. Students who held these seemingly contradictory views explained that the goal of experiments in their lab classrooms is to confirm known results in order to support their conceptual understanding, thus contributing to students' personal knowledge growth. The authors suggest that "incorporating some lab activities for which the outcome is not known to either the students or instructor might have a significant impact on students' understanding of the importance of experimental physics as a mechanism for uncovering new physics and driving the creation of new theoretical models" [37] (p. 10). Further, in this study, students argued that it was not necessary to understand the equations or underlying physics concepts in order to complete an experiment for class, while also recognizing that this was not reflective of authentic experimental physics. The authors thus suggest, in line with other research [5,35,38,39], that instructors should steer away from traditional prescriptive labs in order to provide opportunities for students to reflect on their experimental set up and methods.
Open-ended projects, for which there is not a predetermined outcome and that necessitate reflection about experimental processes and methods, have potential to support students in developing sophisticated views about the nature of experimentation. Further, lab courses and research experiences at the upper-division level may be particularly impactful for students' epistemological development. Hu and Zwickl conducted an analysis of open-ended survey responses from a broad population of physics students and found that, when compared to introductory-level students, upper-division undergraduate and Ph.D. students identified a wider range of unique benefits of experiments and demonstrated a more holistic view of the relationship between experiment and theory [40]. This is likely due to both students' experiences in courses and selection effects.
Other research has explored epistemological aspects of upper-level lab courses and provides suggestions for how to support students' development of these sophisticated views about experimental physics. Dounas-Frazer and Lewandowski identify the idea that "nothing works the first time" as an expert epistemology of experimental physics, and find that helping students learn how to troubleshoot and to see troubleshooting as an important aspect of experimental physics are goals of many lab instructors, especially in upper-division electronics courses [41]. At the beginning of these courses, if students do not expect to have to engage in troubleshooting or encounter and overcome problems with their apparatus and experiments, open-ended projects can be frustrating experiences. In courses involving student-designed multiweek projects, other research has demonstrated correlations between students' views about experimentation and their sense of ownership over their projects [42], and suggests that student ownership is characterized by emotions that fluctuate in time in cycles of frustration and success [43]. In an advanced lab course that includes multiweek open-ended projects, Eblen-Zayas reports that reflective class discussions and individual written reflections helped normalize students' frustration with their project and the struggles of experimental physics in general, in addition to increasing students' confidence around conducting experiments [3]. In a study of upperlevel optics and lasers lab courses that contain multiweek projects, Dounas-Frazer et al. find that students' views about what constitutes experimental physics are shaped by their own experiences with their projects and their perceptions of peers' experiences [44]. Students in their study identified execution-oriented activities (e.g., troubleshooting or keeping a lab notebook) as necessary aspects of experimental physics. Interpersonal (e.g., asking for help) and fabrication-oriented activities (e.g., building electronics) were viewed as conditional, dependent on an individual's expert or novice status and availability of apparatus. Students found propagation-oriented activities (e.g., oral presentations) to be important for experimental physics but explained that there were a variety of possible avenues for sharing scientific work that might be included.
We investigate students' views about experimentation in the context of advanced lab classes with open-ended projects, given that these contexts are particularly important and uniquely situated to support students' epistemological development.

B. Experimental modeling framework
Along with providing opportunities for epistemological development, project-based labs can also engage students in the process of modeling. The ability to construct, use, test, and refine models of physical systems is a common goal across the undergraduate physics curriculum, particularly in laboratory courses [2,8], and many people advocate for model-based instruction in physics [26], and science more generally [10]. A scientific model is generally a representation of a system or phenomenon of interest, used for explanatory and predictive purposes and based on prior knowledge of principles or concepts relevant to the system. Models are simplifications of real systems, containing assumptions and limitations, and thus are tentative and require refinement [20]. In this context, "modeling" refers to the process of creating, evaluating, and refining scientific models.
The EMF [20,21,45] (Fig. 1) describes the model-based reasoning that is central to experimental physics, and depicts the process of experimentation as nonlinear and iterative. It separates this process into five distinct subtasks: • Make measurements-interaction between the measurement equipment and the physical system that results in raw data • Construct models-creation of models for both the physical system and the measurement system using relevant principles and concepts, particular parameter values, and appropriate assumptions and simplifications • Make comparisons-comparison of the prediction from the physical system model and the interpreted data,  [45] as a refinement of the original framework in Ref. [20]. The framework consists of five subtasks-make measurements, construct models, make comparisons, propose causes, and enact revisions-that are arranged in a flow chart that offers many possible paths through the process of experimentation.
in order to answer the question, "Is the agreement good enough?" • Propose causes-when the agreement is not good enough, generation of hypotheses for sources of the discrepancy between prediction and data • Enact revisions-informed by proposed causes, revision of the physical system apparatus or model or the measurement system apparatus or model A unique feature of this framework is that it distinguishes the physical system (right side of the diagram) from the measurement system (left side of the diagram), emphasizing the fact that physicists must construct, evaluate, and refine models for measurement systems and apparatus in addition to the physical system or phenomenon of interest. The five subtasks are arranged in a flowchart that offers many possible paths through the experimentation process. The arrows looping back up to the top, from enact revisions to either make measurements or construct models, as well as the complexity of the diagram, reflects the iterative and nonlinear nature of experimental physics.
In prior and ongoing research, the EMF has been used to characterize students' model-based reasoning in thinkaloud interviews and problem solving activities [20,46], guide course transformations [47], assess students' modelbased reasoning in lab coursework [48], and inform the creation of a research-based assessment instrument designed to measure students' modeling skills [49]. In the context of experimental physics at the undergraduate level and beyond, the EMF is well-established as being representative of the authentic practice of experimental physics [21,45,47,50,51]. In this study, we investigate students' views of the EMF as a depiction of the process of experimental physics.

III. RESEARCH QUESTIONS
Motivated by instructors' goals of having students engage in iteration during multiweek projects and by the understanding that our physics undergraduate curriculum may have to work against an inaccurate view of the process of science that pervades our society, we conducted an investigation of students' views about the EMF as a model of the process of experimental physics. In particular, we wanted to understand the views of students who had just completed open-ended projects in an advanced lab course, given that this particular context has been identified in prior literature as presenting unique opportunities for students to experience and understand authentic experimental physics and may provide experiences that shape students' views about experimental physics [3,36,37,44,52].
In order to elicit student thinking about the process of experimental physics, we showed the EMF to advanced lab students and had them reflect on it as a depiction of the process of experimental physics and of what they did during their projects. We also showed students a common depiction of the traditional linear SM (see Fig. 2) and had them reflect on the EMF in contrast to a linear SM. For the remainder of the paper, when we refer to the SM we are referring to a representation of the process of science as a linear progression of a series of discrete steps [10][11][12]. Our goal in this study was not to further validate the EMF, but instead to use it as a tool to prompt discussion and reflection.
We address the following three research questions.  Fig. 1). On the right is a linear scientific method, a common depiction of the process of science.
3. In discussing the EMF, to what extent do advanced lab students identify iteration as a key feature of experimentation? In this study, we focus on students' views about the EMF as a representation of experimental physics, while noting that these views are situated in context and inform, or are informed by, students' experiences [53]. After addressing the above three research questions, we discuss the ways in which students' views relate to, or may be informed by, their specific experiences with their projects. Future analysis will look more in depth at the ways in which students' experiences with their projects may impact their views about the process of experimental physics.

A. Course context
The students in this study came from advanced lab courses at three different institutions across the U.S., representing a variety of institutional contexts: private and public, selective and inclusive, predominantly white and Hispanic-serving, and offering bachelor's, master's, and doctoral degrees in physics. As part of a broader, multiyear research project, we partnered with instructors at each of these three institutions who were teaching advanced lab courses for physics majors that incorporated a studentdesigned multiweek final project. Detailed descriptions of these courses can be found in Ref. [56]. One overarching goal of these courses is to prepare students to conduct research, or to provide them with opportunities to engage in authentic experimental physics. In each of the courses, students work in groups of 2-3 to propose, design, conduct, and report on their own experiments. They write proposals prior to conducting their projects and then report the findings in a summative oral presentation or written report. It is within this context that we interviewed students about their beliefs of how the EMF represents experimental physics, and their experiences in the advanced lab courses and with their final projects.
This interview study was conducted at the end of the Spring 2020 term, and the usual operation of the three courses was impacted by the COVID-19 pandemic to varying degrees. Course 2 took place almost entirely inperson as originally planned, save for students' final presentations of their projects, which were conducted online. Course 3 switched from in-person to remote for the last few weeks, just as students were beginning to carry out their projects. Some students were able to go into the lab one person at a time to complete their project, communicating with their lab partner via video conference or email, while other students took equipment home or shifted their project to something they could conduct remotely. The entirety of course 1 was conducted remotely, with students working in groups on their final projects from their respective homes. In some instances, each group member had an apparatus or experimental setup in their home, while in other instances students divided the work such that one student had the apparatus, another student worked on data analysis, etc.

B. Interviews
We recruited students from each of the three courses to participate in interviews at the culmination of their course. Participation in the interviews was voluntary, completely decoupled from the course grade (instructors did not know who participated), and students were told that the interview would be an opportunity to reflect on their learning of, and beliefs about, experimental physics, as well as a chance to help improve lab courses for future students at their institution and nationally. Interview participants were compensated for their time, and all interviews were conducted by the first author via video conference.
Across the three courses, 10 students participated in interviews-3 from course 1 (course enrollment ¼ 24), 2 from course 2 (course enrollment ¼ 4), and 5 from course 3 (course enrollment ¼ 21). All of the interviewees were either physics or applied physics majors; there were two sophomores, five juniors, and three seniors (the interviews were conducted at the end of the winter or spring term, so these students were about to complete, or had just completed, their sophomore, junior, and senior years respectively). When asked optional questions at the end of the interview about their gender and race or ethnicity, two interviewees identified as female, one identified as transgender, and seven identified as male. Nine of the participants identified as White, and one identified as Hispanic/Mexican-American. We report these identities exactly as students reported them to us (i.e., the student who identified as transgender did not also specify "male," "female," "nonbinary," etc.).
Aside from the logistical constraints of the course operation, all students and instructors were under a large amount of stress due to the varied and widespread impacts of the global pandemic. Amidst these disruptions and stresses, the courses continued to operate and so we continued to collect data as originally planned, making modifications where necessary. In recruiting for, and conducting, interviews, we were sensitive of the fact that students were dealing with many additional burdens. As with any interview study, we strove to make connections with students, build rapport, and listen intently to their ideas, perspectives, and experiences. We discussed the ways in which the pandemic was impacting students only if and when they brought it up on their own. Though the students were still able to conduct final projects, we acknowledge that the disruptions due to COVID-19 likely impacted their experience.
The interviews lasted between 39 and 57 min with an average of 46 min, were semistructured in nature, and contained a variety of questions about students' experiences in their advanced lab courses, their views about experimental physics, and their experiences with their final projects. For the present analysis, we focus only on a subsection of each interview, in which we asked students about the EMF and the SM. This section of the interview lasted between 6 and 15 min, with an average of 11 min (not including the time spent introducing the EMF). We split this subsection into two parts for the purpose of analysis: (i) modeling framework, and (ii) comparison.
In part one, the interviewer shared her screen and first showed the student a simplified version of the EMF [20,45] (shown on the left in Fig. 2). We used a simplified version of the full framework so as to reduce cognitive load for the students as much as possible. The simplified diagram includes all of the components of the full framework, including a distinction between the measurement and physical systems, but in less detail. After briefly walking through each step of the diagram and explaining to the students what it means (e.g., what we mean by "models," and distinguishing between the measurement and physical systems), we paused to give the students a chance to ask any clarifying questions about the diagram. This introduction of the EMF typically lasted around 4-5 min. Once students felt comfortable continuing, the interviewer asked them to share their thoughts about the EMF. The primary prompt was, "In what ways do you think this diagram describes the process of experimental physics (or not)?" with follow-up questions based on the students' responses. We also asked the students if the EMF reflected any aspects of their final projects.
In part two, the interviewer showed the student a depiction of the SM (shown on the right of Fig. 2), and again paused for clarifying questions if necessary. Once students were ready to continue, the interviewer asked them to compare the two diagrams side by side. The primary question for part two was, "What are some similarities and differences you see between these two depictions of the process of experimental physics or science?" again with follow-up questions based on students' responses. We also asked students which diagram was a more realistic representation of their project.
We created this particular image of the SM (see Fig. 2) because it is similar to various diagrams depicting the process of science that we have encountered over many years of teaching and conducting research. We chose to depict a method that was completely linear, rather than one that contained a feedback loop or any amount of revision, because we wanted to probe students' identification (or lack thereof) of iteration as an important aspect of experimental physics, and because a simplified linear depiction of science is common in our culture [10][11][12][13]. As such, we chose a depiction of the SM that most contrasted the iterative and complex nature of the EMF. Intentionally, we did not ask students directly about the idea of iteration (i.e., the interviewer never used the words "iterate," "iteration," "feedback loop," "cycle," etc.). Instead, we showed them the two diagrams side by side and asked them to compare and contrast. We took this approach because we wanted to see what students would identify and talk about without too much prompting (other than the prompting from the visual representation of the diagrams themselves).
The full list of questions from this subsection of the interview is provided in the Supplemental Material [57].

C. Analysis
We transcribed the interviews, isolated the subsection of each interview about the EMF and SM, and conducted a thematic coding analysis on these transcript excerpts. Codes were both a priori and emergent, and categorized into three main sections of the codebook corresponding to our three research questions: modeling framework, comparisons, and iteration. The first part of the codebook was formed by coding students' responses to the initial question in part one of the interview (how does the EMF describe the process of experimental physics?). Codes in the second part of the codebook describe the similarities and differences the students identified when comparing the EMF and SM in part two of the interview. The third part of the codebook identifies the extent to which students recognize or talk about revision and/or iteration; these codes were applied to both parts one and two of the interview. For each of these three sections, we created and refined codes in an iterative process until the codes were clearly defined and distinguished from one another and the codebook captured all of the main ideas and themes in students' responses. This involved doing a first pass through all the data, creating a draft codebook, applying those codes, discussing with the research team where revisions to code definitions or merging or splitting of codes was required, refining the codes, applying the new codes to the data, and so on. At each step of the process, we revisited the research questions to ensure that the three parts of the codebook would provide us with information to be able to answer each of the three questions. Resulting codes for the main sections of the codebook that address our three research questions are given in Sec. V along with example quotes. The full codebook including operationalized definitions is available in the Supplemental Material [57].
Upon finalizing the codebook, two researchers independently coded a subset of the data (two interviews). Percent agreement between the two raters was initially 94%, and reached 100% after discussion of disagreements. We report percent agreement here instead of Cohen's kappa because the prevalence of individual codes was low across the small dataset, thus rendering the kappa statistic unreliable [58]. Upon establishing interrater reliability, author J.H. coded the entirety of the dataset.

D. Limitations
There are two possible limitations of this study. First, our sample size is relatively small (10 students), and as such, the extent to which we can generalize the results is limited.
In making conclusions from this study, we do not attempt to generalize to all undergraduate physics students, or even to all of the students enrolled in the three courses. Instead, we report these advanced lab students' views about models of the process of experimental physics as examples of the kinds of views students might hold, and to consider how we can further support students' epistemological development in our lab courses.
Second, because the courses in which our student participants were situated were affected by the restrictions and changes to teaching due to the COVID-19 pandemic, students' views about experimental physics that they reported in our study may have been impacted by the remote teaching modality and/or the general stress brought on by the pandemic. That is, because some students in this study were unable to physically be in the lab and conduct traditional experimental projects, they missed out on some aspects of authentic scientific practice, which may have impacted their perceptions of the nature of science. Trauma and stress from the pandemic may also have made it difficult for students to focus on their schoolwork, which could have negatively impacted experiences with, and perceptions of, experimental physics. One prior study of over 3200 introductory physics lab students found no net differences in students' views about experimental physics (as measured by the E-CLASS) from 2019 to 2020 during remote instruction [59], and a prepandemic study found no difference in students' epistemological beliefs between online and in-person labs [60]. Additionally, students' epistemologies develop over long periods of time [61], and thus we would not expect a few weeks to completely disrupt students' views that they have developed over years in an undergraduate physics program (and before). Nonetheless, we know that the time period in which this study took place was unusually stressful, and the possibility remains that the remote instruction environment and surrounding pandemic context may have impacted the views of experimental physics students shared with us for this study. This paper is not an evaluation of the impact of specific courses. Rather, we investigate students' views about experimental physics, which have been informed by the total of their experiences, including experiences with conducting experiments in a remote teaching and learning situation.

A. RQ1: Student views about the Experimental
Modeling Framework To answer the first research question, we coded students' responses to the initial interview questions about the ways in which the EMF describes the process of experimental physics. These responses capture the students' initial reactions to, and perceptions of, the diagram as it relates to the nature of experimental physics. This coding analysis resulted in eight codes for the key features or benefits of the EMF that students identified (shown in Table I).
Testing model or hypothesis refers to the idea that a key feature of experimentation is the goal of testing or checking a model or hypothesis. Measurement system was coded when students talked about the distinction between the physical and measurement systems being a key feature of the EMF, or emphasized the need to understand, model, and revise the measurement system of an experiment. Iteration "I like how it's oriented in a loop that never ends. That's…very…true." Intuitive "I haven't like seen it framed this way before and it's intuitive and I think it's good." Model construction refers, in general, to creating a model. Make comparisons refers to the idea that one crucial step in experimental physics is comparing an experimental result to a prediction or expectation. Propose causes is about the act of proposing causes for a discrepancy between data and a prediction. Revision refers to the act of revising or changing something about an experiment as being a key part of the process of experimental physics. The iteration code, more generally, is an identification of the nature of experimental physics as iterative and a continual process of revision.
Although these ideas of revision and iteration exist on a spectrum (from a single revision to a continual process of revision), we opted to distinguish between them in our analysis because students in our study talked about them in different ways. Often when talking about revision, students mentioned something specific going wrong in an experiment and needing to revise or refine some aspect of the experiment in order to fix it. On the other hand, when students talked about iteration it was more about the fundamental nature of science (e.g., experimental physics is a "loop that never ends" or a "loop of continuous learning"). We investigate students' views about revision and iteration further in Sec. V C.
The last code, intuitive, was applied to a response from one student who said that they thought the EMF described the process of experimental physics well because it was "intuitive." We interpret this code as a confirmation that, for this student, the EMF represents a process of experimental physics that they have engaged in in the lab (i.e., "this makes sense, because this is something that I have done"). Example quotes for each of the codes are given in Table I.
When presented with the simplified version of the EMF, interview participants most commonly identified revision and make comparisons as key features of the framework that reflected the process of experimental physics. We discuss the prevalence of the revision code in Sec. V C. The next most common code was measurement system, followed by testing model or hypothesis and proposing causes. Only two people identified iteration at this stage of the interview (see more in Sec. V C). One person talked about model construction, and one person said the EMF was intuitive.
From these results, we find that the students in our study generally identified the EMF as a description of the process (or parts of the process) of experimental physics (i.e., they identified a wide variety of ways in which the EMF reflects experimental physics). Collectively, the key features they identified encompass the entirety of the framework. In particular, five of the eight codes correspond, partially or directly, to the subtasks of the framework (make measurements, construct models, make comparisons, propose causes, and enact revisions, as labeled in Fig. 1). A sixth code corresponds to iteration, which is central to the framework and is captured by the nature and prevalence of the arrows in the diagram.
Earlier in the interview, before the section in which we showed students the EMF, we asked students one question about their views of experimental physics in general. The prompt was "If you were talking to a first year physics major, what would you tell them are the important features or processes of experimentation that you would want them to understand?" As a check to make sure the responses presented in Table I were not only the result of students reading off the EMF diagram, we reviewed their responses to this earlier interview question, before they had seen the EMF diagram. In response to this question, the ten students brought up a variety of ideas, many of them related to the codes in Table I, including: understanding the limitations of your measurement device (measurement system code), understanding the theory behind your experiment (model construction code), conducting data analysis and attending to measurement uncertainty (related to the make comparisons code), and understanding that your experiment will likely not work the first time and that you will need to be persistent through a continual learning process (revision and iteration codes). Thus, we conclude that the results from the first section of the coding analysis shown in Table I are not simply a result of students repeating back to us what is listed on the diagram. Rather, these are ideas that students held and talked about before being prompted to consider the EMF. In some cases, the EMF may have given students new language to put to their ideas (e.g., "comparison," "revision").
Though the goal of this study was not to validate the EMF with students, it is an important result that students who have just completed an advanced lab course find the framework (or parts of the framework) to be a useful and accurate depiction of experimental physics. It is perhaps an indication of these students' learning and epistemological sophistication that they recognize what we see as the central features of experimental physics. We find that, in a short amount of time, these students are able to digest this complex framework and reflect on it in nuanced and meaningful ways.

B. RQ2: Student comparisons of the Experimental Modeling Framework and a linear scientific method
In order to answer the second research question, we asked students to directly compare the EMF and SM diagrams and coded their responses for common similarities and differences that they identified. Many of the students began the comparison by mapping one diagram onto the other. The most common idea was that the EMF occupies the three middle boxes of the SM (construct hypothesis, test hypothesis, and analyze data and draw a conclusion). Some students specifically mentioned that the EMF loops back and forth between these three major sections of the SM. Other common connections that students identified were between the physical system model construction and prediction in the EMF and construct hypothesis in the SM, as well as between comparison in the EMF and analyze data and draw a conclusion in the SM. Additional similarities that students identified were that both diagrams are "procedural," and both begin with a lack of understanding about something.
While students were able to identify similarities between the two diagrams, most of the conversations in part two of the interview centered around the differences between the EMF and the SM. There are four emergent codes that describe the differences that students identified (shown in Table II, with an example quote for each code).
The most common difference that students talked about was that the EMF is cyclic and the SM is linear (cyclic vs linear code). They often referred to the EMF as a loop and identified that this opportunity to go back to the beginning to revise, retest, or check was missing from the SM. Several students said that this linear nature depicted by the SM was not an accurate reflection of the process of science. We discuss this further in relation to the third research question below (Sec. V C).
Another common difference that students talked about was the fact that the SM depicts the whole process of science, including developing a question, doing background research, and communicating results once the experiment is complete. In contrast, they recognized that the EMF focuses on the actual execution of an experiment and not these other aspects of the overall scientific process (experiment vs whole process code). This is aligned with the students who mapped one diagram onto the other and said that the EMF occupies the middle three boxes of the SM. The third main difference that arose in students' responses was the complexity of the EMF versus the simplicity of the SM (complex vs simple code). Students either referred to the visual representations, stating that there were more elements to the EMF and more possible paths than in the SM (in which only one path is depicted), or they talked about the EMF being harder to understand at first but being more suitable to complex scientific questions. This code is closely related to the cyclic vs linear code, but not all students who mentioned complex vs simple also mentioned cyclic vs linear. Less common, but mentioned by two different students, was the distinction between the EMF being physics specific while the SM is a more general representation of science that might apply broadly to many different science disciplines (physics vs general science code). One of the students elaborated further to suggest that the physics-specific element of the EMF might be the consideration of the measurement system on its own, apart from the physical system.
In their comparisons, students found benefits to each of the diagrams, though, collectively, exhibited a preference for the EMF as a more accurate depiction of the cyclic and complex process of experimental physics. The advanced lab students in this study were able to recognize that the linear SM depiction can be misleading.
The few areas in which the students thought the SM better reflected their understanding of science was the inclusion of developing research questions, doing background research, and communicating results. Several students noted that, in addition to the execution of an experiment, these steps are vital to the process of science. Students also expressed these ideas earlier in the interview before the section in which we showed them the two diagrams. When discussing important features and processes of experimentation at this early stage of the overall interview, students talked about needing to have a clear objective, engaging in literature reviews to understand prior research before starting an experiment, and the importance of communicating results and processes in a lab notebook. Thus, as with the key features of the EMF (Table I), Physics vs general science "[The modeling framework is more] specified to physics research than the scientific method. The scientific method is great and all, but that follows more along the lines of…a broad, general, interdisciplinary method, whereas the modeling framework…is a lot more specified to the physical system." we conclude that these elements of the SM that students identified as being important are ideas that students already held about the scientific process and were not only prompted by the diagram on the screen. These results suggest that the students in our study have a holistic view of the process of experimental physics. They focused on the importance of revision, making comparisons (Table I) and the cyclic and complicated nature of science, but also on the role of developing questions and communicating results. In the following section, we further explore students' ideas about revision and iteration.

C. RQ3: Student discussions of revision and iteration
To answer the third research question, we coded students' responses at any point in the subsection of the interview for whether they mentioned the ideas of revision and/or iteration. In the results for RQ1 above (Sec. VA), two of the codes for key features of the EMF that students identified were revision and iteration. In order to explore students' views about these processes more deeply, we extended the application of these two codes beyond the initial discussion about the ways in which the EMF may represent experimental physics. That is, at any time during the student's discussion of the EMF, SM, or comparison of the two diagrams, we coded instances of students directly mentioning or implying the importance of revision and/or iteration to the process of experimental physics. The definitions of these codes are the same as articulated above, with revision referring to a single modification or refinement of some part of the experiment and iteration referring to the nature of science being fundamentally iterative.
In addition to coding instances of students talking about revision and iteration, we coded for when in the interview these ideas came up, in order to account for different levels of prompting. There were three distinct times when students brought up these ideas: immediately, at the comparison stage, and when prompted to consider what was missing from the SM. "Immediately" refers to the beginning of the interview subsection that we analyze in this study. If a student responded to the initial question about the ways in which the EMF describes the process of experimental physics by talking about the importance of iteration, we coded that as an immediate mention of iteration. The interviewer did not use the word iteration or explicitly draw attention to the arrows in the diagram, so as not to prompt students directly to consider iteration. Instead, we were interested to know if students would bring this idea up on their own. However, as the interview progressed, there were increasing levels of prompting due to the nature of the questions. The second time that students commonly discussed revision and/or iteration was at the comparison stage, i.e., when they were looking at the EMF and SM diagrams side by side (Fig. 2) and identifying similarities and differences. Though the interviewer still did not ask directly about the importance of revision and iteration, we note that the iterative versus linear nature of the two diagrams is visually striking when viewing them side by side. Lastly, in some of the interviews in which students expressed a preference for the EMF or said that the EMF was a better representation of science than the SM, we asked as a follow up question, "Is there something missing from the SM diagram that makes it less accurate as a representation of science?" If a student answered this follow-up question by talking about revision or iteration (e.g., the arrows looping back to the beginning), we coded that as revision or iteration at the "what's missing" stage. Though the interviewer still did not ask directly about revision and iteration, this situation provides the most prompting that might lead students to think about iteration. The prevalence of the revision and iteration codes, along with when in the interview they first occurred, is given in Table III.
Nine of the ten students talked about revision and/or iteration at some point during the interview-one person talked only about revision, two people talked only about iteration, and the remaining six talked about both revision and iteration (though not necessarily at the same time, or in response to the same question).
All seven of the students who identified revision did so immediately, in answer to the interview question about the ways in which the EMF represents experimental physics. Students' discussion of revision ranged from pointing out that the existence of a revision process in the EMF was important to a discussion of specific revisions you might make in an experiment, like revising the model of your system. Examples of the former are displayed in the following two quotes: TABLE III. Prevalence of the revision and iteration codes at any point during the subsection of the interview analyzed for this study. We also report when during the interview students first mentioned the ideas of revision and iteration. Immediately refers to the beginning of the interview subsection in which we presented the EMF diagram to students and asked in what ways they thought it was (or was not) reflective of experimental physics. Comparison refers to the comparison stage of the interview in which we presented students with the EMF and SM diagrams side by side and asked them to compare and contrast. What's missing refers to the interviewer prompting the student to consider if there was anything missing from the SM after expressing a preference for the EMF. • "just having that revision process is really important" • "in all experimentation…it'd be a miracle if you got it right on the first try. And so like, you always need to go back and revise it." An example of a student mentioning more specific revisions is seen in the following quote: • "And then we see the results from that model that we tested, and to the hypothesis that we have created. And if [there is a discrepancy], then we go back and try a different model or refine the model that we started with." Reflected in some of the student quotes (e.g., second bullet point above) is the expertlike epistemology that "nothing works the first time" in experimental physics [62]. In discussions about the EMF, students often referred to their experiences during their final project (both prompted and unprompted by the interviewer), talking about how often things went wrong or did not work the way they expected, requiring many hours of troubleshooting apparatus and/or measurement techniques. We see these experiences reflected in the above example quotes, and in the results presented in Sec. VA. Revision was the most salient feature of the EMF that the students identified as being important to the process of experimental physics, as evidenced by the fact that it was the most common code for key features of the EMF (Table I) and that all the students who mentioned revision did so at the outset of the conversation. This result is aligned with Dounas-Frazer et al.'s research [44] in which students identified troubleshooting as a necessary experimental physics practice, drawing on their personal experience of running into technical difficulties and needing to engage in troubleshooting in order to complete their projects.
Of the eight students who talked about iteration, two did so immediately. These two students recognized iteration as a fundamental aspect of the nature of science: • "I like how [the EMF is] oriented in a loop that never ends. That's…very…true." • "And then you just do that over and over until we find the Theory of Everything or whatever we're going for." Five students first mentioned the idea of iteration at the comparison stage, when identifying similarities and differences between the EMF and SM diagrams. Some students simply pointed out the difference between the two diagrams: • "[The modeling framework] implicitly states that process of revision and kind of like that loop of continuous learning, whereas [the scientific method] doesn't have anything where you'd go back and…reevaluate." Other students made a value judgment and said that the iterative aspect was necessary in order to accurately depict the process of science: • "I liked the [modeling framework] a lot better because it stresses that it is cyclical. Whereas I think that the traditional scientific method…that I learned about in elementary and middle school…[is] a good way of teaching about how things should be rigorous, but I don't think it reflects that well about how science is actually done." One student who suggested that the linear nature of the SM did not accurately reflect science, justified that statement with their own experience during their final project: • "[The scientific method] makes it seem like a straight line, like, Oh yeah, you'll get it right. Or like you'll have something to report. Which definitely was not the case for us [in our project]. Like we relied on going back in loops." While most students talked about iteration immediately or at the comparison stage, there was one student who did not bring it up until, after expressing a preference for the EMF, we prompted them to consider if there was anything missing from the SM: • "I think [the scientific method is] missing some sort of loop. Like it's analyze your data and draw a conclusion, and then it's report. Whereas I feel like in actual science, you would probably link that back to either test your hypothesis or construct a new hypothesis before you report." This code for iteration overlaps with the cyclic vs linear code in Table II. There were eight students who talked about iteration at some point, but only seven who identified that as a difference between the two diagrams. The one student who talked about iteration, but not the cyclic versus linear difference, when prompted to consider if there was anything missing from the SM diagram, said, "No, I'd say pretty much the whole modeling framework is, it's at least implied right in the construct hypothesis and run the experiment and analyze your data. Like, if you're doing those three things, well, then you should be doing this iterative. Like, you come up with your models, you compare it and you do it again." Thus, for this student, the "construct hypothesis," "test hypothesis," and "analyze data and draw conclusion" boxes of the SM hold specific meaning, including the assumption that you will iterate among these three steps, even though that is not depicted directly in the diagram.
In summary, most of the students in our interview study see revision and iteration as important aspects of the process of experimental physics, with seven students talking about revision and eight students talking about iteration. While the need to revise and refine an experiment was immediately identified by students, it was more common for students to discuss the fundamental iterative nature of experimental physics when prompted to compare the EMF and the SM. Students in our study exhibited a range of ideas about iteration, some drawing on, or contrasting with, their own experiences with experimental physics, as we discuss in the following section.

D. Students' views informed by their lab experiences
The views about experimental physics that students exhibited while discussing the EMF and the SM are informed by their experiences with experimental physics in lab courses, as well as in research. The goal of this paper has not been to assess the impact of those experiences on students' views, but rather to first investigate and understand the views themselves. Aligned with prior research [44], in our study, when discussing the EMF and the SM diagrams, many students drew explicitly on their experiences with their final projects. We discuss some of those connections here as they relate to our three research questions.
The 10 students in our study had just completed final projects in an advanced lab course, and some of them had just completed (or were about to complete) their undergraduate physics degree. At least six of the students had participated, or were currently participating, in research. In the interviews, we did not ask students about their research experience, but six students spoke about their past or ongoing research when discussing and evaluating the EMF and SM diagrams. Throughout the analysis, we did not find any differences in students' views based on their research experience (existence, specific discipline or subdiscipline, or theoretical vs experimental).
In part one of the interview, when reflecting on and discussing the EMF diagram, most students said that the EMF was at least partly reflective of what they did for their final projects. Six students said that the EMF definitely described their project, three said it somewhat described their project (i.e., some parts of the EMF were familiar to them from their project, and others were not present), and one student said the EMF definitely did not reflect what they did in their project. This latter student explained that the EMF did not reflect their project because their project was confirmatory (i.e., they were trying to confirm a known result), they did not have a good theoretical understanding of the underlying physics, and they did not revise anything. This student also stated that their project was not a good reflection of how science is usually done. When asked, "Does the modeling framework reflect what you did in your project?" the student replied, "Not as much. I feel like it should have. If we did it well…it wasn't a very good question because we already knew like what the answer was going to be." Thus, this student saw the EMF as an accurate depiction of the process of experimental physics, but recognized that the project they completed for class was missing some key features of authentic scientific practice. This result is particularly aligned with research that demonstrates the disadvantages of confirmatory lab activities [63], and represents a sophisticated view on the part of this student.
In part two of the interview, when comparing the EMF and SM diagrams, six of the 10 students said the EMF was a better representation of their project than the SM. Five students explained that the EMF better reflected their project because it was more detailed and realistic than the SM, and one student said that they did not necessarily start their project with a specific research question, which they saw as being more aligned with the EMF. One student said they preferred a blend of the two diagrams because the SM better represented how they would explain their project to someone else while the EMF better reflected the actual process they went through. As scientists, this point resonated with us because when we communicate our work to others we strive to do so in a clear manner so the audience can easily follow from research questions to methods to results and overall claims; much of the messiness and nonlinearity of the actual process is not fully captured in this type of communication, a point that has been previously articulated in science education communities [11,14,64]. In writing about the myth of the scientific method, Cowles articulates this point: "In the real world, we make mistakes and get bogged down; it is only in hindsight that thinking seems clean and rational. Looking back, we tell stories about how we solve problems even as elegance evades us in the moment." [11] (p. 9).
Though most of the students in our study preferred the EMF as a representation of experimental physics and of their specific projects, they did also identify key benefits of the SM. Three students said they felt the SM diagram was a better representation of their project than the EMF, all citing the fact that the SM focused on identifying a question and doing background research before the experiment and reporting or communicating results after the experiment. Across all ten interviews, students called out these elements as being crucial to the scientific process. All three advanced lab courses included in this dataset required students to complete a proposal prior to their project and share the results in some form of summative communication (report and/or presentation). Further, these three courses all place a large emphasis on written communication as part of the final projects [56], a goal common to many lab courses [2,65]. These structures and emphases may have contributed to the salience of these features for students in the interviews.

A. Research
In this study, we investigated advanced lab students' views about the nature of experimental physics, via their views about the EMF and how it compares to a traditional linear depiction of the scientific process. When discussing and comparing the EMF and SM depictions of the process of experimental physics, students often referenced their own experiences with projects in the lab course or in their past and current research. Students' epistemological views are certainly informed by these lived experiences and are situated in a specific context. In parallel work, we investigated the extent to which students engaged in modeling during their projects and explored how different features of projects may impact students' modeling practices [66]. Future analyses will explore how these features of projects and enacted modeling practices connect to their overall views about the nature of experimental physics.
Students in this study viewed the EMF as a useful and accurate depiction of the process of experimental physics and of their own advanced lab projects. However, they also identified key aspects of the process of science that are not captured in this framework-namely, development of research questions, background research, and communication of results. An implication of these results is that when we discuss the EMF with students (or instructors or researchers), we need to clarify that the framework does not encompass all the important parts of the scientific process, including communication of results, so as not to suggest that the elements absent from the EMF are not crucial to the overall scientific process.
This study opens a window for future investigations of students' perceptions of science, particularly experimental science, at the undergraduate level.

B. Instruction
The students in this study did recognize the importance of iteration in experimental physics, but they were most likely to do so when directly comparing the EMF and SM diagrams. If iteration is a specific goal of a course, as it was for the instructors we partnered with for this research, it may be useful to engage students in this direct comparison and reflection process as part of the class. Additionally, the EMF could be used as an instructional tool to help students design and reflect on their projects, much like it has been implemented in courses in prior research [48]. The advanced lab students in our study were able to make sense of this complex framework in a short amount of time, and reflect on it in nuanced and thoughtful ways, suggesting the potential for easy implementation of a beneficial activity to support students' learning about experimental physics.
Additionally, the students in this study seemed to have holistic views about the process of experimental physics, focusing on the details of the EMF, but also signaling the importance of asking questions and reporting results in the SM. Including student-designed projects in advanced lab classes may present the opportunity to support this holistic view by teaching students disciplinary practices (e.g., writing a proposal or a journal article style paper [65]) and by giving them the opportunity to engage in the whole process in an open-ended and authentic way.

VII. CONCLUSIONS
Motivated by the potentially harmful effects of the pervasive myth of a linear and clear-cut scientific method, as well as the goal to have students engage in iteration in advanced lab projects, we conducted an investigation of advanced lab students' views about models of the process of experimental physics. In interviews with ten students from a variety of institutional contexts, we asked students to reflect on the EMF [20,21] and to compare the EMF and SM depictions of science. Overall, advanced lab students who had just completed multiweek student-designed projects viewed the EMF as being representative of the process of experimental physics. The key features that students identified encompass the entirety of the framework. That is, students recognized what we would identify as the central features of experimental physics, even when they did not necessarily experience them in their own project.
Students in the study found some similarities or areas of overlap between the EMF and the SM, but focused primarily on the differences. They described the EMF as cyclic, complex, and pertaining to the execution of an experiment, while they described the SM as linear, simple, and encompassing the whole process of science, including developing research questions and communicating results. Further, several students recognized that the linear structure of the SM can be misleading. Throughout the interviews, most of the students identified revision and/or iteration as key features of the process of experimental physics. The students who talked about revision all did so immediately upon seeing the EMF diagram. It was less common for students to bring up the idea of iteration immediately; most students who talked about iteration did so when comparing the EMF and SM diagrams side by side.
While on the whole, the students we interviewed expressed a preference for the EMF as a realistic representation of the process of experimental physics and of the projects they conducted in their advanced lab courses, they also reflected on the benefits of the SM depiction-namely, the importance of constructing research questions and communicating results. While depicting science as a linear process can be misleading, there are other benefits of such a framework that the students identified and discussed.
Advanced lab students in our study who had just completed open-ended student-designed projects were able to reflect on two different models of the processes of experimental physics in nuanced and thoughtful ways. Because of this, and the fact that most students who talked about iteration did so only after comparing the two diagrams side-by-side, we speculate that by engaging students in open-ended and authentic experimentation practices (e.g., through student-designed multiweek projects), and facilitating explicit discussions and reflections about the process of experimental physics in our lab courses, we have the opportunity to support students' epistemological development. In doing so, we can support students' development of their identity as physicists in the hopes that when engaging in experimental work, students will see iteration as a success and not a failure.

ACKNOWLEDGMENTS
We thank the instructors and students who participated in this research, Michael F. J. Fox for assistance with interrater reliability studies, Victoria Borish for helpful feedback on the manuscript, Dimitri Dounas-Frazer and Laura Ríos for initialization of ideas and data collection that preceded this work, and the CU PER group for useful conversations and feedback. This work is supported by NSF Grants No. DUE-1726045 and No. PHY-1734006. Viewpoints expressed here are those of the authors and do not reflect views of NSF. Publication of this article was funded by the University of Colorado Boulder Libraries Open Access Fund.