Model-Based Reasoning in the Upper-Division Physics Laboratory: Framework and Initial Results

We review and extend existing frameworks on modeling to develop a new framework that describes model-based reasoning in upper-division physics labs. Constructing and using models are core scientific practices that have gained significant attention within K-12 and higher education. Although modeling is a broadly applicable process, within physics education, it has been preferentially applied to the iterative development of broadly applicable principles (e.g., Newton's laws of motion in introductory mechanics). A significant feature of the new framework is that measurement tools (in addition to the physical system being studied) are subjected to the process of modeling. Think-aloud interviews were used to refine the framework and demonstrate its utility by documenting examples of model-based reasoning in the laboratory. When applied to the think-aloud interviews, the framework captures and differentiates students' model-based reasoning and helps identify areas of future research. The interviews showed how students productively applied similar facets of modeling to the physical system and measurement tools: construction, prediction, interpretation of data, identification of model limitations, and revision. Finally, we document students' challenges in explicitly articulating assumptions when constructing models of experimental systems and further challenges in model construction due to students' insufficient prior conceptual understanding. A modeling perspective reframes many of the seemingly arbitrary technical details of measurement tools and apparatus as an opportunity for authentic and engaging scientific sense-making.


I. INTRODUCTION
Reference to a target system or phenomenon modeling, we mean this dynamic process of constructing and using models. It is our goal here to present a framework for modeling that changes the static picture of Fig. 1 into a dynamic process that is descriptive of students' model-based reasoning in the laboratory.
Although this discussion of modeling has entirely focused on models that have external representations, we acknowledge students' mental models also play an important role in learning science. The Framework for K-12 Science Education makes the distinction that "mental models are internal, personal, idiosyncratic, incomplete, unstable, and essentially functional." It is the permanence of external representations that enables models to be effective tools for communication and scientific explanation, and it is these external representations that can be gradually be improved through the work of a community researchers separated in space and time.

III. FRAMEWORK FOR MODELING IN LABS
A. Review of prior frameworks One of the earlier schematics for the modeling process is a linear 4-step scheme developed by Hestenes and Halloun that includes: model description, formulation, ramification, and validation. 7 A later version also used by Hestenes and Halloun includes an iterative process of prediction, analysis, and validation. 10 Both frameworks are targeted at developing models of physical systems discussed in standard lecture courses (e.g., developing models of mechan-ical systems in motion). Windschitl et al. created a highly iterative framework where they reframe the entire task of inquiry-based science as centered around modeling, something they call Model-Based Inquiry. 11 The Model-Based Inquiry framework is perhaps the most comprehensive look at a broad range of scientific practices (e.g., asking questions, building models, generating hypothesis, constructing arguments, and seeking evidence) within a framework that emphasizes modeling as the primary tool for scientific explanation. Another framework from Schwarz et al. is directed at learning progressions and emphasizes the interconnectedness between metamodeling knowledge and the elements of modeling (e.g., construction, prediction, evaluation, revision), but doesn't describe how these elements form a process. The Framework for K-12 Science Education lays out a very general schematic for how modeling fits among several other scientific practices important to scientists and engineers. 4 Finally, the prior framework that most tightly couples experiments with modeling is the learning cycle utilized throughout the introductory-level Investigative Science Learning Environment (ISLE) curriculum. The ISLE cycle progresses through making observations, identifying patterns, creating explanations, articulating assumptions, making predictions, testing predictions, and revising models and experiments. 12

B. The need for a upper-division lab modeling framework
The broad challenge that inspired our new framework was the desire to describe modeling in a way that could be readily applied to upper-division physics lab courses for analyzing student reasoning or design of curricula. The earlier frameworks were focused on general K-12 science and introductory college physics. While there are many extremely helpful insights in each of the frameworks cited above, none of them has upper-division physics labs or experimental physics research in mind. In the language of modeling, upper-division physics labs were outside of their domain of applicability. Some frameworks operate at too general a level to really link to the details of the lab; some do not fully capture the iterative nature of the lab; and some do not present the connections between the elements of modeling.
The frameworks that accompany Modeling Instruction 7,10 are some of the most detailed and relevant to physics, yet they have a singular focus on the development of models of physical systems, and give little attention to the experimental tools that must be used to make the measurements.
In order to demonstrate why the new framework includes measurement tools while prior frameworks were able to largely ignore them, it is worth contrasting the apparatus and upper-division labs involve advanced concepts and use equipment that is often unfamiliar, students must do significant intellectual work in order to understand the experiment.
The holistic framework for modeling in the laboratory in Fig. 2 divides the full experimental apparatus into the physical system (right side of Fig. 2) and the measurement tools (left side of Fig. 2). (Italics are used throughout the paper to highlight the connections to key stages in the framework in Fig. 2 and in the modeling codes used in Figs. 5 and 6 and Table I.) This conceptual division between measurement tools and physical system is useful because it recognizes that models used to understand the physical system (e.g., a block sliding down a ramp) are quite different from those used to understand the measurement tools (e.g., a motion detector). Sometimes the division is obvious in an experiment, while other times, the measurement tools and physical system are more integrated. In such cases, while there might be multiple ways to conceptually divide the system into a physical system and measurement tools, any reasonable division is still a helpful for modeling the full apparatus. The left-right symmetry of the framework emphasizes that the measurement tools, in addition to the physical system, can be understood through mechanisms rooted in principles of physics.
To begin working through the modeling diagram, we will start at the model construction predictions (a stage represented in other modeling frameworks), but the measurement tools model is used to interpret raw data or translate raw observations into physically meaningful quantity (e.g., a motion detector converts a time delay between sent and received pulses into a position). The predictions and interpreted measurements converge at a comparison, and the results of that comparison are either satisfactory, or prompt a revision in the models or apparatus. The framework identifies four major pathways for revision: refine the measurement tools apparatus, refine the measurement tools model (e.g., calibrate the measurement tools), refine the physical system apparatus, or refine the physical system model. Depending on likely sources of discrepancies (e.g., limitations were identified in the model construction phase, or a parameter was unknown in the apparatus) and upon which revisions are most Model easy to implement, the various pathways can be prioritized and explored. The process is represented as a cycle that can be repeated until the experimental goal is met.

D. Application of the framework to lab design
In order to briefly demonstrate the general utility and detail included in the modeling framework in Fig. 2, we first apply it to analyze the structure of an introductory lab activity and then apply it as a curriculum design tool that can guide the creation of activities that emphasize different aspects of modeling. For a more in-depth discussion of the framework applied to upper-division laboratory design, see Ref. 18, where the framework was used to develop a lab activity on the polarization of light.
To demonstrate the application of the framework in the analysis of modeling in an activity, consider a standard lab activity where students construct a simple pendulum of length L and measure it's period T for small angle oscillations to determine the gravitational acceleration g using a relationship previously derived in class (T = 2π L/g). Applying the framework to this hypothetical activity, we identify the physical system as the pendulum, while the measurement tool is a stop watch. The model construction stage for the physical system was mostly accomplished prior to the lab, perhaps by the instructor or textbook. The relevant principles and simplifications were already applied in the derivation of the period.
The only missing element in the model are two parameters: L, the pendulum length, which may be chosen by the student, and g, which is treated as an unknown model parameter. The measurement tools produce raw data, which are the total time elapsed on the stop watch ∆t and the number of oscillations N . The measurement tools model is very simple and has a mathematical representation as T = ∆t/N . The use of the measurement tools model gives the interpreted data, which is the period per oscillation. Viewed as a modeling exercise, this "measurement of g" lab is an exercise where Newton's laws of motion are known, the gravitational force has been mathematically modeled, but we imagine that within this theory of the gravitational force, there is an unknown parameter g, which must be found. There is a small degree of model refinement in the activity, in the sense that the previously "unknown" parameter g is optimally chosen to produce the best agreement between the predicted period and the measured period. Although this activity has some elements of modeling, it only explores a limited subset of the modeling process.
As a curriculum design tool, the framework offers a way to generate various alternative pendulum labs that emphasize additional aspects of the modeling process that go beyond

IV. THINK-ALOUD LAB ACTIVITIES
In addition to using the framework for curriculum design, we also use the framework to measure students' engagement in different aspects of the modeling process in the laboratory.
Because one new and significant component of our framework was the emphasis on modeling the measurement tools, we wanted to find examples of how students engage in a modeling process with their measurement tools during laboratory activities (Sec. V A). Further, we were curious if students found particular aspects of the modeling process to be especially challenging (Secs. V B and V C).
In order to capture students' experimental work involving modeling, we designed a short laboratory activity that would provide students with ample opportunities to design, use, test, and refine models of the physical system and measurement tools. The think-aloud activity was based on an instructional lab that students had previously completed in a junior-level electronics course. Students were given an LED connected to a DC power supply and asked to measure the LED's optical power output and compare that to a value they predict using information from a product data sheet. The short prompt for the activity is shown in Fig.   3, and Fig. 4 shows the lab bench and equipment that students were able to use during the activity. The experiment was designed to provide students opportunities to construct and refine models of the measurement tools and the LED with varying assumptions and levels of sophistication.
You have been hired by an LED manufacturer to run quality control tests of the optical power output of their new high efficiency red LEDs. Answer all the questions on the graph paper provided: 1. What is the predicted total power emitted by the LED based on the manufacturer's specification?
2. What is your measured total power emitted by the LED?
3. How do the measured and predicted powers compare?
4. What assumptions and approximations were included in your prediction or measurement?
5. What could you do to improve the comparison?
FIG. 3. Written prompts used for the think-aloud experimental activity.
The physical system consisted of a red LED connected to a DC power supply in series with a resistor to limit the maximum current through the LED. A multimeter was used to measure current through the LED. The measurement tools used to determine the optical power output were a Thorlabs PDA36A Adjustable Gain Photodetector and an oscilloscope.
Students were also provided with data sheets for the LED, photodetector, multimeter, and oscilloscope along with assorted optical mounts.
The student participants were drawn from an upper-division optics and modern physics lab course at a large public research university. The course has a typical enrollment of 20-25 students. All students enrolled in the course were invited to participate in the think-aloud interview, and eight students responded. All eight participants were male, a result of the fact that 20 of 21 students enrolled in the course were male during the semester of the data collection. All students were juniors or seniors majoring in physics.
The interviews were conducted during the final two weeks of the semester, by which time the lab space and equipment shown in Fig. 4 (other than the LED) were part of their weekly lab experience. The think-aloud lab portion of the interview typically took 30-45 minutes.
After the activity, there was a 20-30 minute follow-up discussion where the students' models were reviewed in order to clarify missing details from the think-aloud portion. At the very end, the participants were allowed to request explanations for any of the physical or mathematical details of the experiment.
Audio and video of the think-aloud interviews were collected along with written obser- vations and copies of students' work. The full interviews were transcribed, but only the think-aloud laboratory portion was coded for instances of model-related activity identified in the framework. Table I provides a list of high-level modeling codes and a brief description of the coding criteria. Several of the codes (model construction, predictions, comparison, limitations, revision) are common to most frameworks that discuss modeling. Several of the high-level codes had more refined sub-codes (e.g., revision had sub-codes to indicate whether apparatus or models were revised), but these are not described in Table I or shown in the data. Several other modeling codes in Table I are unique to the framework: physical system, measurement tools, interpretation of data, and making measurements. Finally, some significant moments of student reasoning were placed into categories that were outside the framework. Troubleshooting was coded whenever a student recognized a problem with the apparatus and attempted to solve it. Sometimes the resolution was simple and unexpected (pressing the "autoset" button on the oscilloscope), while other times students engaged in what appeared to be a rapid modeling cycle involving a series of qualitative predictions and qualitative measurements (e.g., if I put my hand in front of the detector, the photodetector output voltage should decrease) in order to identify the source of the problem. In these model-based troubleshooting episodes, the timespan for a series of predictions, measurements, comparison, and refinement could occur in the span of a minute or less. Although not all troubleshooting activity utilized model-based reasoning, it was remarkable to see how the process of identifying and solving the problem became a genuine episode of scientific inquiry activity where the solution was unknown to the student and yet of the utmost importance to find.
The three interviews that are included in this analysis (see Figs. 5, 6) were independently coded by two of us and then compared for consistency. As the coding criteria were refined, the two coders were able to come to an agreement level exceeding 95% for the high-level modeling codes presented.

V. RESULTS
There are three main results in the sections that follow. The first, and most important result, is that students do engage in meaningful modeling of the measurement tools. Students' modeling is significant in both duration and in quality as Figs. 5 and 6 and several interview excerpts will demonstrate. The second and third results are about common challenges that students' had during the modeling process. The second result reviews challenges in identifying assumptions during model construction and in justifying those assumptions and connecting to limitations. The third result describes challenges students had during model construction because of their insufficient understanding of key concepts. Although it probably comes as no surprise, the third result does highlight the need for lecture and lab to be linked because prior conceptual understanding can support a richer experimental experience just as doing an experiment can provide additional conceptual understanding.
A. Students' modeling of measurement tools Fig. 5 shows the occurrences of physical system (LED) and measurement tools (photodetector and oscilloscope) codes throughout the interview. As Fig. 5 shows, each of the students spent a significant fraction of their time using and modeling the measurement tools in this particular upper-division lab activity. Based on Fig. 5 alone, it would be impossible to understand the character or quality of the measurement tools activity because it may just be that the students were mindlessly turning knobs and writing down measurements.
We demonstrate the quality in two ways. First, Fig. 6 shows the occurrences of specific modeling codes for Student A over the entire think-aloud activity. The making measurements code does specifically code the activity of turning knobs and making observations, but Fig. 6 shows this only occupies about 4 minutes of the activity, while a broad range   Excerpt: Model construction-Principles and concepts. One element of model construction is identifying the key principles and concepts in the model. In this excerpt, Student A is trying to understand the physical mechanism by which the incident light on the detector is converted to the output voltage V being measured by the oscilloscope. The student realizes that the intensity of the broadly dispersed light from the LED must be integrated over some area to get a total incident power P in . The student correctly identifies that the power from the absorbed light produces a current in the photodiode (according to a material-dependent proportionality constant R, known as the responsivity). The current is converted to a voltage using a circuit known as a transimpedance amplifier with gain G in volts per amp. Mathematically, this model is represented as V = GRP in . The student goes on to successfully construct a quantitative model connecting a measured voltage to an incident power. However, by first identifying the principles of operation, the mathematical equation that is used is not just a computational tool, but has been clearly linked to the physical setup. ...The difference is 23.2 mV from the diode being on to off. Now, we're gonna turn the voltage output into total power emitted by the LED." Both the quantitative and qualitative data support the idea that modeling frameworks applicable to upper-division physics labs must include modeling of the measurement tools.
The ratio of students' effort devoted to modeling the physical system versus measurement tools will almost certainly vary depending on the particular laboratory activity. The particular balance shown in Figs. 5 and 6 is due in part to the intentional use of a common measurement tool (the photodetector), which requires modeling as a natural part of gathering and analyzing data. However, most upper-division laboratories include a wide variety of measurement tools and techniques for which this modeling framework will be well-suited.

B. Students' challenges in model construction when articulating and assumptions
Although articulating assumptions is a common element of modeling in nearly all descriptions of model construction, we found several instances during the think-alound interviews where students would utilize a model, but not recognize the assumptions that supported that model. This difficulty is significant because it may hinder other aspects of the modeling process. First, an unidentified assumption is not going to be justified, and so will be included without any critical evaluation as to its appropriateness. Second, the assumptions will not be connected to limitations of the model. Finally, those unidentified limitations are unlikely to inspire any iterative refinements to the experiment.
A common occurrence in the think-aloud activity was that students predicted the optical power using P = IV where V is the voltage drop across the LED and I is the current through the LED. Students identified the relevant parameters in the LED data sheet and computed a numerical result for the power P = IV . For example, Student B said, "I need to predict that total power of the LED. And so the power is the current times the voltage.
And the forward voltage drop is like 2 volts . . .well, I'll try and put 20 milliamps in it because that's what it told me to do... which means my power should be 2 volts times .02 is 0.04 watts. So predicted power is 0.04 Watts, so good stuff." The student used a particular principle from electronic circuits (P = IV ), and identified relevant parameters in the device, yet did not recognize that the model assumes 100% of the electrical energy used would be converted into light. Because the assumption was not identified, no attempt was made to justify its appropriateness, and a modification of the assumption was never an option for model revision. Toward the end of the interview, after the think-aloud portion was complete, the student was directly asked "So you calculated [power] based on the voltage drop across the diode, the current running through the diode. So, what assumption were you making about the optical power output?" The student replied, "I guess that the optical power output would be the same as the power used in the circuit like whatever power was in the circuit was all emitted light, which isn't necessarily the case I guess." In this case, with a direct question, the student did reflect upon the LED model and recognize the assumption and that it didn't have any justification. A general pattern was that when a mathematical representation for the model could be readily identified (e.g., P = IV ), then an explicit discussion of the assumptions was bypassed.
A second example was in the construction of the measurement model involving the conversion of optical power P into photodetector output voltage V . A mathematical relationship was provided in the photodetector data sheet: V = GRP in , where the responsivity R depends on the material (in this case silicon) and the wavelength of the light. However, the LED emitted a spectrum of wavelengths with a spectral width of about 45 nm. When listing the various assumptions that were made as part of the prediction or measurement, none of the students listed the assumption that the light was monochromatic, although many other assumptions were listed. When specifically asked if there were any assumptions about the spectral properties of light, all students immediately responded that they assumed the wavelength of the LED was a single wavelength at the peak of the spectrum shown on the data sheet. One of the students went on to justify this assumption: "I would expect that half of the wavelengths [in the spectrum] would give less responsivity and half would give more because we are kind of in this spot where [the responsivity] is almost linear but it does vary... So half would be less [and] half would be more responsivity which would lead to hopefully the same calculated [value as when assuming a monochromatic source]." In this case, the student was able to provide a justification, but only when the assumption was brought to the student's attention.

C. Students' challenges in model construction from insufficient conceptual understanding
In addition to recognizing assumptions, there is a certain amount of prior knowledge that is needed for the construction of a model. In the think-aloud activity, the most accurate model for predicting the optical power output of the LED required the use of an angular distribution of power in microwatts per steradian. Because LEDs are designed to have a particular emission pattern for the light output depending on their application, the data sheet provided a polar coordinate plot of the relative intensity as a function of angle. Also, the data sheet specified the numerical value of the maximum power output per unit solid angle (in microwatts per steradian) and the approximate angular width of the emission pattern. We anticipated students would be able to estimate the emitted power by roughly determining the solid angle of the emitted cone of light and multiplying that solid angle by the peak angular intensity.
The following brief explanatory comment about solid angle was also provided on the data sheet: "A steradian is a measure of solid angle, which is the area of the angular region on a unit sphere. A full sphere subtends 4π steradians." Despite the explanatory comment and the fact that a similar activity in a prerequisite lab course used solid angle, most students were unsure about the meaning of steradians as a unit. Within the interviews, the related concept of intensity as power per unit area was used by the students. However, the conceptual modification of angular intensity as power per solid angle produced significant confusion in the think-aloud activity.
After Student A completed the entire think-aloud activity, the interviewer mentioned the student had omitted any reference to the use of angular intensity in microwatts/steradian on the data sheet. When asked "What do you make of this angular intensity spec?" Student A replied that, "I guess I kind of glossed over that." Student A's response could have indicated a simple oversight, but only after a 10 minute back-and-forth discussion was the student able to get a rough estimate of the solid angle of the LED emission pattern and use it to produce a new prediction for the optical output power. This indicates that the student's oversight was likely connected to a lack of familiarity with the concept of solid angle and the unit of steradian.
Other students also had difficulty with the concept of solid angle. In the process of making predictions, Student B said "I am calculating the power IV for the diode according to the data sheet. I'm not sure what to do with the angular intensity. I don't know the units of microwatts per steradian." Student C, when first reading through the provided data sheet, expressed "What is that kind of graph? Relative intensity...I have no idea what that is." Each of these students ended up utilizing a P = IV model for optical power output because it was the most obvious model that avoided the concepts of solid angle and steradians.
Further consultation with undergraduate instructors confirmed that the one-week lab done in the earlier electronics lab course was probably the only time in their core curriculum where the concept of solid angle was directly addressed.
When a lab activity utilizes concepts that are largely outside of students' prior knowledge, it has a significant impact on how they engage in the laboratory. In this case, it caused most students to largely ignore key information in the data sheet, and construct significantly less accurate models of the optical output power (the P = IV model produced a prediction about 500 times larger than using angular intensity in the model). Beyond model construction and predictions, the angular emission pattern played an additional role in designing the experiment, as it provides one justification for whether or not the photodetector could capture the full emission pattern of the LED.
Because meaningful modeling and experimental design depends on sufficient prior conceptual knowledge, it may be worth revisiting the connection between lecture and lab in the upper-division curriculum. At the upper-division level, many physics lab courses are composed of an assortment stand-alone activities that cover a wide range of advanced undergraduate physics topics. If modeling and experimental design are to be integrated into such lab classes, then the appropriate level of prior knowledge should be considered for each individual activity. Also, greater integration of the upper-division lecture and lab courses could produce richer experiences in both modeling and experimentation.

VI. CONCLUSIONS
Reviewing the prior literature on modeling suggests most earlier modeling frameworks emphasize modeling physical systems that closely relate to core physics ideas (e.g., mechanics or electricity and magnetism). In these frameworks, much of the experimental apparatus is largely overlooked in the modeling process in the pursuit of key principles and concepts. However, in upper-division physics laboratory courses, there is often a non-trivial relationship between the phenomena being studied and the raw measurements, which require students to have an understanding of the design and operation of the full experimental apparatus.
We presented a new framework for modeling that treats the full experimental apparatus in two parts: the measurement tools and the physical system, and both parts are subject to a modeling process as a way to understand the design and operation of the experiment.
The framework serves as both a descriptive tool for characterizing students' model-based reasoning and has applications as a curriculum design tool. Use of the framework in several think-aloud activities showed that the identification of assumptions was a common difficulty for students, especially when a pre-constructed mathematical model was available. Further, the interviews provided a striking example of the link between a lack of prior conceptual knowledge and an inability to construct models.
Future studies will look at the model construction process, in particular the links between assumptions, limitations, and model revision. Also, we are looking at the relationship between the accuracy and sophistication of students' models of measurement tools and how they connect their own measurements to the key concepts in the lab activity. We hope the framework provides useful insights for those physics education researchers and instructors who are looking for ways to describe laboratory sense-making and for faculty who want to integrate more conceptual and mathematical understanding into experimental activities.

VII. ACKNOWLEDGMENTS
The authors would like to thank the CU Physics Department for input on learning goals and pointing out the importance of modeling measurement tools. This work is supported by NSF TUES DUE-1043028 and NSF TUES DUE-1323101. The views expressed in this paper do not necessarily reflect those of the National Science Foundation.