Qualitative investigation of students ’ views about experimental physics

This study examines students’ reasoning surrounding seemingly contradictory Likert-scale responses within five items in the Colorado Learning Attitudes About Science Survey for Experimental Physics (E-CLASS). We administered the E-CLASS with embedded open-ended prompts, which asked students to provide explanations after making a Likert-scale selection. The quantitative scores on those items showed that our sample of the 216 students enrolled in first year and beyond first year physics courses demonstrated the same trends as previous national data. A qualitative analysis of students’ open-ended responses was used to examine common reasoning patterns related to particular Likert-scale responses. When explaining responses to items regarding the role of experiments in confirming known results and also contributing to the growth of scientific knowledge, a common reasoning pattern suggested that confirming known results in a classroom experiment can help with understanding concepts. Thus, physics experiments contribute to students’ personal scientific knowledge growth, while also confirming widely known results. Many students agreed that having correct formatting and making well-reasoned conclusions are the main goal for communicating experimental results. Students who focused on sections and formatting emphasized how it enables clear and efficient communication. However, very few students discussed the link between wellreasoned conclusions and effective scientific communication. Lastly, many students argued it was possible to complete experiments without understanding equations and physics concepts. The most common justification was that they could simply follow instructions to finish the lab without understanding. The findings suggest several implications for teaching physics laboratory courses, for example, incorporating some lab activities with outcomes that are unknown to the students might have a significant impact on students’ understanding of experiments as an important approach for developing scientific knowledge.


I. INTRODUCTION AND BACKGROUND
Laboratory courses have historically been called out as an important and unique component of the undergraduate physics curriculum [1,2].Demonstrating the value of these lab courses through the use of standardized assessments of student learning is a new, but growing, area of interest for the physics education research community.However, only a few lab-focused, research-based assessments have been developed.Examples include the Concise Data Processing Assessment [3], the Physics Measurement Questionnaire [4], and the Colorado Learning Attitudes about Science Survey for Experimental Physics (E-CLASS) [5].These assessments target a range of different aspects of lab learning from students' understanding of error analysis, to students' point-and setlike conceptions of measurement, to students' ideas about the nature and process of experimental physics and its place within the discipline.This range in coverage reflects the diverse set of potential learning goals for lab courses [2,6,7].
Of the available lab assessments, the E-CLASS, in particular, has now aggregated a significant amount of data [8].The E-CLASS is a 30 item, Likert-style assessment targeting students' beliefs, attitudes, and expectations about the nature and importance of experimental physics.Each E-CLASS item presents students with a statement (see below for examples) and asks them to rate their level of agreement on a 5-point Likert scale from strongly disagree to strongly agree.Students rate their agreement both from their own perspective when doing experiments for their lab class and from the perspective of a hypothetical experimental physicist.Through roughly 4 years of data collection using a centralized, online system [8], the E-CLASS has now been administered to more than 130 distinct lab courses at 75 different institutions.
This extensive data set has previously been used to demonstrate the validity and reliability of the instrument for a broad student population [9].It has also been used to examine the impact of different types of lab activities [10], instructional approaches [11], and emphasis on skills vs conceptual development within the lab environment [12] on E-CLASS scores.These data have also been used to investigate the relationship between other factors, such as student gender and major, and E-CLASS performance [13].Additionally, this data set has also allowed us to begin exploring trends in the E-CLASS data that can shed light on various aspects of students' ideas about the nature of experimental physics, which can then be used to inform changes to the lab learning environment [14].Of specific interest are items that elicit student responses that are contradictory in some way either with the goals of the course or with their responses to other items.Recent analysis of the national E-CLASS data revealed five items for which trends in students' responses suggested that additional qualitative analysis was needed to make sense of students' reasoning [14].These five items are given below.The numbering here is designed to facilitate grouping and referring to these items, and does not reflect their actual placement within the E-CLASS survey.Q1a: The primary purpose of doing physics experiments is to confirm previously known results.Q1b: Physics experiments contribute to the growth of scientific knowledge.Q2a: When communicating the results from an experiment, my main goal is to have the correct sections and formatting.Q2b: When communicating the results from an experiment, my main goal is to make conclusions based on data using scientific reasoning.Q3: I am usually able to complete an experiment without understanding the equations and physics ideas that describe the system I am investigating.The first two items in this list (Q1a and Q1b) deal with the importance and value of physics experiments.Within the national data set (N students ¼ 7167), the majority of students [66% in first-year (FY) courses and 50% in beyond-first-year (BFY) courses [14]] responded unfavorably (i.e., inconsistent with experts, in this case agree or neutral) to the prompt stating that the primary purpose of doing physics experiments is to confirm previously known results.However, the majority of students (>90% in both FY and BFY courses [14]) responded favorably (i.e., consistent with experts, in this case agree) to the prompt stating that physics experiments contribute to the growth of scientific knowledge.Together, students' responses to these two prompts indicate that many students argue that experiments primarily confirm known results while also contributing to the growth of scientific knowledge.
The second two items in the list above (Q2a and Q2b) deal with the primary goals in the process of scientific communication.Within the national data set [14], the majority of students (79% in FY courses and 63% BFY courses) responded unfavorably (agree or neutral) to the prompt stating that, when communicating the results from an experiment, the main goal is to have the correct sections and formatting.Alternatively, the majority of students (87% in both FY and BFY courses) responded favorably (agree) to the prompt stating that making conclusions based on scientific reasoning is the main goal when communicating the results of an experiment.Together, these results suggest that many students agreed with both of these seemingly contradictory prompts.
The final item in the list above (Q3) was of interest not because of a contradiction with another item, but rather, a contradiction with one of the many possible goals of physics lab courses-reinforcing physics concepts taught in the lecture courses [15].In the national data set [14], many students (50% in both FY and BFY courses) responded unfavorably (agree) to the statement that they did not need to understand the equations and physics ideas that describe the system of interest in order to complete their lab experiments.This result is concerning given that, particularly for FY courses, development of conceptual understanding is a significant goal of the lab for many instructors.
Deeper investigation into the reasons for the trends seen in the national E-CLASS data set on these five items requires a more qualitative insight than the quantitative E-CLASS data can provide.To address this, we added open-ended prompts asking students to explain the reasoning behind their selection for each of these five items and administered the modified E-CLASS survey to five courses at two institutions.Here, we describe these courses and institutions along with the methodology used to qualitatively code students' responses to the open-ended prompts (Sec.II).We then present our emergent coding categories and findings with respect to the following three research questions: RQ1: What reasoning leads students to argue that physics experiments primarily confirm known results while also contributing to the growth of knowledge?(Sec.III B) RQ2: What reasoning is present in students' justifications that accounts for students who agree that both having the correct sections and formatting and making conclusions based on scientific reasoning are the main goal when communicating the results of an experiment?(Sec.III C) RQ 3: Why do students feel it is not necessary to understand the equations and physics ideas that describe a system when completing an experimental activity using that system?(Sec.III D).
We end with a discussion of the limitations, instructional implications, and future directions of this work (Sec.IV).

II. METHODOLOGY
To investigate the research questions presented above, we developed prompts to elicit the reasoning that motivated students' Likert-scale selection.The original E-CLASS asked about both students' views when doing physics experiments in their class and their perspectives about what an experimental physicist might say about their research.We kept the original E-CLASS questions and added prompts to the five questions presented in the previous section.The prompts were (i) Please explain your selection towards the last question for what you think when doing experiments for class, and (ii) please explain your selection towards the last question for what an experimental physicist would say about their research.An example of the Likert-scale question with follow up prompts is shown in Fig. 1.In Q1a and Q2a, both prompts were used while only the first prompt was used for the other three questions.The prompts were placed right after the Likert-scale question.
The survey, with embedded prompts, was administered to a total of 216 students, including 50 students from an introductory algebra-based physics course with labs aligned with lecture content at Rochester Institute of Technology (RIT); 64 students from an introductory calculus-based physics course with an integrated lecture and lab studio environment at RIT; 76 students from a beyond first year physics lab courses taken almost exclusively by physics, engineering physics, and astrophysics majors at University of Colorado Boulder (CU); and 26 upper-division physics majors from a stand-alone modern physics lab class at RIT.
In the algebra-based physics course at RIT, students had one lecture session and two 110-min activity sessions each week in a workshop physics environment with 6-person tables.However, most of the sessions involved problem solving or paper-pencil activities.There were only about nine hands-on lab activities for the whole semester covering topics such as kinematics, forces and motion, torque, and energy.In the lab sessions, students were asked to follow an activities manual with detailed step-by-step instructions and built-in questions.Students' grades were based on the correctness of their answers toward those questions.
The calculus-based physics course at RIT covered electricity and magnetism in a workshop physics classroom environment that met for about 6 hours per week.Most of the students were majoring in engineering or computing fields.Lab activities constituted 15% of the overall course grade with most of the grade coming from exams (70%) and homework (10%).About 15%-20% of the in-class time was spent on labs, which were clustered at a few points in the semester around the topics of electric potential, circuits with capacitors and resistors, and geometric and wave optics.The lab manual instructed students about which quantities to vary, to measure, to calculate, to plot, and sometimes emphasized uncertainty analysis.Lab notebooks and formal lab reports were not emphasized.
The beyond first year lab at CU covered topics in modern physics, including atomic physics, solid-state physics, electron diffraction, radioactivity, and quantum effects.This course was completed primarily by physics, engineering physics, and astrophysics majors in their second or third year of their program.Students met for a 1 hour lecture and 2 hours lab of each week.In lab, students completed experiments in pairs based on written instructions with each group working on a different experiment in a given class period.Grades for each experiment were based exculsively on students' lab notebooks in which they included all their data along with their formal lab writeup.Students' overall course grade was based almost entirely on their aggregate scores for each experiment with a small amount of additional credit for lecture participation.
The beyond-first-year modern physics lab course at RIT was for physics majors.The course met each week for a 1-hour prelab to develop a preliminary understanding of the experiment and apparatus and then for a 3-hour period for the main experiment.Lab notebooks were used to document work and constituted 50% of the grade.A formal lab report with one revision constituted 20% of the grade.The remaining grade was divided between participation, prelabs, and a final exam lab practice.Students completed five 1-week labs at start of the semester, and then did three  multiweek labs during the remainder of the semester.The course focused on developing proficiencies in statistical uncertainty analysis, computing for data visualization and fitting, using standard lab equipment such as oscilloscopes and multimeters.Lab topics spanned eight topics including quantum dot spectroscopy, blackbody radiation, and the photoelectric effect.The survey was administered through the Qualtrics online platform toward the end of the semester.Students took the survey outside of class within one to two weeks of receiving the survey link in their Emails.Instructors offered extra credit for students who completed the survey.
All data were then imported into NVivo qualitative data analysis software.The coding process consisted of several stages: (i) open coding, (ii) developing a coding dictionary, and (iii) coding comparison.In stage one, the primary coder (D.H.) went through all students' responses question by question and highlighted key words or phrases in their responses.Similar patterns that occurred several times in the data were then grouped together as a single code.Codes from each of the five questions were kept separately.In stage two, two coders (D. H. and B. M. Z.) took the initial emergent codes and developed a coding dictionary with code definitions and criteria.In some cases, the same code appeared in different questions and was only defined once in the coding dictionary.The primary coder then coded all student responses using the coding dictionary, and at the same time refined the coding dictionary.In stage three, a secondary coder (B.R. W.) coded 45% of the data independently with the guidance of the coding dictionary.The two coders then compared their coding.When resolving differences, they made further refinement to the codes, including combing two codes into one, removing overlapping codes, and clarifying some coding criteria to avoid ambiguity.After discussion, the two coders made changes to their own coding independently, and the percentage agreement was calculated according to Here, N both is number of references both coders coded, and N either is the number of references at least one coder coded.
The final percentage agreement between the two coders for all codes was above 90% with the exception of three codes (as the frequency of those codes was relatively low and even a slight difference on one response would affect the percentage agreement significantly), but their percentage agreement was still above 80%.The emergent codes and coding dictionary are presented in Sec.III.

A. Comparison with national trends
To determine whether our smaller data set shows trends similar to those of the national data set [14], we first look at the distribution of students' Likert-scale responses and calculate the percentage of students who had expertlike views (i.e., consistent with experts).Table I shows the percentage of students with expertlike responses in both our sample of 114 FY and 102 BFY physics students as well as the national data.In general, the results from our small sample are consistent with the national statistics across all five questions.

B. RQ1: Confirming known results versus contributing to the growth of scientific knowledge
This section focuses on our analysis with respect to the first research question (RQ1)-what reasoning leads students to argue that physics experiments primarily confirm known results while also contributing to the growth of knowledge?
We investigated student reasoning through a qualitative analysis of the explanations they provided to support their Likert-scale choice on Q1a and Q1b.All major codes emergent from the data are presented in Table II.As described below, we identified a consistent line of reasoning among students who agreed with both of these questions: students viewed experiments for class as primarily confirmation based on their classroom experience, but saw this type of confirmation activity as valuable for learning and thus contributing to the growth of personal scientific knowledge.Table III shows students' reasoning broken out by their Likert-scale choice regarding their personal views on Q1a.Table IV shows students' reasoning broken out by their Likert-scale choice regarding their personal views on Q1b.The value in each table cell represent the percentage of students with expertlike or nonexpertlike views whose explanations contained each code.For example, of the 68 students who had expertlike views on Q1a, 32% (N ¼ 22) were coded as including discovery as a justification.
For students with unfavorable responses (nonexpertlike) with respect to their personal views about confirmation of previous results, students often justified this primarily based on their own classroom experience as well as the importance of the lab activities for learning.For example, 33% of students with nonexpertlike responses argued that experiments conducted in a classroom setting were usually A small portion of the students (about 7%) argued about the use of previous results as references to check if their experiments were done correctly or not.In introductory physics laboratory courses, some of the experiments are to measure physics constants (e.g., acceleration due to gravity), and in those cases, students often compare their own experimental results with standard known values to confirm "Physics is life and we put numbers to it.""I am unsure so I put neutral." if they have done the experiments correctly.Given that all students with this type of reasoning had a nonexpertlike response, we suspect that they interpreted "confirm previously known results" as also including the use of wellestablished results to confirm the validity of current results.
For students who responded favorably (expertlike) in their personal views of confirming previously known results, the primary reason cited was that experiments are for discovery, and that discovery is as important as confirmation of previously known results.Some others argued that the classroom experiments are for a learning purpose; that is, doing experiments is to teach students scientific knowledge.
Notably, the justification that classroom experiments are for the sake of learning scientific knowledge appeared in responses from students with both expertlike and nonexpertlike personal responses with a significant amount of students in each Likert-scale group.We suspect that for students who responded unfavorably, they were making a connection between "replicating previous experiments" and "conceptual learning as the primary goal in the lab," and they saw replicating experiments as an important avenue for improving conceptual understanding in the lab.However, many students did not point out that replicating experiments could help with understanding explicitly.Example quotes are provided below.
"When preforming experiments in class we are trying to replicate results in order to understand how they are found.""In a class room setting all experiments are used to reinforce the material taught." On the other hand, students with the learning purpose code who also responded favorably emphasized that the goal of classroom experiments was for improving students' learning instead of verifying or adding support to previous results.
"I disagree because it is quite obvious that the purpose of doing experiments in class is to gain the understanding of the underlying mechanics in physics principles that we are learning." Students' justifications for their predictions about physicists' views of confirming known results can provide additional insight on students' reasoning.The primary reasons for favorable predictions (disagree) was similar to the reasoning provided in expertlike personal views, including discovery and both confirming and discovery.However, instead of learning purpose in a classroom context, students argued that professional physicists do experiments to test their own hypothesis, existing theories (uncertain or new), or any scientific ideas.
"Researchers would likely use experiments to test new hypothesis that haven't been done before.""They probably agree with me a bit less, because they may use their research to strengthen a known theory."Some others argued that experiments might not necessarily add more support to existing theories or results as they might prove those notions wrong.We suspect that students interpreted the term "confirm" in the question statement as "back up or add support to." "An experimental physicist would also probably agree that experiments can be run to disprove previous notions.But experiments can also be run to validate new theories as well."When responding to Q1b regarding physics experiments and the growth of scientific knowledge, the two most common justifications for favorable responses (expertlike) were that experiments enhance personal understanding or growth (code personal knowledge growth), and that experiments test hypotheses and theories (code testing ideas).Personal knowledge growth includes the views of experiments for improving students' scientific knowledge through multiple facets, such as educating students on fundamental science, developing students' scientific skills, and enhancing conceptual understanding.
"Physics is another broad subgenre of science.Learning physics means learning and gaining more scientific knowledge.""The labs let you see that what we are talking about in class is real so it helps reinforce core concepts." When analyzing students' reasoning across the two questions Q1a and Q1b, it is reasonable from students' point of view that experimentation in class helps with the learning of the scientific process, concepts, and skills through repeating existing well-known experiments.And because of that, experiments also contribute to the growth of their own scientific knowledge.Additionally, we found that students' justifications were not always as simple as making a straightforward distinction between confirming known results and discovering unknowns.Instead, they provided lots of context-based explanations; for example, experiments for learning purpose in a classroom setting, and experiments for testing ideas in a research lab.C. RQ2: Formatting versus making conclusions as the main goal for communicating results We now discuss our analysis toward answering the second research question RQ2-what reasoning is present in students' justifications that accounts for students who agree that both having the correct sections and formatting and making conclusions based on scientific reasoning are the main goal when communicating the results of an experiment?
As described below, the major finding was that students focus on the importance of clear communication when discussing communicating with correct sections and formatting, while they focus instead on the importance of making and justifying conclusions when discussing communicating results based on scientific reasoning.All majorcodes emergent from students' explanations are defined in Table V.The distribution of student reasoning versus their Likert-scale choice is shown in Table VI regarding their personal views on communicating with correct sections and formatting.
For students who responded unfavorably (nonexpertlike), the primary justification was the importance of formatting for fostering clear communication of results both generally and professionally.The majority of students who gave clear communication focused reasoning also specifically emphasized the benefits of correct sections and formatting for facilitating communication.
"Proper formatting and presentation are necessary to accurately and efficiently communicate one's findings.Even stellar data can get lost or misinterpreted if the formatting is off."However, other students discussed only the importance of presenting content in a more organized, efficient, and professional way without specifically explaining how formatting can help them achieve that goal.
A secondary justification for nonexpertlike response was the emphasis of correct sections and formatting toward earning a good grade in class (code grades oriented).Most of these students talked about formatting as an important factor toward lab reports, and others mentioned about how their professors or graders would use formatting as an important criteria for grading.
"When writing a lab report for a class, the formatting and inclusion of all sections is worth a larger portion of the grade than the accuracy of the data collected.The formatting therefore gets more of my attention and effort.""For giving it to my professor, it is very important that the formatting is correct compared to needing to have a correct result." Alternatively, students with favorable responses (disagree) regarding communication with correct sections and formatting almost exclusively emphasized the importance of content being presented or communicated (code content focused).These students often specifically emphasized that content or information being presented was more important than how it is organized or formatted.
"While the way information is presented (by correct sections and formatting) is important to receive the highest grade mark possible, it is more important that the information is correct and accurate." A small portion of students whose justifications were coded as content focused still responded agree or neither agree nor disagree with respect to whether formatting was the main goal.These students typically emphasized the importance of having great content but at the same time admitted that they need to be aware of the formatting as well.Here is an example from students who chose neither agree nor disagree."It's more important to have correct information and detail in your report than it is to have correct formatting, but correct formatting or organization often can have a huge impact on helping a person understand what is going on." Here again, the justifications students provided regarding their prediction about physicists' views were quite similar to those of their personal views.The primary justification for nonexpertlike predictions was also clear communication; however, the secondary justification was professional requirement instead of grades oriented.According to students, in the professional world, physicists need to have a higher standard when writing professional-related documents or communicating with other professionals.Most students discussed that publishing peer-reviewed journal articles requires strict formatting.
"Physicists have to get their paper approved by review boards.Therefore they spend a lot of time formatting their results." For students who provided uncodable responses (code uncategorized), the majority of them chose "agree."Overall, when explaining their personal views as well as physicists' views about the role of correct sections and formatting for communicating experimental results, students who agree seemed more focused on articulating why sections and formatting are important rather than reflecting on whether or not section and formatting are really the most important goal when communicating results.It is very likely that those respondents tend to bias their responses toward agreeing and then justifying why agreeing was okay.
When responding to Q2b regarding communicating results with making conclusions based on scientific reasoning, students' justifications included a number of distinct ideas.Although about 88% of students had favorable responses (expertlike), there was no one or two dominant line of reasoning from students' explanations.Students' explanations often targeted different aspects of the statement, including "make conclusions," "based on data," "using scientific reasoning," and "main goal."Some of the example quotes are provided below.
"The conclusion is a summary of the entire experiment, so if you have a conclusion then you've already completed the lab." (focusing on "making conclusions is the goal") "To make any conclusion in any field you need to have data to support your reasoning."(focusing on "evidence or data based") "I always need to make conclusions off of scientific reasoning.Bias or opinion should not play a role in drawing conclusions in science."(focusing on "role of scientific reasoning") Some other students provided related, but not quite relevant responses.About 22% of students discussed how making conclusions was about making comparison between experimental data and theoretical or previous experimental results.
Overall, when discussing communicating results with correct sections and formatting, students primarily focused on the importance of clear communication and most of them implied that correct sections and formatting would facilitate clear and efficient communication.Many of them seemed to focus on why formatting is valuable, rather than on whether or not it is the primary goal when communicating results.When discussing the relationship between communicating results and making conclusions, nearly all students had favorable responses for the importance of communicating conclusions based on scientific reasoning; however, few of the open-ended responses discussed how making well-reasoned conclusions is an essential part of scientific communication.

D. RQ3: Completing experiments without understanding equations and physics concepts
We now discuss our analysis toward answering the third research question RQ3-why do students feel it is not necessary to understand the equations and physics ideas that describe a system when completing an experimental activity using that system?
In general, students who responded unfavorably (agree or neither) to this prompt argued that following procedures is enough and this is consistent with their classroom experience.Justifications for favorable responses (disagree) are varied, but the three main ones are that concepts and equations guide experiments (code concepts guide experiments), analysis requires understanding, and quality work requires understanding.Students with unfavorable responses (agree or neither) most often argued that following procedures is enough, and some used their classroom experience to backup their reasoning.Most of them argued that they were often provided with specific step-by-step lab instructions to guide them through the lab, and they felt it was possible to complete a lab without understanding the equation and concepts that describe the system.
"The experiments we do in class have step by step instructions so I don't have to understand the physics, I just have to follow instructions.""Given the great detail in which the procedures for inclass experiments are explained, it's quite possible that a 3rd grader could complete the majority of the experiments done in class without actually understanding anything that's happening in front of them." Alternatively, the primary justification for favorable responses (disagree) was that understanding of concepts and equations would help guide experiments and make students actively engaged in experimental activities.Many students also discussed that understanding concepts can help them make sense of what they are doing instead of blindly following instructions, thus making it easier to complete an experiment.
"I think that understanding the equations will make the lab click all together.It shows a purpose for understanding what the ideas and equations mean.""I think it's really hard to finish an experiment without knowing the motives behind it, because then you have no idea what you're doing." Other less common justifications for favorable selections included analysis requires understanding and quality work requires understanding, as well as understanding is the goal of experiment.
In summary, the majority of students believed that understanding equations and related concepts was necessary for sense making during a lab activity, conducting data analysis, and performing quality work.However, there was still a significant amount of students (about 40%) that said that simply following the procedure was enough for completing lab activities.Although this finding is somewhat surprising and concerning given that conceptual understanding is one of the main goals in physics laboratories courses, these results are likely driven by the cookbook nature of our current lab curriculum.This suggests that our current lab curriculum needs to be reevaluated and modified to make sure that we meet our goals.

IV. CONCLUSIONS
We explored seemingly contradictory trends identified in students' responses to five questions on the E-CLASS assessment.To better understand the source of these trends, we implemented a modified version of the survey, which included open-ended prompts asking students to explain the reasoning behind their selections for these five items.The quantitative scores on these items suggest that our small sample demonstrated the same trends that were observed previously in the national data.A qualitative analysis of students' justifications revealed their reasons for either favorable or unfavorable selections.
For students who argued that physics experiments primarily confirm known results while also arguing that they contributed to the growth of scientific knowledge, we identified a consistent line of reasoning-that classroom experiments primarily confirm known results in order to help with understanding of physics concepts, and thus, they contribute to students' personal knowledge growth about science.Alternatively, students who did not agree that physics experiments primarily confirm known results most often focused on discovering new ideas or phenomena as an equally, if not more, important component of physics experiments.
For lab instructors, this finding suggests that incorporating some lab activities for which the outcome is not known to either the students or instructor might have a significant impact on students' understanding of the importance of experimental physics as a mechanism for uncovering new physics and driving the creation of new theoretical models.Labs that focus on analyzing an unknown sample (e.g., the spring constant of a rubber band chosen at random from an assortment of different size TABLE VIII.Student views about Q3 "I am usually able to complete an experiment without understanding the equations and physics ideas that describe the system I am investigating."The percent of students with expertlike or nonexpertlike Likert-scale selections whose reasoning contained particular codes.Students' reasoning may be coded under several codes when appropriate, so percents in each column may not add up to 100%.N is the total number of students with expertlike or nonexpertlike views and the number of students with each code is given in parentheses.Codes with total frequency less than 10 (around 5% of the total population) are not shown.rubber bands), or investigating a curious everyday phenomenon, may be ways to foster a sense of inquiry and discovery.Also, labs that follow a modeling progression of first observing a phenomena and then building explanatory models emphasize experiments as a source of insights about the physical world, rather than a confirmation of things known previously from lecture.

Codes
Students who argued that having the correct sections and formatting and making conclusions based on scientific reasoning are both the main goal when communicating results of an experiment tended to focus on different aspects of the prompt in their responses to the two items.For example, students often focused on the importance of clear and efficient communication when asked about sections and formatting, while later focusing on the importance of drawing or validating conclusions instead of just communicating them when asked about making conclusions based on scientific reasoning.Some students also argued that correct sections and formatting were what they were primarily graded on and, thus, were most important to them as students.Alternatively, students who did not agree that sections and formatting were the main goal when communicating did so primarily based on the idea that the scientific content was equally or more important than the formatting.
This finding suggests that lab instructors should ensure that their grading structure explicitly rewards the aspects of scientific communication they value and want students to focus on.Sections and formatting, which are easily visible surface features of scientific communication, are not a substitute for the weightier aspects of a good scientific argument that takes substantial effort to communicate, interpret, and grade.
Finally, for students who argued why it is not necessary to understand equations and physics ideas that describe the system in order to complete the experiment, the most common justification was that they can simply follow the instructions from the lab manual and instructors so that understanding was often unnecessary.However, these same students often acknowledged in their justifications that this was an artifact of classroom experiments and not reflective of authentic experimental physics.Students who argued that understanding equations and physics ideas was necessary most often focused on the importance of this understanding when performing the analysis of experimental data or as a guide to performing and setting up experiments.
For lab instructors, this finding supports a common critique of highly procedural "cookbook" labs that do not require or provide opportunities for students to reflect on the physics foundations of the experimental set up and data collection process.According to the AAPT Recommendations for the Undergraduate Lab Curriculum, developing and testing theoretical models of an experiment is one of the primary goals of a lab course.The act of examining an experimental setup, observing behavior, identifying relevant assumptions and underlying principles, and developing a mathematical model is a critical component of the physics lab experience for all students at all levels.Lab guides that provide a highly detailed theoretical introduction with ready-made derivations of the key equations are good candidates for revision to foster more sense making around how the mathematics connects to the design and execution of the experiment.In lieu of a full modeling cycle in the lab [16], faculty could consider requiring students to quantitatively justify why particular principles are valid to apply (e.g., conservation of energy) or to list assumptions and experimentally justify their validity in the theoretical model as a way to expose more of the ideas underlying the equations.
There are several important limitations to this study.While the trends in students' E-CLASS responses that motivated this study came from national data spanning many courses and institutions, the qualitative nature of the work reported here necessitated a much smaller scale data collection involving only two institutions.Though the quantitative trends in this small sample were consistent with the national trends, the trends in the qualitative analysis may not be representative of the broader student population.Additionally, this study investigated only interesting trends in a small subset of E-CLASS questions.Future work investigating other assessment items using this method of including open-ended prompts embedded in the original survey could serve to lend additional qualitative detail that can help instructors make sense of their students' responses.Moreover, advances in machine learning and automated analysis of short text responses offer a potential avenue for making this approach scalable for use with additional questions and a larger student population.

FIG. 1 .
FIG. 1.An example of one of the five E-CLASS prompts alongwith the open-ended bids for students to explain their Likert-scale choice.

TABLE I .
Percent of students with expertlike views in our data (N ¼ 216) and national data (N ¼ 7167).
"For students enrolled in the class, the experiments are done based on previous proven theories.There isn't a lot of room for new discovery in the concepts taught in class.""The experiments we do have already been performed by many previous classes and professionals."

TABLE II .
Code definitions and example quotes regarding students' responses to Q1a "The primary purpose of doing physics experiments is to confirm previously known results" and Q1b "Physics experiments contribute to the growth of scientific knowledge." "Experiments broaden and enhance students knowledge and interest in physics which is beneficial to the scientific community and its future.""The labs we do in class help us tremendously in class to fully understand the topic "The point of the labs in class is for students to learn what has already been discovered.""Most experiments done in the classroom test long known phenomena, so they do not contribute to overall knowledge of the scientific community."Uncategorized When a student response cannot be coded with any of the above codes.It includes a simple repeat of their selection, and ambiguous or unrelated responses.

TABLE III .
Student views about Q1a "The primary purpose of doing physics experiments is to confirm previously known results."The percent of students with expertlike or nonexpertlike Likert-scale selections whose reasoning contained particular codes.Students' reasoning may be coded under several codes when appropriate so percents in each column may not add up to 100%.N is the total number of students with expertlike or nonexpertlike views and the number of students with each code is given in parentheses.Codes with total frequency less than 10 (around 5% of the total population) are not shown.

TABLE V .
Code definitions and example quotes regarding students' responses to Q2a "When communicating the results from an experiment, my main goal is to have the correct sections and formatting" and Q2b "When communicating the results from an experiment, my main goal is to make conclusions based on data using scientific reasoning." Table VII shows the major codes emergent from qualitative data.Table VIII shows students' reasoning vs their Likert-scale selections.

TABLE VII .
Code definitions and example quotes regarding students' responses to Q3 "I am usually able to complete an experiment without understanding the equations and physics ideas that describe the system I am investigating."