Examining problem solving in physics-intensive Ph . D . research

Problem-solving strategies learned by physics undergraduates should prepare them for real-world contexts as they transition from students to professionals. Yet, graduate students in physics-intensive research face problems that go beyond problem sets they experienced as undergraduates and are solved by different strategies than are typically learned in undergraduate coursework. This paper expands the notion of problem solving by characterizing the breadth of problems and problem-solving processes carried out by graduate students in physics-intensive research. We conducted semi-structured interviews with ten graduate students to determine the routine, difficult, and important problems they engage in and problem-solving strategies they found useful in their research. A qualitative typological analysis resulted in the creation of a three-dimensional framework: context, activity, and feature (that made the problem challenging). Problem contexts extended beyond theory and mathematics to include interactions with lab equipment, data, software, and people. Important and difficult contexts blended social and technical skills. Routine problem activities were typically well defined (e.g., troubleshooting), while difficult and important ones were more open ended and had multiple solution paths (e.g., evaluating options). In addition to broadening our understanding of problems faced by graduate students, our findings explore problem-solving strategies (e.g., breaking down problems, evaluating options, using test cases or approximations) and characteristics of successful problem solvers (e.g., initiative, persistence, and motivation). Our research provides evidence of the influence that problems students are exposed to have on the strategies they use and learn. Using this evidence, we have developed a preliminary framework for exploring problems from the solver’s perspective. This framework will be examined and refined in future work. Understanding problems graduate students face and the strategies they use has implications for improving how we approach problem solving in undergraduate physics and physics education research.


I. INTRODUCTION
Problem solving receives a significant emphasis across the undergraduate physics curriculum, and teaching students to develop their problem-solving skills has been an emphasis of physics education research (PER) for several decades [1,2].Research on students' development of problem-solving skills, transfer of problem solving-abilities to new contexts, and the effectiveness of particular pedagogical approaches for teaching problem solving has provided a foundation for physics curriculum development and direction for PER [1].
Within PER, most research on problem solving has focused on problems posed in introductory-level physics [3,4].Students taking introductory physics classes diverge into a range of careers in STEM and beyond.While the physics content may be of varying relevance to future careers, it is hoped that problem-solving abilities are transferable to the problems these students will face later in their careers and lives.As such, problem solving is included on lists of 21st-century skills that all citizens need [5].
While PER most often focuses on problem-solving strategies appropriate for introductory-level physics and supports student learning in these courses, students who participate in physics research may be ill prepared for the problems they will face.Although a significant amount of research exists on how experts (i.e., faculty and graduate students) solve introductory-level problems, there is limited research on the kinds of problems that experts solve as part of their professional scientific work [1].Some research does exist on workplace problems faced by engineers [6] and more complex physics problems [7], yet these may not reflect the problems faced in physics research and how graduate students solve these.
We attempt to expand our understanding of problem solving in PER by providing a description of problems faced by Ph.D. students as part of physics-intensive Ph.D. research and an emergent framework for classifying problem types.We then contextualize problem-solving strategies the Ph.D. students use to solve routine, difficult, and important problems related to their research and examine their perspectives for what makes a good problem solver.Examining problems and strategies for solving problems has implications for the undergraduate physics curriculum and how we prepare students for future problem solving.To do this, we explore the following research questions in this paper: (1) What problems do graduate students face in their research, (2) What strategies do graduate students use to solve problems related to their research, and (3) In what ways are these strategies used to solve routine, difficult, and important problems they face?
A. Problems: Defining real-world scientific problems from the solver's perspective Most textbooks are full of worked example problems and back-of-the-chapter problems, and a "problem set" is a commonly used synonym for homework.Though these problems fit general consensus about the abstract definition of a problem-the difference between the current state (e.g., knowledge and resources) and a goal state (e.g., a desired outcome), actual concrete problems come in an enormous variety of forms [6,8].While some PER studies define a problem as "a task which requires one to devise a sequence of actions leading from some initial situation to some specified goal" [2,9], problems are not always so simple.Some problems, such as Sudoku puzzles, have a well-defined form, known solution strategies, and a single correct solution.Researchers have recognized the limitations of this definition and sought to "change their concept of a 'problem' from the bare, abstract questions posed in textbooks to more open-ended questions based on realworld situations [2]."Real world problems such as minimizing anthropogenic climate change have an ill-defined end goal (e.g., which climate parameters should be focused on and over what time scale), constraints (e.g., economic or technological feasibility), and a large number of possible solution paths (e.g., solar energy, energy efficiency improvements, carbon sequestration).In order to characterize the breadth of problems, we drew inspiration from Jonassen's framework that classifies problems according to learning activity, inputs, success criteria, context, structuredness, and abstractness [10].This framework enabled us to research a multitude of problems, including real-world problems with open-ended, multiple, and unknown solution paths.
Design fields such as engineering tend to view problems broadly.Problems and problem solving are woven throughout the design process, with varying perspectives on what constitutes a problem and the distinction (if any) between problem solving and design.Eris defines problem solving as a narrow activity with respect to situations (problems) with given requirements and static final solutions [11].On the other hand, he defines design as a richer activity focused on a dynamic situation (problem) with negotiated or constructed requirements and perpetually evolving solutions.Kolko and Buchanan go beyond well-defined and ill-defined (openended) problems to define wicked problems as involving extreme complexity, interdisciplinary design and large social issues [12,13].In information science, Dervin generally defines a problem as a barrier to forward progress, concern, angst, or an information or other gap [14].Problem solving is via a user-based sense-making approach involving individual construction of information needs and uses to understand problems and bridge gaps, considered central to all communicating situations.These studies show the broad spectrum of complexity that problems can span.In our study, we formatted interviews to reflect this by addressing routine, difficult, and important problems.
In psychology, Hayes defines a problem as the gap between where we are now and where we want to be, without knowing how to cross the gap [15].Hayes identifies ill-defined problems as ones requiring problem solvers to contribute to the definition of the problem from their own knowledge, values (gap-filling decisions), or quest for new information.He defines a special class of ill-defined problems for which what we contribute to the problem definition is that there is a problem [15].Many people observe and dismiss a situation as an annoyance until one person recognizes it as an important problem to solve.
In this study, we also broaden the definition of problem to focus on the perspective of the problem solver.To do this, we expand on the description of problem solving used in the research of Adams and Wieman [7].For Adams and Wieman, "the designation of a task as a problem must be based on the solver's response to the task, rather than the task itself.If a task does not qualify as a problem for a particular individual, then it becomes merely an exercise for them" (p.459).As Adams and Wieman explain, a task only becomes a problem when the problem solver perceives it as one.In a review of problem solving in PER, Hsu and colleagues note that in 1962 Polya defined problem solving similarly as 'the search "for some action appropriate to attain a clearly conceived, but not immediately attainable, aim" and adds that "where there is no difficulty, there is no problem"' [2].This makes problems highly situated in the context of the solver.In our study, we apply this definition by letting the interviewee determine what counts as a problem.

B. Problem solving: Developing expertise via representations, strategies, and reflections
While it is important to identify and describe problems, it is equally valuable to explore the ways in which problems are solved and how people learn to become good problem solvers.Key foci for prior research on problem-solving strategies include studies on the differences between expert and novice approaches, the use of representations, identifying strategies, and the integration of reflection and metacognitive approaches when solving problems [1].

Expert versus novice problem solving
Expert-novice research in PER suggests that there are distinct differences in the ways that undergraduate students and faculty solve physics problems.Existing PER focuses on why experts solve problems more successfully than novices including differences in both knowledge structures and procedures.Knowledge of the underlying concepts is one essential prerequisite for experts to solve problems.As Larkin and colleagues explain, "The expert is not merely an unindexed compendium of facts, however.Instead, large numbers of patterns serve as an index to guide the expert in a fraction of a second to relevant parts of the knowledge store" [16] (p.1342).They find that this capacity to reference relevant information quickly is what we think about as intuition [16].
Some studies in PER find that problem-solving procedures are reversed for experts and novices where, "experts used a 'knowledge-development' approach, starting from the given information and working forwards to find the desired quantity, while novices used a 'means-end' approach, starting from the desired quantity and working backwards to connect it to the given information" [2] (p.1151).Yet, Priest and Lindsay challenged this commonly reported difference in direction of problem solving.Instead, they suggested that experts plan solutions better without relying on equations.
Research in other fields provides insights into necessary problem-solving skills and means of developing expertise.Information behavior researchers Brand-Gruwel, Wopereis, and Vermetten found that experts took longer to complete problem-solving tasks (all available time), performed many subskills more often (reading the task, activating prior knowledge, elaborating content, judging content, formulating problem) and were better at writing style and layout.Eighty percent of experts referenced sources.Novices did not reference sources or reformulate the problem [17].In engineering education and design research, Adams and Atman revealed behavioral and performance differences between freshman and senior students, suggesting that iteration is a significant and essential design activity.Study measures of iterative activities correlated with design success, with engineering seniors exhibiting more iteration, more effective iteration, and more evidence of metacognition than freshmen [18,19].
While comparing expert and novice problem solving provides valuable insight for teaching and learning physics, some PER suggests that the expert-novice division itself can become blurred.When experts were faced with difficult problems, they approached them with more novicelike processes [2].Alternatively, students who were identified as "good" problem solvers showed similar problem-solving characteristics to experts, while other students who were considered "bad" problem solvers exhibited more novicelike characteristics [20].In fact, when faced with contextrich multifaceted problems in a semester-long course, students' problem solving changed to resemble that of experts [3].This change was even seen during a single contrived experimental problem-solving interview [8].During the problem-solving interview, experts formed multiple hypotheses and tested each, revised their ideas and developed new hypotheses, and relied on multiple representations.While students differed from these approaches at the beginning of the problem-solving activity, their methods more closely resembled the experts after working on the problem for awhile [8].There are links between the problem-solving context and methods that are used and learned [3].In particular, this research suggests that open-ended experimental problems might encourage expertlike problem solving.

The use of representations and models
Representations are an integral part of solving physics problems.As Chi, Feltovich, and Glaser explain, representations are a constructed semantic network connecting components.They correspond with initial state (givens), desired goal, and operators.Representations can also contain other elements, inferences, and abstractions [21].Most importantly, they suggest that quality of representation influences problem solvers' abilities."Experts tended to categorize problems into types that are defined by the major physics principles that will be used in solution, whereas novices tend to categorize them into types as defined by the entities contained in the problem statement" [21].Here, experts integrated qualitative analysis into their problem solving prior to working with appropriate equations, allowing them to categorize problems based on conceptual principles.Other studies propose that this integration and blending of informal conceptual and formal mathematical reasoning should occur throughout problem solving to be more expertlike [22].
Representations are also used to describe problems.Ton de Jong and Ferguson-Hessler found that, "both good and poor novice students demonstrated that they possess models of problem situations…both groups gave reconstructions of problem descriptions at a level that is comparable to the expert level" [23].They propose that successful students use models based on physical concepts rather than simply surface features from the problem description that poor problem solvers rely upon.Additionally, students who were good problem solvers were better able to reconstruct important situational information of the question and solution [23].Using representations for incorporating situational information and physical concepts is an important task in the problem-solving process.

Problem-solving strategies and skills
PER has identified several important strategies and skills for solving problems [7,8].Adams and Wieman identified over 40 subskills for solving complex physics problems in different contexts.These included skills related to physics and math-specific knowledge, beliefs and motivation, and processes [7].Etkina and colleagues visualized sequences of problem-solving steps using "transition graphs" to identify patterns and compare groups.They found that expert problem-solving strategies supported the ISLE framework and included steps such as observation, patterns, hypotheses, experimental design, prediction, experiment performing, judgment, assumptions, representations, and others.Moreover, their research process enabled them to identify patterns in how each of these steps were used by different groups [8].They found that "expert scientists solved the light cone problem by forming several hypotheses for the observations they conducted and by subsequently testing those hypotheses…They not only designed and performed new testing experiments, but also checked the consistency between the new hypotheses and the outcomes of previous testing experiments" [8].Understanding the specific steps and how they were used to solve problems provides a basis for visualizing the process and developing learning situations that foster building expertise.
Examples of curricula also provide explicit expert problem-solving strategies such as the use of multiple representations, a stepwise process (e.g., isolate the unknown, substitute, evaluate) [24], or provide worked examples that follow a standard process (e.g., identify, set up, execute, evaluate) [25]; however, most PER has restricted attention to problem-solving strategies that students use in their coursework rather than problem-solving strategies that experts use in real scientific contexts.

Reflection and metacognitive approaches
Reflection and metacognitive approaches have also been found to guide problem solving.These are often less tangible than specific strategies; yet PER has found them to be valuable for solving physics problems, especially ill-defined and open-ended ones similar to those faced in real scientific contexts.For these more open-ended problems, "extensive practice and study is a necessary but not sufficient condition for becoming an expert" [20] (p.147).Students who are good problem solvers use self-explanations, which are guided by accurate monitoring of their own understanding and misunderstanding, while poorer students rely heavily on examples [20].In problem solving, self-explanations also lead to self-instruction and motivation to persist.For example, such metacognitive strategies are crucial for enabling students to get "unstuck" and to catch mistakes.
In a skills analysis of problem solving, Brand-Gruwel and colleagues determined that regulation activities take place continuously while solving problems in information science [17].Problem solvers must track and evaluate their iterative process, monitor or steer performance, orient on task, manage time, test content and quality, and evaluate product.This research suggests that stressing the repeated asking and answering of questions that support selfreflection and metacognition may help students learn regulation, as can cognitive apprenticeship through a combination of modeling, coaching, and practice on the how, when, and why of strategies.Additionally, education researchers Leahy and Sweller investigated metacognition and imagination in problem solving by comparing the effectiveness of learning problem-solving procedures using open-ended study approaches versus imagining the procedures step by step.They found that under certain conditions, instructing learners to imagine procedures facilitated learning more than studying those procedures did, resulting in the imagination effect and acting as a form of deliberate practice [26].Though it is uncommon to explicitly teach these strategies in physics, reflection and other metacognitive approaches can have substantial advantages to problem solving.
In this study, we examine a multitude of problem-solving strategies used by graduate students.Graduate student problem solving provides a bridge between experts and novices as well as between the existing body of research on novice and expert problem solving and problem solving in professional contexts.We also had participants discuss problem-solving strategies for tasks that they felt were problems, rather than contriving our own problem-solving situations.Based on Cancula, Planinsic, and Etkina's findings, the act of doing open-ended experimental research may be encouraging the transition from novice to expert problem solving [8], so we felt it was crucial to ascertain the problemsolving approaches used in their actual research.Like the research of Chi and colleagues as well as Adams and Wieman, our strategies include metacognitive approaches in addition to more standard strategies [7].Our framework also integrates the problem and contexts graduate students faced since, as Ogilvie suggests, these are intrinsically related to the strategies problem solvers employ [3].

II. METHODS
We used a qualitative methodological approach to investigate problem solving for physics graduate students.
Interview data collection and analysis procedures are explained in the following sections.

A. Data collection
Applying an ethnographic interview framework, we conducted semi-structured interviews with ten Ph.D. students at a private university about their experiences with problem solving as part of their research [27].The study population included participants from their first to sixth year of graduate study.Six were males, and four were females.Although all students attended the same university for graduate school, each came from different undergraduate institutions and were enrolled in a variety of graduate programs at the time of this study.Students' research specializations spanned a range of physics-intensive projects in astrophysics, astronomy, optics, microfabrication, and materials science, and included a mix of theoretical and experimental methods.
In 30-45 min semistructured interviews, students were asked to describe their overall research project then prompted to discuss any routine, difficult, and important problems they faced.They were asked questions such as, "What are some of the most important problems you have encountered during your time as a grad student?" and additional follow-up questions related to specific examples and solutions.This allowed routine, difficult, and important problems to be defined by the solver.For each of these problems, students provided examples of how they solved them, which we refer to as strategies.Students were also asked to consider characteristics of good problem solvers in graduate school.

B. Analysis
Interviews were transcribed and analyzed using NVivo Qualitative Data Analysis software.To code the interviews, we used a typological analysis framework where the goal is the development of distinct categories within a phenomenon.The categories should be important in all cases, not hierarchical in relationship to one another and important to the entire phenomenon [28].This approach directs researchers to use a systematic process of identifying an organizing framework for typology development, reducing the data, creating thematic categories, and drawing conclusions [29,30].
In a first pass coding, we identified specific problems or attributes of problems that were discussed in the interview as either a routine, difficult, or important problem.A problem was operationally defined as any description or attributes of a task that the solver found challenging.All interviews were coded by at least two researchers and interrater reliability was calculated using NVIVO software to determine codes that were especially inconsistent (less than 85% agreement).Through discussions between coders, each code and definition was refined, removed, or merged until reaching complete agreement.Within the 10 interviews, 230 instances of problems faced by graduate students were identified.These passages were then assigned an initial short descriptive open code (e.g., data analysis) that described some salient feature of the problem.The initial codes were collapsed into a minimum number of well-defined codes through consensus discussions of the research team; however, this list still included over 20 codes with no particular organization.We used a typological analysis framework [29,30] to reduce the data into three categories: context (nouns that describe where the problem was located, see Table I), activities (verbs that describe an active aspect of the problem, such as designing, see Table II), and features (adjectives that describe specific attributes that made the problem challenging, such as complexity, see Table III).This three dimensional framework in Fig. 1 of context, activities, and features provides a

Models
Simulating an experiment or theory visually to understand or make predictions.Software Downloading and using software that is often unfamiliar and field specific.Collaboration Working with others (e.g., researchers, advisor, group members, etc.) to accomplish common goals.

Plans
Establishing steps for research, setting project goals, and determining research direction.Shared results Communicating findings with the broader scientific community through publication, presentation, or conversation.preliminary problem framework that can inform teaching and further research in physics education.

Solver-defined problem
To examine problem-solving strategies, we used the interview transcripts described above to identify approximately 370 instances where problem solving was used, which we operationally defined as anything the students did themselves in response to the problems they identified in their research.We also coded these as relating to routine, difficult, or important problems depending on where in the interview they were discussed.Two coders were used across all interviews to ensure interrater reliability.In a second iteration of coding, we labeled examples using an emergent coding strategy and collapsed codes into broader categories.For the second pass, two coders jointly negotiated all categories, codes, and definitions.Graduate students' reflections on what makes a good problem solver and their suggestions for improving preparation in problem solving were also used to further explain findings and provide implications for undergraduate curriculum development and PER.

III. FINDINGS
From the problems coded, each was categorized into three dimensions of a framework: contexts (Sec.III A), activities (Sec.III B), and features of problems faced by graduate students (Sec.III C).Developing a tentative framework allowed us to expand on a simple list of problem attributes to understand what are the problems student faced as well as how and why are they problematic for the solver.Specifically, context explores when and where problems take place, activities describes what the problem solver is attempting to do, and features describe why this specific situation was problematic.
From the problem-solving strategies coded, we distilled our analysis into findings focusing on three emergent themes: students rely on multiple cognitive approaches (Sec.III D), inter and intrapersonal characteristics are helpful for approaching problems (Sec.III E), and moving beyond routine problem solving requires broadening strategies to connect to a social context (Sec.III F).Across these findings, our study provides an in-depth look into commonly reported strategies in their original context from the perspective of graduate level researchers.
In addition, because the interview protocol separately addressed routine, difficult, and important problems, we further refine results to contrast routine, difficult, and important problems.Routine problems included any issues that the participants encountered on a daily basis in their research.Difficult problems were those that the interviewee stated were among the most challenging tasks faced in their research.Important problems included those necessary to complete their thesis research as well as problems they considered to be important to the wider scientific community.We present findings in each section by theme broadly as well as by problem type to compare between routine, difficult, and important problems as they relate to each theme.

New problem
Problem is discovered by researcher and has never been solved by anyone in the field.

Self-directed
Problem is independently directed, defined, and solved.Unknown solution Knowledge of the process or outcome of a problem is ill defined, uncertain, or unavailable.

Conflicting goals
Clashing expectations and disagreements between group members make a solution method complicated.

Complex
Problem is multifaceted and draws from multiple fields of knowledge.Often requires iterative solution methods.

Impactful
Problem is relevant and important to the scientific community and its solution is valued by the real world.
A. Problem contexts: Prevalence of experimental, computational, and social contexts Physical principles are most compactly and accurately expressed with mathematics.Perhaps because of this, most undergraduate-level problems are also mathematical in nature, and there is substantial research on the role of mathematics in physics problem solving [31].However, when asked to discuss problems faced in their research, graduate students often described problems outside of mathematics or modeling (see Table I).Specifically, the contexts of experiment, programs, software, and data analysis were all very prevalent (Fig. 2).The contexts were part of tasks such as setting up equipment, testing devices, collecting and analyzing data, and programming, which are prevalent in research settings.The problem contexts of math and modeling, while frequently occurring in research, represented only a fraction of the breadth of problems encountered in research.
Again, focusing on Fig. 2, we can see that important problems often involve more social contexts of research, such as collaboration, planning, and sharing results (with the broader scientific community).In fact, the most common important problems students faced related to sharing their results with the broader scientific community.One participant said the following, regarding working with a group to publish results: It's really difficult to get everybody to be in the same mindset and everybody to agree in the end that, yes you're finished, and yes what you have is good data and we don't need to do anymore with it, the paper is finally ready to be in publishable form.So I think getting the data ready and then getting all of the people you worked with to agree, has been a struggle.
This example highlights that some of the most important problems in Ph.D. research have an intrinsically social component.As discussed in more depth in Sec.III E, for these social contexts, students discussed the relevance of strategies such as teamwork, communication, and project management, as fundamental to the solution.

B. Problem activities: Important problems
are open ended The structure of most undergraduate level problems involves designing (constructing) some mathematical model or mathematical derivation as part of the solution [32].However, Table II shows several other activities were common in graduate-level research.These included troubleshooting, interpreting results, evaluating options, and assessing quality.Designing, however, was the most commonly reported difficult problem students faced (see Fig. 3).In our coding scheme, designing included designing a mathematical model, and designing in other contexts (e.g., designing a program or designing an experiment).
The activities associated with the most important problems included interpreting results, assessing the quality of those results, and the evaluating options.Each of these activities is usually open ended, involves considering multiple possible solution paths, and has a solution that is unknown to the wider scientific community.

C. Problem features: Important problems with impact
Problem features were characteristics of a problem that made it challenging, and could be coupled with any combination of context and activity.Table III defines the various features, while Fig. 4 shows the occurrences of those features within the interviews for routine, difficult, and important problems.For the graduate students in our study, the most important problems tended to be impactful.
Solving the problem would matter to society or at least to some members of the scientific community.When discussing how graduate research could effect the scientific community, one participant explained the following.
I think the most difficult problem that I've encountered there is with trying to convince people that what's going on is significant…So I have to convince them that this is something that matters, that's going to affect their measurements, that maybe has already affected their measurements, and can cause them to draw false conclusions, or cause them to throw out data that they could otherwise use.
The need to make a difference in the research community, and have their research matter, seemed to drive the sentibehind these types of important problems.Again as noted before, this problem feature has a clear social dimension and is not merely challenging for its complexity or ill-defined structure.

D. Problem-driven strategies: Using diverse cognitive strategies
Table IV summarizes a list of all strategy-related codes.The table is divided into two sections, general strategies and problem-driven strategies.All coding passes were integrated into NVivo Qualitative Data Analysis Software to link codes with source data for context and to perform frequency counts.
Within Ph.D.-level research, planning ahead was frequently identified and included strategies like considering expected outcomes in order to determine how to approach a problem.For difficult problems, it was also important for students to be flexible in the approaches they tried and to pursue alternatives as needed.This was most important for solving problems with multiple possible methods or solutions.Additionally, graduate students applied problemdriven strategies such as breaking the problem down into smaller steps.This allowed students to simplify the  In this case, breaking the problem down was a necessary strategy since each smaller problem required different tools and approaches.While such planning and organizational steps to problem solving are discussed at the introductory level, the development of the plan for an unfamiliar problem type or the subdivision of a complex problem are less emphasized.
Evaluating the quality of results was also identified as an important strategy (see Fig. 5), since it is the step whereby the solver deems when a solution is "good enough."In this example, the student compared graduate research to an undergraduate lab course.
You need to do this titration… and make sure that you're confident in your results.Whereas in your lab report you can be like, 'Human error.Maybe something happened.Maybe it wasn't the right amount.'But in lab, if you tell your boss that maybe it wasn't the right amount he's going to be like… In graduate research, students need to be able to defend the quality of their results whereas in undergraduate labs, they just need to report that something happened or likely happened to decrease the quality.Again, contrasting with undergraduate-level problem solving, the WISE strategy includes an evaluate step where students check the sign, magnitude, and units of a result [33], but it does not push toward defending aspects of a solution to experts.
Pursuing alternatives and identifying the source of a problem was another cognitive strategy.One student said, Usually I'll do something like where I don't put my sample in there, but I'll try to check if it's either that something is wrong with the machine or something is wrong with the way I'm using the machine.So I try to figure out which one of those things it is.Or I'll usually just kind of go back through all of the things I would do to get the machine running and make sure that I didn't miss a step, something like that.This cognitive step of pursuing alternatives and identifying sources of problems is only possible when problems have diverse solution paths, which is not typical of many introductory-level problems.
Additionally, the use of test cases was useful for routine problems involving troubleshooting program code or experiments.In regards to a troubleshooting situation with lab equipment, one student said, There's no answers in the back of the book anymore.So it's you oftentimes need to go make up a number of dummy test cases and make sure my algorithms or my logic makes sense.So, try to find the simplest minimal working example…I know if I give it this input, here's the output I should get.
Students rely on test cases to solve problems since they have known solutions that can be built upon to slowly introduce new ideas that do not necessarily have known solutions.Such cognitive strategies, and others discussed in our findings, emphasize that scientific research is open ended with multiple solution pathways.

E. General problem-solving strategies: Relying on inter and intrapersonal skills
Since graduate level research is rarely done in isolation, interpersonal skills are critical.In fact, some problems had more to do with working with others than with technical issues.In the following example, a student reflected on learning to ask questions as a strategy for solving problems.So that's one thing, it's learning how to get other students to pass on the knowledge to you…demanding, 'I want to learn this thing and you know this thing.Please teach me this thing.'And learning to ask preemptively for help before they leave and everyone's stuck.
In this example, it was important for the student to reach out to others and ask for instruction in order to solve problems.Asking questions is a valuable interpersonal skill referring to knowing when and how to ask questions to obtain necessary information [34].
While students solved some problems by asking others for help, they also recognized the need to take initiative and learn independently.As one student describes, Like lots of times, if I had a question about why something isn't working, and there was another grad student that I knew would be able to answer the question, but they weren't available, I would kind of still poke around with it, even if it took me a few hours.Versus I know other people who would just like go home for the day and wait for that person to come in the next day.
This student noted that persistence was not a characteristic of all graduate students, but was useful for moving research forward.As another student pointed out, "I guess grad school is more independent.It's really your research, your work, and you have to motivate yourself to move it forward."Motivation and initiative are driven by a sense of research ownership.
Intrapersonal strategies were also necessary for solving problems lacking known solutions.As one student attests, If you encounter a problem…and you don't know how to solve it immediately that shouldn't scare you.You should just calm down and take the appropriate steps to learn how to solve the problem.I feel like the textbook doesn't teach you that, the textbook says, 'Read this problem you have 30 seconds to answer it, or you're going to run out of time to do the rest of the test.'You don't have time to actually think and problem solve, in that sense.
This highlights a need for characteristics like confidence, grit, and optimism to solve problems.
In addition to students using general strategies to solve their own problems, they also reflected on what characteristics make someone a good problem solver.Figure 7 shows the characteristics students found valuable in problem solvers, which are similarly reflected in the intrapersonal skills students relied on themselves (See Fig. 6)."Initiative" was the most commonly reported characteristic and this code aligns with the strategy "take initiative" that students used in solving their own problems.Additionally "flexibility" was viewed as a valuable characteristic that also reflects the skill "pursuing alternatives" that was frequently helpful for solving important problems.Though often considered personality traits, students identified important intrapersonal skills both as strategies they used and characteristics of good problem solvers.

F. Solving difficult or important problems: Moving beyond routine problem-solving strategies
For solving routine problems, most graduate students searched resources broadly, often using Google.Other useful strategies included test cases and interpreting results, both allowing the student to independently apply strategies within a specific problem.Yet as problems became more difficult or more important, students needed to rely on strategies that better connected them to other researchers and their field.Students solved difficult problems most often by developing a broad view, getting help from others, or pursuing alternatives (often by looking at what other researchers have tried).Figure 6 shows that students often start with an online search (search resources) for solving routine problems.Online searching via Google worked well for frequently encountered problems that needed one quick solution.Though as problems became complex, searching became restricted to a specific science domain.One student needed a place to start solving a problem and narrow down various options, The first thing I did or would do is look online to see if anyone else has solved it already.I would look in the literature and try to see if there's any relevant papers that have come out in the last 30 years or so.Then if there are, then I'd take that down to maybe the last 15, the last 5, and see what the most recent thing might be,  For more difficult problems, graduate students often sought help from peers or others in close proximity.Even if these sources did not have exact solutions, they were often able to share ideas that moved research forward.The social nature of research also posed problems for students, for instance, positioning research findings in the field and ensuring potential impact.Students needed strategies for assessing the quality of their findings and positioning findings in a way that others in the field would find valuable.One student explained that taking logic courses in philosophy helped develop strategies for assessing quality and justifying findings to others.
It [taking logic courses] really just gave a way to make definite conclusions and to make conclusions in a way where people couldn't deny them.Especially that's important when you're talking to other people about your research, and when you're writing papers, when you're trying to make sure that everything that you do is entirely defensible.
Philosophy courses are not typically part of science majors requirements, but in this case provided strategies to solve more difficult problems tied to the social nature of science.

IV. CONCLUSIONS
In this study, the individual interviews with graduate student researchers provided a window into physics problem solving that is grounded in evidence of students' reasoning and behaviors and in a context that aligns with professional practices of the physics community.Prior research has relied on an expert's view [7] or experimental context in a laboratory classroom [8], yet this study integrates both the professional context and detailed student perspectives.Graduate student researchers also provide a unique perspective at the intersection between novice and expert as they are generally advanced compared to undergraduates, but still strongly benefit from the mentoring and advice from their faculty advisor and research group.For our findings, we highlight some key ways they align and extend existing research and also provide reflections on implications for the undergraduate physics curriculum.
First, because we examined problems from the perspective of the graduate student researcher, we identified a wide range of problem contexts even though these were encountered by a small number of participants.The diversity of contexts overlaps with lists of cognitive processes for complex problems [7], frameworks for engaging in scientific investigations in a laboratory environment [8], and scientific practices in the Next Generation Science Standards [35].What is most notable about the problem context from a curricular and PER perspective is that so much problem solving takes place in situations where mathematics and modeling are not the central focus.Rather, research frequently involves hands-on laboratory work, measurements, programming, data analysis, and using specialized software.Because laboratory courses provide substantial opportunities to incorporate a breadth of problem-solving experiences, there is a significant opportunity to focus more physics education research on problem solving in these lab courses and research environments.Our findings also suggest that efforts to include more computation across the physics curriculum are well suited to preparation for research, but the physics community should also consider "data across the curriculum" as a means to incorporate data and analytical techniques in theory courses.PER can make significant contributions to understanding the role of computer-based tools and data within the problem-solving process in multiple contexts.
Second, problem activities within research are diverse.While many undergraduate-level mathematical exercises involve designing (or constructing) a mathematical model, it is less common to provide undergraduates with opportunities to design apparatus or computer code.Without providing such opportunities, students become users, but not producers of technology.Beyond design problems, several other activities were highlighted: troubleshooting, interpreting findings, assessing quality, or weighing multiple options.In future studies, it would be valuable to conduct indepth think-aloud interviews assessing how students solve particular problem activities, such as troubleshooting situations or when deciding among multiple options.Within the curriculum, instructors could provide problems that require students to articulate a set of plausible options, evaluate them, and make a well-reasoned choice.Similarly, students should gain experience with problems where the interpretation of data or a theoretical prediction is emphasized-where gaining new physical insight takes priority over simple checks of accuracy (e.g., consistency of units).
Third, problems are not merely cognitive but also social in nature.Our findings show that the most common difficult and important problems involve the student as part of a wider community.When working closely with collaborators, the challenge is developing a shared set of goals and responsibilities.When interfacing with the wider community, the challenge is convincing this audience that work is significant.Efforts at collaborative group problem solving fosters an ability to develop a shared set of goals and responsibilities [36].However, convincing an audience of the significance of your work requires a deep knowledge of what your audience understands and values.Perhaps undergraduate research is the most natural venue to develop such knowledge and PER could investigate students' views about the significance of their undergraduate research to a wider community.It is critical to realize that emphasizing these social interactions within problem solving introduces students to a broader set of important scientific problems and serves a bigger purpose than improving content learning of a particular topic.
Fourth, the identification of a diverse set of problemsolving strategies highlights that when problem types become more diverse (as in research), then a wider range of strategies are employed.Problem-solving frameworks in PER could reflect a more diverse range of strategies, which in our study included general strategies (e.g., searching resources and planning ahead), but also problem-driven strategies (e.g., applying test cases, estimation).Many open questions remain in problem solving research, such as how experts and novices employ these strategies differently, or what role planning and representations play in solving different types of problems.Regarding implications for undergraduate curriculum, instructors should be conscious of the strategies their students employ to complete activities in or out of class and could refine problems to require more sophisticated strategies.For instance, design problems foster an iterative process of planning, building, testing, evaluating, and refining.
Fifth, graduate students' reliance on interpersonal strategies emphasizes the need for experiences in team problem solving and group work as well as additional opportunities for explicitly learning interpersonal communication approaches.Creating a collaborative culture within physics departments and within classrooms should support building communication skills and in making better problem solvers.Intrapersonal skills (e.g., motivation, persistence, confidence) sustain the problem solver in the face of adversity.While measures of intrapersonal ability exist (e.g., Duckworth's GRIT scale) [37] and link to research on metacognition and self-regulation, how to teach these abilities as part of problem solving is a largely open question [20].Finally, our research highlights the need to move beyond routine problems that have known solutions and strategies.When confronted with a routine problem (e.g., a bug in their computer code), graduate students do not "reinvent the wheel" as an intellectually stimulating exercise, but regularly use online searching to find resources that may quickly and efficiently solve their problem.However, we note that when undergraduate students turn to online resources to quickly and efficiently solve their homework problems it is typically regarded as cheating.For the researcher, the reward is a solved problem that enables them to proceed in their research.For an undergraduate, the reward is points for completing and/or getting the correct answer, which defeats many of the pedagogical aims of homework.Perhaps the temptation to cheat could be reduced and the pedagogical benefits enhanced by reducing the use of routine problems with known solutions and strategies.Instead, instructors could use more problems with multiple plausible solution paths, social implications, and unique contextual features which are rarely easily solved through online resources.By challenging our students with more open-ended real-world problems, we can reduce reliance on cheating and provide better preparations for problem solving.
Our research has implications for enhancing and structuring the way we think about problem solving in physics education.Our preliminary framework (Fig. 1) provides a simplified structure for identifying problems from the perspective of the solver in a wide range of physics-relevant situations.Moreover, our research indicates that the problems we provide our students will strongly influence the strategies they use and learn.Therefore, we need to carefully understand the contexts, activities, and features that make situations into physics problems.We hope to explore, apply, and refine this framework in future work.These findings are only the first steps in an expanded set of problem-solving research that explores the role of representations, expert-novice differences, and explicit teaching of problem-solving strategies across the contexts, activities, and features identified in this study.

FIG. 5 .
FIG. 5. Problem-driven strategies used to solve routine, difficult, and important problems in Ph.D.-level research.

TABLE I .
Problem context.

TABLE II .
Problem activities.something (e.g., circuit, output signal, program, hardware or software interface, etc.) to produce an artifact that meets a goal.Troubleshooting Debugging, finding errors, fixing equipment, and correcting mistakes.Interpreting Results Understanding and making sense of results to find physical meaning in mathematics and understand the mechanisms that explain the findings.Evaluating Options Finding the best solution, model, or approach to optimize results and maximize efficiency.Assessing Quality Determining that results are good enough to move on, convince others, or publish.

TABLE III .
Problem features.

TABLE IV .
Problem-solving strategy codebook.Italicized headings refer to grouped categories of several codes.Whenever I have a problem, now my technique is I just want to break the problem into small steps, and then go step by step…For example, I am working on a partial differential equation.First of all, I look up what kind of equation it is…I try that tool.Now I have another equation, so what I can do?It's an eigenvalue problem.Now I can use another tool…So knowing that techniquebreaking the problem in steps-that's really helpful.
General strategies used to solve routine, difficult, and important problems in Ph.D.-level research.andtry to use what the science has gone forward in time and see if I can get something that's the most recent.In my research group there are, like now, around eight to ten people.And we are working on different group projects… We have a weekly meeting and we discuss those problems.If somebody comes up with a solution, [then] that actually helps the group.Going to peer is a certain thing that we do.