Student experiences with authentic research in a remote, introductory course-based undergraduate research experience in physics

[This paper is part of the Focused Collection on Instructional labs: Improving traditions and new directions.] Participation in undergraduate research experiences (UREs) has been identified as an important way of increasing undergraduate retention, interest, and identity within the sciences. Course-based undergraduate research experiences (CUREs) have been shown to have similar outcomes to UREs but can reach a larger number of students at one time and are accessible to any student simply through enrollment in a course. One key component of a CURE is that students must participate in authentic scientific discovery in which they answer a question where the answer is initially unknown to both students and the scientific community. Here, we present student experiences with authentic research in a large, introductory physics CURE conducted remotely during the COVID-19 pandemic. We use student responses to a closed ended survey question, as well as written responses to an open-ended end-of-course assignment to investigate what aspects of real research students felt that they participated in and the extent to which students felt that they participated in authentic research. Most students in the course felt like they engaged in real-world research during the course and a large number of students highlighted their experience with authentic research when asked to describe their experience in the course more broadly. We discuss which elements of the course may have contributed to the students ’ experiences of authentic research.


I. INTRODUCTION
Students receive many personal and professional benefits from participating in undergraduate research experiences (UREs). UREs have been shown to boost students' persistence in science, technology, engineering, and mathematics (STEM) fields [1], help students develop valuable research skills, better prepare students to enter graduate school [2], and help students develop their identity as scientists [1]. Additionally, students report higher levels of confidence after participating in an URE [1]. Although these characteristics of UREs can benefit all students, they have been shown to be especially valuable for students from historically marginalized backgrounds [3].
While UREs show many benefits for students, there are also barriers to participation in UREs. Students may not be aware of research opportunities and their benefits or they may not have the time and financial resources required to participate in research. Furthermore, faculty may overlook students with lower grade point averages when choosing undergraduate researchers to mentor or may have an unconscious social bias (including gender and racial bias) that causes inequity in the hiring process [4]. There are additional barriers present for students from historically marginalized racial backgrounds, including cultural barriers and lack of representation in science [5].
Course-based undergraduate research experiences (CUREs) are a proposed alternative to traditional UREs that have the potential to remove many of the barriers to participating in traditional UREs because they take place in a formal course, so students receive course credit and can enroll in the research experience simply by enrolling in a course [6]. Remote CUREs, specifically, may also improve access to undergraduate research for nontraditional students, as well as students with disabilities who may have limited access to a traditional lab space [7]. Similar to UREs, a CURE allows students to participate in authentic research and work to answer a novel research question that is of genuine interest to the scientific community [8].
While CUREs can be an effective solution to the access barriers of UREs, they are still not widespread among all STEM disciplines and levels. CUREs are most common in chemistry and biology courses [9,10], and typically occur in upper-division courses, where students already have some subject area expertise [9]. Most also occur in small courses with <100 students [11], leading to questions about whether CUREs can be effective in larger courses. In physics, few CUREs are currently represented in the literature, and no physics CUREs in the literature take place in large, introductory courses [9].
Despite the lack of CUREs in physics, a large, introductory laboratory course at the University of Colorado (CU) Boulder was redesigned as a CURE during the COVID-19 pandemic. Alongside many other courses, this course was required to be run in a remote format in the Fall 2020 through Fall 2021 semesters due to the pandemic. Traditionally, this course emphasized teaching students experimental skills [12] and maintaining this emphasis was one of the largest considerations while redesigning the course [13]. Because participation in authentic research requires students to engage in scientific practices, redesigning this course as a remote CURE was an ideal solution to the challenges brought on by the COVID-19 pandemic [13]. The development of this course and some of its preliminary outcomes are detailed by Werth et al. [13]. For more information on the course design, see Sec. III.
Because this course is the first reported large, introductory CURE in physics, and because participation in authentic research is a key element of what differentiates a CURE from a traditional laboratory course, we sought to understand students' experiences with authentic research in this course. Particularly, this work addresses the following research questions: RQ1: To what extent did students feel that they were participating in authentic research during this course? RQ2: When students talk about participating in authentic research, what aspects of authentic research did they feel that they had experienced? This course was designed to engage students in authentic research, so the answers to these research questions are important metrics for the success of this course. We demonstrate here that a large, introductory-level physics CURE can engage students in authentic research and propose some potential features of the course that led to these outcomes. Furthermore, we hope that the answers to these questions will help future instructors consider implementing CUREs at their own institutions in order to help their students participate in authentic research.

A. Course-based undergraduate research experiences
A CURE is a relatively new course format that has been implemented, studied, and defined only within the past decade. The key features of CUREs, as determined by a 2014 meeting report [8], are as follows: 1. Use of scientific practices: Includes asking questions, building and evaluating models, proposing hypotheses, designing studies, selecting methods, using the tools of science, gathering and analyzing data, etc. While a single CURE cannot include all of these components, it is important that students have the opportunity to engage in more than one. 2. Discovery: Involves students investigating scientific questions that have not been answered by either the student or the instructor. 3. Broadly relevant or important work: The student's work must be relevant to individuals or communities outside of the classroom. 4. Collaboration: Students work together in order to improve their work, develop teamwork skills, and practice communicating. 5. Iteration: Can appear in many ways in a CURE.
Students iterate on their own work, revise aspects of other students' investigations within their course, or revise work that has been done across successive offerings of the same course. While each of these aspects can occur individually in any lab course, it is the presence of a combination of these aspects together that defines a CURE [8].
CUREs can occur at many scales and types of institutions. Some CUREs are part of national programs sharing a common research goal where faculty join the program through an application process and receive support from the national leader to prepare them to run the CURE at their institution [10,14,15]. Instructors of these types of CUREs also tend to have some degree of communication with other CURE instructors [10]. SEA-PHAGES is an example of such a CURE at the introductory level [14], while the Genomics Education Partnership serves upper-division students [15].
CUREs can also take place nationally through the use of a common framework. For instance, the Small World Initiative is a nationwide project that crowdsources the development of new antibiotics through introductory biology courses. In this program, instructors receive some training and course materials from the initiative before teaching the course but do not have as much formal structure as those involved in a national program [16].
Finally, CUREs can occur at a single institution and allow students to participate in the research of an individual faculty member or local researcher. Most of these CUREs occur in biology or chemistry but within a wide range of subdisciplines.
Many more CUREs likely exist than are represented in the literature, but, currently, few CUREs are documented that serve lower-division students, students in large courses [9], nor students in physics classes. More information on individual CUREs can be found at CUREnet [17].
Additionally, there are few documented CURE-like experiences within physics departments. One such research experience takes place within the context of the Freshman Research Initiative (FRI) at the University of Texas at Austin. While physics research experiences are sometimes offered as "research streams" or areas of interest within this program, the FRI is different from a traditional CURE in the sense that they do not occur within the context of a single course [18]. In another physics course, students built light microscopes but then used them to answer a biological question rather than a physics one. This course, therefore, taught students important scientific practices, but the physics done in the course was not relevant to stakeholders outside of the course [19].
Studies that have compared CUREs to traditional UREs have shown that students in CUREs have many of the same outcomes as students in traditional UREs [8]. Furthermore, some of these outcomes tend to last for an extended period of time after the CURE instruction [8]. This suggests that CUREs have the potential to be positive experiences for students and to increase access to the gains associated with undergraduate research experiences.
While CUREs provide immense benefits to students, there are many challenges that instructors face when implementing a CURE [20]. One of these challenges is finding a research project that is amenable to a CURE. While this challenge is certainly present for CUREs of any size and time scale, it is especially salient for instructors seeking to run a long-term CURE. Over the lifetime of a CURE, students will eventually answer the research question(s) of the course, creating a perpetual need for new and relevant research questions for students to answer. As the research questions change so, too, must the instructional materials, creating an additional burden on the instructor. For this reason, the long-term sustainability of a CURE is often an issue, especially in physics, where appropriate research questions for large groups of introductory students may be more challenging to find. In the case of this CURE, these limitations meant that the CURE version of this course was only offered for three semesters (Fall 2020 through Fall 2021).

B. Defining authentic research
In order to understand whether students found this research experience to be authentic, we must first clarify what we mean by authentic research. Authentic research is currently ill-defined among researchers and educators [21]. For instance, Rowland et al. [21] summarized the existing literature on authenticity in STEM education and identified 26 distinct definitions of this type of authenticity from the three decades prior. These definitions varied based on the educational context of the respondent, for instance, whether they were addressing K-12 level or college-level science education [21].
Most commonly, the literature describes authentic learning in science to include "experience of what scientists do, how science is done, and what science is." Literature focused on college-level education narrows this definition slightly, sometimes requiring that authentic learning must also include finding "novel results" or doing "experimental design." There is also some disagreement on whether "authentic" research should focus on the research process or on creating a final product that is novel and relevant to the scientific community [21,22]. Some courses, for example, have been redesigned for "authenticity" by introducing students to scientific practices that are done by real researchers in the field within a traditional laboratory course (i.e., where there is no expectation that students are creating novel results) [23].
Nevertheless, a key component of the research done in a CURE is that it must be "broadly relevant or important work." This phrase was chosen by Auchincloss et al. intentionally in order to avoid the ambiguity that comes with the word authenticity. Broadly relevant or important work can appear in different ways depending on the course, including through publication, reports to a local community, etc. [8].
We adopt the following definition of authentic research experiences from Rowland et al.: "those that flexibly engage students in research practices while they work on novel experimental questions to produce potentially publishable data for audiences who are interested in the scientific outcomes of their work" [21]. This definition captures both participation in "'broadly relevant and important work' that matter to external stakeholders," as well as gaining scientific skills, which was a fundamental goal of this course.

C. Framework of authentic research
In order to understand our students' experiences with authentic research in this course, we examined frameworks that others have used to understand authentic research experiences. Goodwin et al.'s 2021 paper [24] comparing a CURE to an inquiry-based lab course highlights some aspects of a research project that contributed to students' perception that they were doing authentic research. They identified that students in the CURE class perceived their work to be more authentic than students in the inquirybased class and proceeded to conduct weekly reflections in order to understand why this was the case. [24].
Goodwin et al. then coded these reflections for instances where the students mentioned feeling that the research project was or was not real research. A subset of these reflections was then coded again for justifications of why the students felt like the research experience was or was not real. Their coding scheme captured both important elements of the CURE framework [8] and emergent codes from the student responses. The aspects of authentic research that they looked for in these weekly responses were [24] as follows: 1. Failure: Experiencing failure or setbacks. 2. Iteration: Repeating experiments or doing the experiment over a period of weeks. 3. Scientific practices: Using the practices, methods, tools, or processes of science.
4. Relevant discovery: The potential for novel scientific discovery and/or the relevance of the project to the scientific community. 5. Autonomy: Having autonomy, ownership, or creative license (including in experimental design and interpreting results). 6. Collaboration: Working with classmates on their research project. 7. "Successful" Science: Producing data or results, experiencing success in experiments, or answering research questions. We used these ideas to develop a codebook to understand what our students might have to say about authentic research in our specific CURE. While our codebook differs somewhat from Goodwin's, we used their ideas to scaffold our understanding of which elements of authentic research we might expect our students to connect to their experience within the CURE.

III. COURSE STRUCTURE
This CURE was a 15-week course in which students engaged in a solar physics research project called the Colorado Physics Laboratory Academic Research Effort (C-PhLARE). Over 400 students each semester participated in C-PhLARE, with most of these students being physical science or engineering majors. Students engaged in weekly 2-h laboratory sections via Zoom, where they worked in teams of 3-4 students. One graduate teaching assistant was present for each section. In addition to these weekly labs, there were seven prerecorded asynchronous lectures that students watched prior to their lab section. These lectures were about 15 min long and contained questions that students could answer for a small amount of course credit. There were six main phases to this course that students participated in. These phases were 1. Project onboarding. 2. Research plan development.
3. Data analysis. 4. Peer review. 5. Calculating alpha. 6. Documentation and reflection. C-PhLARE was an introductory level course, so the majority of participants are at the beginning of their academic careers (first and second years). The CURE version of this course was offered for three semesters (Fall 2020 through Fall 2021) and, during that time, there was no alternative traditional laboratory option for students looking to enroll in this course. For more information about the details of the course, see Ref. [13].

A. Learning goals
We had several important goals for this course that motivated its structure and components. These goals were as follows: 1. Learning "skills" remotely: We wanted to maintain the emphasis of this course on student learning of lab skills over physics concepts. This course had no learning goals explicitly connected to learning physics content, instead focusing on goals that connected directly to working in lab environments. 2. Group work: In order to both continue to provide the benefits associated with teamwork [25] and combat loneliness associated with the pandemic [26], this course focused heavily on teamwork. 3. Developing a unique and motivating experience: One of the goals of the prepandemic version of this course was that students should have a positive attitude about the course and about experimental physics. We hoped to maintain this learning goal during the pandemic version of the course.

B. The CURE research project
Students in this course partnered with a scientist who studies the sun in order to participate in authentic research during the CURE. This scientist was a researcher at the Laboratory for Atmospheric and Space Physics (LASP) and, in partnership with the course instructors, was able to identify a research question that could be investigated by the students and would be beneficial to his research program. He was the principal investigator (PI) of the project and was introduced to the students as such. This particular project was chosen for our CURE because the research question was of genuine interest to the scientific community, required the contribution of a large number of scientists, and first-year undergraduates possessed the technical skills necessary to complete the project work.
C-PhLARE addressed a well-known question in solar physics: why is the corona of the sun several million degrees kelvin hotter than its photosphere, despite being farther from the center of the sun? Some scientists have speculated that this heating is caused by the energy of many nanoflares, whereas others posit the dominance of some other mechanism, like magnetohydrodynamic waves [27].
One way to answer this question involves looking at the flare frequency distribution (FFD), which is the solar flare frequency versus energy. If nanoflares are the dominant mechanism creating the high temperature of the corona, they must occur with a high enough frequency for their small individual energies to add to the necessary magnitude [28]. The FFD is given by the power law where n is the number of events per year, E is the radiated flare energy (usually calculated from the long x-ray region of the spectrum), A is a constant, and α is the exponent. When these data are presented on a log-log plot, A can be calculated from the intercept, and −α appears as the slope.
Hudson et al. have shown that if α < 2, then nanoflares are not frequent enough to be the dominant mechanism of coronal heating [28]. Therefore, determining the value of α is important for answering our research question. However, collecting enough data to accurately determine α is challenging. In order to determine α, the energy of many flares must be accurately determined from a light curve (measured irradiance as a function of time). While this seems simple, it is notoriously difficult for an automated computer algorithm. For instance, one proposed algorithm for analyzing coronal mass ejections was unable to analyze 70% of candidate events [29]. The reason for this is because, in order to determine the energy of a flare, its start and end times must be determined and a baseline correction must be applied. These decisions, while difficult for computers, are relatively straightforward for humans to make.
Applying background corrections and determining the start and end point of flares is nonetheless time consuming, meaning that the groups that have calculated α in the past have typically done so with a small number of flares or only within a specific part of the solar cycle [30][31][32]. This made finding α an ideal project for a large number of undergraduates with knowledge of introductory calculus. Our students were capable of applying a background correction, determining the start and end point of a flare, and doing a numerical integration to find its energy. Furthermore, since this is something that is not easily done by a computer, these students could make a meaningful contribution to science through their work on this project.
Throughout this course, teams of 3-4 students worked to find the total energy of a solar flare in the long x-ray region. Each team could then use the aggregated data collected by the class to determine a value for α.

C. Designing the course for authentic research
Many aspects of this course were intentionally designed to engage students in authentic research experiences. First of all, the fact that students participated in research that made a genuine contribution to science was emphasized to students at all stages of the project. This messaging was consistent in all course materials and communicated to teaching assistants so that they could also convey the messaging to the students. The following authentic research practices were also built into the course in order to give students a research experience that was representative of the way physics is actually done.

Literature review
The course began with students conducting a review of some relevant literature to familiarize themselves with the existing research on this topic. Excerpts from relevant literature (See Appendix B) were selected by the instructors and PI, and students were asked to read the literature before their lab section. In the lab section, students then discussed aspects of this literature with their team. The chosen literature featured similar studies calculating α, but they differed in key ways from the work conducted by the students in this course. For example, some past studies [30][31][32] used data from other solar cycles, did not look at data across the entire solar cycle, did not cover as large a range of solar flare energies, and calculated α using far fewer flares than our students were able to collectively analyze [13]. Students were prompted to consider, "What is unique about this project that differentiates it from past research?" Because these students were early in their undergraduate career, significant scaffolding was done for them so that they would have some idea of what to consider while reading these new materials.

Research meetings
Twice during the semester, students attended whole-class research meetings via zoom with the PI. These meetings occurred during the third week of the semester and the final week of the semester. These meetings were an opportunity for the students to understand some more of the broader context of their work and to connect with the researcher who was guiding their work.
At the first research meeting, the PI introduced the students to the research that they would be working on for the semester. He emphasized to them that their research was something he had been interested in working on for a long time but had been unable to accomplish on his own. He specified the goals of the research and the specific work that the students would be doing in order to answer the research question. He, again, specified that the students were participating in real research that was important to him and to the scientific community.
At the final research meeting, the PI highlighted the results that the students had achieved, emphasizing the importance of the results. He also explained the publication process to students.
There were opportunities at both research meetings for students to ask questions. Some of these questions were prepared by students in advance and others were asked via the Zoom Q&A feature. Students were therefore able to interact directly with an expert in the field, which is an important part of authentic research.

Metacognitive reflections
Each week, students responded to one or two questions designed to get them thinking about their experiences in the course throughout the week. Responses to these questions were required but graded only for completion. These reflections were designed to give students the opportunity to think about their research, the challenges they were encountering, and the successes that they had. These metacognitive skills are important for problem solving in real research and, therefore, we made an effort to support students in developing these skills. Some examples of these reflection questions are provided in Appendix C.

Working on a team
Students were divided into teams of 3-4 at the beginning of the semester and worked with a consistent team throughout the project [13]. Teams of students were selected intentionally such that no groups contained only one student who was not a man, and all groups contained students with similar coding experience [13]. At the beginning of the semester, teams participated in a teamwork training where they read and discussed various teamwork scenarios with their teammates and identified ways to deal with difficult situations that might be encountered when working on a team [13].
Students worked in Zoom breakout rooms with their teammates each week to develop a research plan, analyze their solar flare data, and compile the classwide results. Students were encouraged to define clear roles and alternate between those roles during each laboratory session [33]. Many assignments were submitted as a group, and each group member received the same grade for these assignments [13]. This allowed students to practice engaging in teamwork in an authentic research setting with scaffolding to support the development of teamwork skills. For more information on teamwork in this course, see Ref. [34].

Peer review
After teams determined the energy of their flare, they engaged in a double-confidential peer review process with students from another section of the course. Each student was assigned one other group's flare analysis to review in order to ensure its validity. After the students received feedback on their work from another student, they were given the opportunity to revise their work and resubmit it [13]. Confidential peer review is an essential part of authentic science, and the peer review process was emphasized by the PI as one way that he could trust that the students had done accurate work.

Memos to future researchers
At the end of the semester, students wrote a "memo to future researchers," which was an informal letter discussing their experience in the course, with the framing of onboarding future students to the project. Students were asked explicitly to reflect on the project in a semi-open-ended format for an audience of students who would be joining the course. We found that students used this opportunity to discuss many aspects of the course that were important to them, including their experience with participating in authentic research [13]. Students were therefore able to participate in the process of passing their work off to the next group of researchers, which is an important part of authentic science.

Publication
The work that students did throughout this course is currently in preparation for publication in The Astrophysical Journal [35]. All students were given the option to be listed as coauthors on this paper due to their important intellectual contributions to the work. Because publication and sharing results with the scientific community is a crucial part of authentic scientific work, it was important that the research done in this course be suitable for publication. At the end of the semester, students were informed about the process of publication in a scientific journal and when they could expect the manuscript to be ready for review.

A. Study participants and demographics
The data used in this study were collected from students in the Fall 2020 semester of this course. In the Fall 2020 semester, 440 students participated in the course. Of these students, 407 (92.5% of the class) completed a postcourse survey that asked students to self-report their gender, race and/or ethnicity, major, and year at CU. The students' demographic data are presented in Table I.

B. Data sources 1. Postsurvey question
The first data source we consider is a question from the postsurvey given at the end of the semester. This question asks students: "During your research experience in this course, how much did you engage in real-world science research?" The question was scored on a 5-point scale, with 1 corresponding to "none" and 5 corresponding to "a great deal." This question, in particular, was part of a group of questions asked on the postsurvey from the Undergraduate Research Student Self-Assessment (URSSA) [36]. All 407 students responded to this item on the postsurvey.

Memo to future researchers
The second data source used in this work is student responses to the memo to future researchers assignment. This was an assignment given to students at the end of the semester and was framed as an opportunity for them to onboard future students to the project. Students were asked to write an informal letter to future students discussing their experience and giving recommendations to the students who would be continuing the project the following semester; 405 students (92.04% of the class) responded to the memo to future researchers assignment [henceforth referred to as memo(s)].
In the memo assignment, which took place in the final week of the course, the students were asked to respond to the following questions: 1 What should their goals for the research be? Although these questions centered around parts of authentic research (i.e., students were asked to describe their research goals and conclusions), students were not explicitly asked to discuss their experience with doing real research in these memos.

C. Codebook
In order to analyze the student responses to the memos, we developed a codebook based on Goodwin et al.'s findings about which aspects of a CURE contributed to students' feelings that their research was authentic [24], as well as Rowland's definition of authentic research as mentioned in Sec. II. We modified Goodwin's et al.'s codebook in a few crucial ways: First, we changed their code labeled "autonomy" to "decision making" in order to clarify that we are applying this code to instances where students discuss making their own decisions about the direction or process of the research [24]. We also added a code for "comparison to prior studies" since this was something that our students regularly discussed due to the nature of their research. Finally, because we did not explicitly ask our students about the authenticity of their research experience, we added three codes to capture instances where students explicitly talked about their research experience as authentic: These codes were did "real" research, felt like a scientist, and understood real research. The entire codebook can be found in Appendix A, Table IV.
Direct authenticity and indirect authenticity codes were used to identify when students were speaking about having an authentic research experience in the course. We include subcodes within the indirect authenticity code because these subcodes capture instances where students discuss having participated in "novel experimental questions to produce potentially publishable data for audiences who are interested in the scientific outcomes of their work" [21] and therefore align with Rowland's definition of authentic research. These two codes, then, help us to answer RQ1.
Indirect authenticity and research components codes were used to identify what types of activities students engaged in when they felt that their research experience was authentic. These codes can therefore answer RQ2. Because we were interested in only what activities students associated with an authentic research experience, research components codes were coded only alongside a direct authenticity or indirect authenticity code. For instance, student discussions of engaging in data analysis were not coded as scientific practices unless the student discussed this data analysis in the context of the authentic research project.
In order to ensure that this codebook could be applied to the memos consistently, the interrater reliability (IRR) was assessed. Nineteen student responses were co-coded by authors K. A. O. and A. W. in order to determine agreement. They achieved a Cohen's κ of 0.79, which is defined as "substantial agreement" [37]. After this was performed, all authors discussed where there was disagreement and clarified criteria for applying these codes. For instance, the collaboration code was redefined to include only instances of collaboration with professional scientists, where teamwork between students was placed in scientific practices. Cohen's κ after these discussions was 0.82, which is considered "almost perfect agreement."

D. Limitations
While this work allows us to characterize student perceptions of authentic research in this course, it is important to note that there are several limitations to our work that impact both students' ability to participate in the full process of authentic research and our ability to identify the degree to which students felt that they had participated in authentic research.
Because of our intention to have students participate in authentic research to the highest degree possible, we strove to engage students in many aspects of authentic research. However, due to the large number of students in the class and our desire to prioritize research that had an impact on the scientific community and society more broadly, our course was structured such that students had few opportunities to fail on a large scale or to make big decisions about the research questions or the methodology used in the project. This limited the degree to which students were able to participate in some aspects of authentic research. This is described further in Sec. V.
Another limitation of this work is that, because we did not explicitly ask students to discuss whether they participated in authentic research in an open-ended format, our data analysis process is inherently making inferences about what students were considering authentic research in their memos. While we did our best to exclude cases where we were unsure of the students' intent, it is impossible to guarantee that there are not places where our interpretation of a student's words are different than what they would have said if directly asked.

V. RESULTS AND DISCUSSION
We begin by examining the results from the postcourse survey question in order to motivate our analysis of the student "memos to future researchers." Then, we present some qualitative data from the memos to future researchers in order to illustrate the nuanced ways in which students thought about their research experience in this course. We then conclude by discussing the quantitative results from the coding of the memos. Although the quantitative memo data synthesizes the overall sentiments of the students well, we discuss the qualitative data from the memos first to give readers the opportunity to engage deeply with the richness of the student responses before presenting the quantitative data, which may mask some of the depth of the students' experiences.

A. Survey results
Nearly all students (93.9%) in this course felt like they "engaged in real world science research" at least "some" throughout this course, with most students (72%) falling into one of the top two categories ("a fair amount" or "a great deal"). Figure 1 shows the fraction of students responding to each multiple-choice option on this survey question.
These survey results show that, when directly asked, most students felt that they engaged in authentic research during this course. However, this survey question gives us little insight into why students felt that the course was an authentic research experience or what that experience was like for them. Therefore, we use the memos for more detailed evidence of what students considered authentic about their experience in the course.

B. Individual student memos
Here, we present portions of four student responses to the memos. These data are presented in Figs. 2-5, and sections of the memos are highlighted to indicate how they were coded. These particular student memos were chosen to capture the variety of ways that students responded to the prompt; we intentionally chose to present responses that are different from one another. Furthermore, these student responses were selected to highlight the depth with which many students chose to respond to the prompt and, therefore, we present student responses that answered the prompt in detail. Direct authenticity codes are highlighted in green, indirect authenticity codes are highlighted in blue, and research components codes are highlighted in yellow.
Within these examples of student writing, we see students discuss their authentic research experience to varying degrees throughout their memo. Some students directly discuss what caused their research to feel authentic, whereas others simply draw connections between different aspects of their experience. Nonetheless, these memos are examples of the variety of different ways that students discussed their experiences in this course. Figure 2 shows Student 1's response to the memo prompt. In paragraph 1, the student discusses their research as "novel" and indicates that it is important in order to help scientists (relevant discovery). This suggests that not only did this student feel that they had an authentic research experience during this course but also that it was important to emphasize the novelty and significance of this research to the scientific community to future students.

Student 1
This student reiterates the importance of participating in authentic research in paragraph 3, when they say that they learned a lot about using the scientific method in "realworld contexts." This sentence, coded as understood real research, suggests that this student believes that their learning about the way science works was facilitated by the context in which it took place and follows this up by connecting their statement about understanding real research to the research skills that they practiced and learned: collecting and analyzing data, doing peer review, and engaging in iteration. In paragraph 4, this student makes suggestions to a future researcher about research components that benefited them, emphasizing that all of these skills are "helpful in increasing one's confidence as a researcher." From their memo, we can see that this student believed the research in this course to be authentic. Beyond this, this student also perceived their research to be of use to the scientific community and connected their experience with understanding real research to the skills that they learned and activities they participated in throughout this course.

Student 2
Similarly, Student 2 (Fig. 3) emphasizes the importance of their daily activities in the course to their authentic research experience. For example, in paragraph 2, Student 2 mentions that in the class "you will find, name, and analyze your very own solar flare." This statement occurs between two indirect authenticity codes, showing that this student connects their experience engaging with a scientific practice (data analysis) to why it was important: benefiting the scientific community and protecting society. They then emphasize that their work was important by stating that their data analysis will be communicated to others through publication in a "real scientific journal." Then, in paragraphs 3 and 4, this student goes on to discuss their experience with understanding real science in this class. They mention that, while science is "just a bunch of people struggling with code until it finally works and you learn something," the most important things they learned were not about writing code but instead about how to conduct "meaningful science." In multiple places, Student 2's memo tells us that they felt like a key component of their authentic research experience was participating in data analysis and coding. Nonetheless, they felt like learning those skills was not the most important part of the research experience. Instead, they emphasize the importance of learning how to do "real" science, and that the data analysis they did was in service of the larger goal of helping scientists to positively impact society.

Student 3
This portion of Student 3's memo provides a more cautious framing of the authentic research experience and FIG. 2. Student 1's response to the memo assignment. Green highlighting corresponds to a "direct authenticity" code, blue highlighting corresponds to an "indirect authenticity" code, and yellow highlighting corresponds to a "research components" code. begins with their response to the third memo prompt asking them to describe their personal experiences in the course. The responses to the first two prompts are not included here as the student writes a great deal about the course mechanics without additional insight about their experience. This student (Fig. 4) begins their memo by discussing that they felt unsure about their ability to do authentic research-they write in paragraph 1 that they found it challenging to "wrap [their] head around the idea that [they] were doing real research." Nonetheless, this student goes on to say that the research components they participated in, such as peer review and receiving guidance from other researchers, helped them feel that the quality of their work was high.
In paragraph 2, this student goes on to express that they felt like their research had the potential to make contributions beyond the course itself. Student 3 mentions that their work could contribute to "a better understanding of space weather? Finding stars similar to the sun that could support life on other Earth-like planets? A system to predict and prepare for extreme flare events?" This quote reiterates that Student 3 did feel like their research could have value to individuals outside of the course, despite their uncertainty about their own competency. This student's response to the memo to future researchers demonstrates that while Student 3 felt challenged by the research and did not fully believe in their abilities, they still felt as though they were engaging in authentic research throughout this project. Furthermore, they were able to have additional trust in their abilities due to engaging in some of the activities involved in real research, like peer review.

Student 4
Student 4 (Fig. 5) is one of the few students who indicated that they wished they could have engaged with the research in a more authentic way. This student says, "for the first few weeks, I very rarely took a step back and really dug into the work… I blindly followed the pseudocode, and my group and I followed the tasks until we derived what we believed to be a reasonable answer." However, later in their memo, student 4 says, "Eventually… it felt as if we were actually conducting research and not just copying code because we all invested ourselves in the course." This suggests that, while this student initially questioned the authenticity of the research, they eventually came to understand that they were conducting real research.
Despite this student's initial hesitancy about the authenticity of their work, this student talks extensively about the real research they did end up doing. For instance, in paragraph 1, they discuss that they were thinking about the impacts of this research on the scientific community (relevant discovery) and on "our Earth in general" (societal impact). This student also talked about collaborating with the PI during the course's research meetings and feeling like the PI was a "phenomenal source for information" and that collaborating with a professional scientist in the form of a research meeting allowed them to gain context behind the "numerical data that [their] computer would spew out every week." While student 4 brings up several ways that the research project could have engaged them more deeply, for instance, in paragraph 5 where they mention wishing they had had more "real world applications" of the data, they also discuss feeling like they engaged in authentic research in several places and in varying ways within their memo. This student also emphasizes that many of the activities they participated FIG. 5. Student 4's response to the memo assignment. Green highlighting corresponds to a direct authenticity code, blue highlighting corresponds to an indirect authenticity code, and yellow highlighting corresponds to a research components code. in throughout this course, such as participating in research meetings, affected their belief that the research they did was authentic.

Summary of individual student memos
These student responses are examples of the richness and level of detail that many students provided in response to the memos, as well as the ways they talked about authentic research. The fact that these students independently discussed their experience with authentic research in such detail suggests that they likely felt it was a particularly important part of their overall course experience-enough so that they wrote about it to future students.
Additionally, all of these responses frequently contained direct authenticity or indirect authenticity codes that were connected to research components codes. This connection between research components and authentic research suggests that these students found participating in research practices valuable because it contributed to their research and not necessarily for the sake of simply learning and engaging in experimental skills.
Although we present only four student memos here, these students were far from the only ones to express these types of views. In the following section, we present some quantitative data to generalize the student experiences presented here and demonstrate that these experiences were common among students in this course.

C. Quantitative results from memos to future researchers
Of the 405 students who responded to the memo assignment, 314 (77.5%) mentioned having participated in authentic research (at least one part of their memo was coded as either direct authenticity or indirect authenticity). This suggests that the majority of students not only felt like their research experience was authentic but that the authenticity of their research experience was important enough to mention to a future student in the course without being prompted. Counts of the number of students whose memo contained at least one instance of each code can be found in Table II.
When students did discuss having had an authentic research experience at any point during the course, they most frequently expressed having participated in relevant discovery, societal impact, scientific practices, and doing real research (see Table II). This is to be expected given that the project was aimed at answering a novel question that mattered to individuals outside of the course and this fact was emphasized to students repeatedly throughout the course. Students were also engaging in coding and data analysis almost weekly due to the course design, both of which are important scientific practices.
The codes occurring least frequently in the memos were failure, iteration, felt like a scientist, and decision making. This is likely also due to the nature of the course.
The research that students participated in was designed to not have a high probability of failure that would derail the project as a whole and, because so many students were working on a large-scale project collaboratively, most major decisions were made by the instructional team beforehand. We may also expect students to underreport "negative" codes, such as failure, due to the self-reporting nature of the data. Finally, while students did engage in iteration through the process of revising their research plans and engaging in peer review, these aspects of the course were largely coded as scientific practices since, for the most part, students did not discuss the iterative aspects of these activities.
Students discussed several specific types of scientific practices within their memos. The specific scientific practices discussed were likely a result of the course structurestudents were analyzing data and working with code on an almost weekly basis. The types of scientific practices students discussed are (in order of frequency) As may be expected, students frequently discussed the scientific practices that the course structure required them to do most often, including data analysis, programming, and teamwork. Furthermore, the scientific practices that students discussed least frequently were those that happened less frequently in the course or were done early in the course, such as meeting with PI and literature review. Students also made connections between codes from multiple different subcategories within their memos. For example, connections between direct authenticity statements and indirect authenticity statements, as well as between indirect authenticity statements and research components were made quite frequently. This suggests that students' experience engaging in the everyday tasks of their research project was intimately connected to their perception of the research as authentic. One instance of this overlap between codes occurred in the following student response: Being able to analyze real-world data for a real-world experiment is a wonderful opportunity, and it is comforting to know that this hard work is actually going towards something tangible and useful.
This student, and many others, indicate that the everyday research components tasks they participated in were important because they contributed to real research.
Appendix D shows the counts of how frequently each code co-occurred with other codes. We define co-occurrence of codes as an instance where a student's single idea is coded multiple times. This could either be multiple codes appearing within the same sentence or adjacent to each other within a paragraph. The co-occurrence of codes differed widely between codes. Some codes like comparison to prior studies, occurred most frequently with one other code (in this case, relevant discovery) while others, like collaboration, occurred frequently with many other codes.
Notably, the scientific practices code occurs frequently with every other direct authenticity and indirect authenticity code, reemphasizing that students were regularly connecting their everyday activities in the course to a wide variety of reasons why those activities were important. This is shown in Fig. 6.
Furthermore, comparison to prior studies was coded with relevant discovery about 45% of the time (see Fig. 7). Students regularly talked about having discovered something new about science and then compared their results to those of prior studies, suggesting that this research was fundamentally different for them than a standard laboratory experiment where they were replicating an existing experiment to find a known quantity. Instead, these students felt that their experience in comparison to prior studies was tied to their discovery of new things. For instance, one student wrote, Looking at the final results compared to previous studies that have been conducted (there are not that many), our data fit in between two of the studies but is multiple orders of magnitude different from other studies. This is the reason that we need to do more studies to fill in the graph and see which of the studies are actually accurate and which are garbage.
This student's quote emphasizes that they understood the purpose of replicating prior studies to be to add to scientific knowledge rather than in order to meet the requirements of a course or to exclusively confirm the same results.
Both direct authenticity codes displayed in Appendix C most commonly occurred either alone or with a scientific practices code. The lack of overlap between direct authenticity and indirect authenticity codes, as shown in Table III, may be because students who discussed having done novel research that benefited the scientific community or the world as a whole may not have felt the need to emphasize to a future student that this research was real in a direct way, instead allowing their description of the research to speak for itself.
There were also some references within the student memos that were coded 3, 4, or 5 times. These consisted of 86 (17.3%) of all references that were coded as authenticity.  A high percentage (88%) of references that were coded three or more times were co-coded with scientific practices.

VI. CONCLUSIONS AND RECOMMENDATIONS
The goal of this work was to answer two main research questions about students' feelings of participating in authentic research. We answer the first question, "To what extent did students feel that they were participating in authentic research during this course?" (RQ1) by examining students' responses to the memo to future researchers assignment, as well as student responses to a postcourse survey question. From these, we are able to provide evidence that most students did feel that they were participating in authentic research during this course. In the memos, many students discussed this unprompted, suggesting that their participation in authentic research was an important aspect of the course for them.
Additionally, we probed the question, "What aspects of authentic research do they feel that they have experienced?" (RQ2). We conclude that many students felt that, throughout their research project, they engaged in a variety of aspects related to authentic research, such as answering novel research questions, contributing to science that will positively affect the world, engaging with scientific practices, and doing research that will be communicated to the scientific community through a publication. Furthermore, many students noted connections between these aspects of authentic research, frequently writing about two or more of them together.
While our results do not allow us to definitively determine how much each component of the CURE led to students' perceptions that their research was authentic, we present some aspects of the course that could have led to these beliefs in our students.
First, although students did not often talk about their doing publishable science, they often talked about relevant discovery. The fact that this research was going to be part of a peer-reviewed astrophysical publication may have highlighted that the students' research was relevant and of value to individuals outside of the course. Additionally, we saw that students made many connections to their authentic experience in the class and the research components. Perhaps, the fact that this work was going to be published in a real journal emphasized that they were responsible for doing work that was of the quality required to publish in a scientific journal. This was emphasized by the instructional team throughout the course, likely leading students to believe that this was an important part of the course and, therefore, of their research experience. Finally, students often wrote that they did real research. Since real research gets published, the students may have connected these two aspects.
Furthermore, the PI's framing of the students as important contributors to the research and frequent reminders of how excited he was to be involved in working with so many students likely influenced students' perceptions of the research as authentic because they knew that they were helping a "real researcher" with work that he had been wanting to accomplish for some time. Although the PI meetings happened only twice-at the start and end of the semester-and students rarely discussed them in the memos, the meetings were the students' primary exposure to the societal impact of their work which was commonly discussed by the students.
It is important to emphasize that despite the prevalence of students discussing research components in their memos, we do not intend to claim that the presence of activities involving research components in this course caused students to feel that the research was authentic. On the contrary, it is much more likely that engaging in authentic research helped students feel that the research components they learned and participated in were meaningful. Therefore, we encourage future instructors to incorporate research components into authentic practices but not to rely on them alone to foster authentic research experiences. We propose that the same activities, such as data analysis and peer review, done without the goal of answering a real research question that has value to the community beyond the course would not have led to the same results. We also encourage future researchers to continue investigating the connection between authenticity in labs and research components.
Our results from this study demonstrate that a large, introductory physics CURE can provide students with an authentic research experience. Although finding sustainable research project ideas continues to be an issue for developing CUREs, especially ones that can serve high-enrollment courses and are suitable for introductory physics students, we encourage instructors to continue thinking about how to develop these courses in order to foster research experiences for future students. We encourage instructors to think about CUREs in various subdisciplines of physics: while our implementation of a large-enrollment physics CURE was within solar physics, the components of CUREs could certainly be implemented with research questions from other areas of physics. Furthermore, we encourage researchers to study these CUREs as they are developed so that the community can gain a better understanding of which course elements best lead students to perceive that they have participated in authentic research. Table IV is a table consisting of the full codebook used in this work. The codebook was first developed and then grouped into three larger codes (direct authenticity, indirect authenticity, and research components) and the original codes became subcodes of these overarching codes. However, when the semester is over and you look at the data collected by the entire class, you will feel a certain sense of pride in being part of actual scientific research.

APPENDIX A: CODEBOOK
Felt like a scientist Explicit statements about feeling like they were a scientist.
At the end of the project, your research results will be added to the scientific community's knowledge. This will make you feel like a real scientist working in physics.

Understanding real research
Student's statement about the authenticity of the research project was based on the feeling that they now understand more deeply what scientists do.
Overall, remember that this class is kind of a "How to Science" class. The most important things you learn are not how a solar flare works or how to write code. It is how to conduct good and meaningful science. It is how to ask scientific questions and work collaboratively.

Indirect authenticity Relevant discovery
Statements about feeling like their authentic research produced (or could produce) novel results that are meaningful to the scientific community. (must explicitly be relevant to individuals outside of the course).
You will be producing research results that would contribute positively to the scientific community's knowledge.
Societal impact Statements about students feeling like their authentic research had impacts that were relevant to the broader community.
After doing all of this research, we can look at our results and see which flares occur the most and how frequently each level of energy is seen. This is important for predicting future sun flares that can be harmful to space equipment and take out power on earth.
Publishable science Statements about students feeling like their authentic research produced publishable results.
The work that you do in this class is more than classwork: the information you come up with will be used and published in a research paper with your and your classmates' names on it! Collaboration Statements about feeling like they collaborated with outside and professional scientists as a part of their authentic research experience.
Anyways, take advantage of Dr. Mason, he loves this research project and he is a phenomenal source for information. The two research meetings with him were probably my favorite parts of the course, because I was able to learn some conceptual information in contrast to the usual numerical data that my computer would spew out every week during lab.

Research components Failure
Statements about feeling like they experienced failure during their participation in an authentic research project.
I learned that doing research is a lot more strenuous than it puts on: There's a lot more steps, a lot of frustration, a lot of annoying moments, and a lot more blockages that we have to overcome in order to move onto the next step.
(  Dr. Mason also said that the results were impressive with respect to what kind of results he thought we were going to get. In the final full class meeting, he showed us how our results compared with other research studies, and it was within similar data ranges as the other ones.

Iteration
Statements about students feeling like repetition of certain activities (peer review, revising their research plan, etc.) was a part of their authentic research experience.
I learned a lot about applying frameworks such as the scientific method in a real world context. The iterative process of collecting data, analyzing that data, and improving our methods through peer review was an invaluable skill I could leverage in other research experiences I am involved in.
Scientific practices Statements about feeling like they engaged in activities that scientists engage in (i.e., scientific "skills"-coding, working with data, engaging in research meetings, and teamwork) as a part of their authentic research experience.
You will be joining a real research project to analyze and study the energy and frequency of solar flares in our Sun. An instrument on a spacecraft that has been sent to outer space had collected data which you will be analyzing using PYTHON to do basic experimental data analysis.
Decision making Statements about students feeling like they were able to make their own decisions about research direction, data analysis, etc. during the course of their authentic research experience.
New students involved in the research should focus on being creative with their individual flare analysis! As long as the decisions are scientifically justifiable, students have the freedom to attempt different methods from their peers for flare analysis. There are a lot of different possibilities with coding and flare analysis, as well as the decision to choose any flare within a specific range.