Impacts on student learning, confidence, and affect and in a remote, large-enrollment, course-based undergraduate research experience in physics

Over the last decade, course-based undergraduate research experiences (CUREs) have been recognized as a way to improve undergraduate science, technology, engineering, and mathematics education by engaging students in authentic discovery. CUREs have been shown to have positive benefits similar to traditional undergraduate research experiences; however, they can reach a larger number of students and are open to all students who enroll in the course. Motivated by the need to redesign the large introductory physics lab at The University of Colorado – Boulder to be fully remote in response to the COVID-19 pandemic, we designed and implemented the first remote, large-enrollment, physics CURE. Here, we detail the motivations and the challenges when designing the course, and provide detailed descriptions of the course components. Throughout the course, we collected course artifacts and administered surveys to the students. Based on these data sources, we find that this course helped students gain research skills and coding confidence, engage in productive and enjoyable teamwork experiences, and feel motivated and interested in experimental physics research.

However, traditional models of undergraduate research experiences can come with substantial barriers [16][17][18]. For example, limited research positions funded by individual faculty members can create a competitive application process that may exclude students with lower course grades, and even a well-funded summer research program risks excluding students with other responsibilities over the summer, or who are simply unaware of the existence or importance of such research experiences [2,19].
One model proposed to address these challenges is the "course-based undergraduate research experience" (CURE).
A CURE is a formal course that engages an entire class of students in a research question that is of genuine interest to the scientific community [1,8]. The CURE model has gained popularity in certain fields in part because it may promote equity in ways traditional undergraduate research experiences cannot, since CUREs are open to everyone who can enroll in the course [2].
A key feature of a CURE is that it engages students in authentic discovery: the outcome of an investigation is unknown to the students, instructors, and scientific community. CUREs also make use of authentic scientific practices, they build on, and contribute to, current scientific knowledge, they feature collaboration and teamwork, and generally involve some form of iteration.
There has been a lot of work in the last 10 years to define, implement, and study the impact of CUREs on students [1]. Currently, the majority of CUREs described in the literature are centered in chemistry or biology [20,21], and often take place in upper-division courses [20]. There has been work [22] to transform large, introductory labs into a CURE in fields such as biology and chemistry, but even in those fields most CUREs described in publications are fairly small (< 100 students) [23]. Reported instances in physics [24,25] have also been small [26] or have not featured work with high relevance to the current scientific literature [27]. In this work, we detail the implementation of the first large (> 400 students per semester for the course), introductorylevel physics CURE and discuss some of the key findings on the students experience.
The decision to redesign the introductory physics lab at University of Colorado (CU) Boulder was motivated by the need to teach remotely during the COVID-19 pandemic, which began in Spring 2020. In the remote course, we sought to teach essential experimental research skills, foster productive teamwork, and build a unique and motivating learning experience for the students. After weighing our key motivations and working constraints (see Sec. III), we developed and implemented a CURE in which students would analyze the energy distribution of solar flares in order to collectively answer an important open question in solar physics (see details in Sec. III C).
The course was divided into six phases, designed to track the structure and process of a typical physics research project. First, students were onboarded with skills and background knowledge by participating in teamwork training, completing an introductory programming packet, and conducting a literature review. Second, students developed a research plan and practiced their plan using test data from a previously studied solar flare. Third, students chose their own flares from an open data source and conducted a data analysis with their team. Fourth, the data analysis write ups were deidentified and sent to other teams for peer review, and the analyses were then revised by the original teams, based on the reviews received. Fifth, students combined all of their individual flare data together to perform the final analysis and draw conclusions. Last, students individually summarized their findings in a "Flare Archive Entry" and reflected on their classroom and research experiences. Details of the course design are given in Sec. IV.
Based on students' responses to reflection questions (which were required after each lab), their pre-and postcourse survey responses, and a final reflection written in the form of a memo to future researchers, we found overwhelmingly positive outcomes from the course. In this paper, we highlight the findings that addressed with our original motivating factors: authentic research practices and skills, productive teamwork, and student enjoyment of the course.
The goals of this work are twofold: 1. To detail the design and structure of the class and, 2. to summarize the most important, initial findings. Together, these results will inform future design, instruction, and research of physics CUREs. Given the positive outcomes we found, we encourage the physics community to continue to explore ways to implement CUREs in their curricula that have long-term sustainability. In addition, this work will be followed by subsequent research providing detailed analysis of each of the three major outcomes described here.
This paper is organized as follows: in Sec. II, we provide background on the current state of CUREs and introductory physics labs. In Sec. III, we present our process in developing the course, with a focus on our primary motivations and the constraints we faced. Section IV presents a detailed overview of the course structure as experienced by the students, and Sec. V, we describe the methods we used to evaluate the success of the course and the results obtained with respect to our three primary outcome goals. Section VI discusses the implications for large-scale physics CUREs more generally and directions for future research.

II. BACKGROUND
First, we examine the relevant background on CURE development and research, as well as the recent calls (and responses) to transform physics labs at the undergraduate level. In addition, we take a brief look at how physics labs have been impacted by the COVID-19 pandemic and the strategies some instructors used to conduct labs remotely.

A. Course-based undergraduate research experiences
There have been calls for an increase in course-based undergraduate research experiences (CUREs) in response to numerous studies showing the overwhelmingly positive benefits of undergraduate research experiences and the increased inclusivity and accessibility of the course-based research format [1][2][3][4][5][6][7][8]19]. CUREs are unique learning environments, which differentiate themselves from both "traditional" and "inquiry-based" labs because the discoveries that are made in the course are initially unknown to both students and instructors and have value to the broader scientific community [1].
The CUREnet (Course-Based Undergraduate Research Experiences Network) was initiated in 2012 with funding from the National Science Foundation in order to address topics, problems, and opportunities inherent to integrating research experiences into undergraduate courses [1]. In 2017, they published a meeting report to provide a working definition of a CURE [1]. They identified some key components that make up a CURE: 1. Engaging in multiple scientific practices.-There are many scientific practices essential to research such as asking questions, building and evaluating models, designing studies, using the tools of science, gathering and analyzing data, interpreting results, and communicating findings. Although students in a CURE are not necessarily expected to engage in all of these practices, it is essential that they engage in more than 1 (e.g., not only data collection). 2. Scientific discovery.-One of the unique aspects of a CURE is that students make authentic discoveries where the results of their work are initially unknown to both the students and instructors. In addition, the research of a CURE should be of interest to stakeholders in the broader scientific community. Because of this, CUREs present opportunities for impact and action beyond the classroom (e.g., authorship or acknowledgment in a scientific publication).

3.
Collaboration.-Group work has many benefits and is recognized as an important skill within many scientific disciplines [28][29][30][31][32][33]. Development of collaboration skills is often cited as an explicit goal of lab instruction [29,34]. In addition, group work is an essential pedagogical tool in a CURE because it exposes students to an authentic element of science: science is inherently a collective endeavor that requires many minds to tackle a problem. 4. Iteration.-Iteration is fundamental to the scientific process [35]. In a CURE, students' work is iterative, meaning that students must troubleshoot, problem solve, and repeat aspects of their work in order for the research to progress. CURES may also offer students opportunities to review each other's work and to revise their work based on such feedback. There has been a significant number of CUREs implemented at the undergraduate level, particularly in biology and chemistry. Much of the research about CUREs has been done on large, multisite programs such as the SEA-PHAGES Program [36], the Small World Initiative [37], and the Genomics Education Partnership [38]. However, there has been an increasing amount of research exploring smaller scale CUREs, including some in the field of physics [24,39]. Most of the published works assess student achievement of specific learning goals (e.g., ability to pipette) or use student self-reported gains from surveys such as the CURE survey [40] or PITS [10] to comment on the effectiveness of the experience. However, there have been few studies that look at how specific elements of a CURE impact students' achievements [1]. In addition, there have been no publications on the implementation of an introductory physics CURE at a large scale (i.e., over 100 students).

B. Introductory physics lab courses
Similar to the calls to incorporate more CUREs into the undergraduate curriculum, there has also been a push to transform physics labs to better meet the goals outlined by the 2014 AAPT Recommendations for the Undergraduate Physics Laboratory Curriculum [29]. In the last several years, there has been a number of published introductory physics lab transformations including, but not limited to lab transformations at Cornell University [41], Investigative Science Learning Environment (ISLE) [42], DATA [43], and an introductory lab transformation at CU Boulder [44].
The introductory calculus-based physics lab series at Cornell University was transformed under five main learning goals: "(i) Collect data and revise an experimental procedure iteratively and reflectively, (ii) evaluate the process and outcomes of an experiment quantitatively and qualitatively, (iii) extend the scope of an investigation whether or not results come out as expected, (iv) communicate the process and outcomes of an experiment, and (v) conduct an experiment collaboratively and ethically [41]." Resulting research has shown that students in the Cornell University physics labs have more expertlike views of experimental physics than the traditional, concept-focused labs [45] and have shifted from a model-verifying mindset to better engage in authentic experimentation [46]. ISLE, started at Rutgers University, helps students learn physics by engaging activities that mirror what professional physicists do: observing, finding patterns, building and testing explanations of the patterns, and using multiple representations to reason about physical phenomena [42]. ISLE is a comprehensive learning system that can be used in lectures, recitations, and labs or some combination of these in a new, nontraditional format. ISLE instruction has now been adopted across many institutions and has been shown to help students navigate novel experimental tasks (in physics and biology) and build scientific habits of mind [47].
DATA, a two-course sequence of algebra-based physics laboratories at Michigan State University called the design, analysis, tools and apprenticeship (DATA) lab "removes physics-specific content from the overall learning goals of the course and, instead, uses physics concepts to focus on specific laboratory practices and research skills that students can take into their future careers [43]." The course is focused on students gains in understanding of experimental process, data analysis, collaboration, and scientific communication [43].
Last, the transformed introductory physics laboratory course at CU Boulder-which we will discuss in more detail in Sec. III, as the immediate precursor to the remote CURE described here-was designed with the intent to better develop students' views surrounding measurement uncertainty, and to foster a more expert epistemology about experimental physics. This was done through activities often involving the explicit use of measurement uncertainty concepts to make predictions, compare data among student groups, and draw conclusions. This course was shown to successfully increase students' expertlike epistemology surrounding experimental physics [48] and to increase students' understanding of measurement uncertainty [49] compared to the nontransformed version.
While many traditional physics labs are designed primarily to reinforce theoretical physics concepts, evidence suggests that they are not generally successful in this regard [50][51][52]. Instead, these transformed courses have in common a focus on introducing or reinforcing concepts about experimental science and engaging students in experimental practices. For example, in a light refraction lab from the CU-Boulder transformed course, students determine which of their groups has the same concentration of sugar in an aqueous solution. The concentration of sugar in a particular aqueous solution is not a "known" value (unlike, say, the gravitational acceleration g) so the emphasis is on using experimental measurement, scientific reasoning, and measurement uncertainty to draw a practical conclusion, rather than students trying to match their results to a predetermined "correct" answer.
These types of labs "offer students glimpses of what it means to do experimental physics," which may be a more valuable goal for a laboratory course [53]. However, since students are surely aware that "identifying the concentration of a sugar solution" is not typically an important activity in real-world experimental research, the experimental work they are glimpsing is at best an imitation. Hence, the CURE model potentially provides an additional step forward by offering not merely a constructed model of experimental science, but rather an authentic instance of discovery and research.

C. Introductory physics labs during the pandemic
In addition to the increasing number of planned lab transformations in the past few years, the COVID-19 pandemic in Spring 2020 caused nearly all departments to adapt or redesign their curricula as colleges and universities moved to emergency remote instruction. Many students were suddenly forced to leave campus and faced numerous challenges in addition to health concerns for themselves and their families. Similarly, instructors faced overwhelming personal and professional challenges with little time to plan new class activities [54][55][56].
This transition presented particular challenges for laboratory courses, which typically rely on hands-on activities and group work. In a previous study conducted by our group on emergency remote-lab instruction, we found that (i) many instructors changed their learning goals of the courses to be more focused on reinforcing concepts and (ii) instructors tended to reduce group work due to equity and technological concerns [55]. However, some instructors used this as an opportunity to redesign their courses. For example, in a case study presented by Hoehn et al. [57], an advanced lab course completely redefined its learning goals and transitioned from traditional prescriptive labs to more open-ended projects [57]. The study found that the open-ended projects afforded students opportunities to make decisions and think deeply about their experiments, which students report as contributing to their enjoyment and satisfaction with the course [57]. Still, in this remote environment, students had mixed group work experiences, with some describing positive and meaningful interactions and others describing group work as a source of frustration and stress [57].
In the same manner, our CURE was motivated by both the need to move away from activities based in physical equipment and a belief that the disruption represented an opportunity to attempt a more ambitious course redesign. However, this context also placed significant constraints on the course development process, which we now discuss in detail.

III. COURSE DEVELOPMENT
As we developed our CURE in the summer of 2020, the ongoing COVID-19 pandemic required us to carefully consider what learning outcomes were most important for the students in our lab course, and how they could be pursued most effectively within the online learning environment. It is important to note that our goals explicitly did not include reducing the effort or time put in by the instructors. The course was designed by two faculty members and a postdoctoral researcher who was dedicated full time to the project throughout the summer and fall of 2020. Our course was designed around the goals of teaching "skills" (rather than physics concepts), emphasizing the importance of group work in science, and providing a unique and motivating experience to students. At the same time, the design was constrained by our limited preparation timeline, the large class size, limited teaching experience of the teaching assistants, appropriate physics research questions, and concerns about accessibility and mental health for students in the online environment. Ultimately, these goals and constraints determined the course structure, the research question at the heart of our CURE, and the technical methods used by students in their analysis.
A. Learning goals

Learning skills remotely
Physics education research literature generally categorizes learning goals for labs into two general categories: developing experimental skills or reinforcing physics concepts [50,52,58]. While there exists a wide range of learning goals for labs that do not fall into these two categories (for example, fostering students' understanding of, and appreciation for, the nature of science), many labs tend to focus on lab skills, physics concepts, or a mixture of both. During the emergency transition to remote instruction, there was a shift among instructors to emphasizing physics content rather than lab skills in the remote setting [55], particularly amongst instructors who tried previously to teach both concepts and skills in their lab. However, the large first-year lab at CU Boulder (PHYS-1140), which serves approximately 500 engineering and physical science majors each semester, had already been recently been transformed to emphasize learning skills. This one-credit course, which was not directly connected to a lecture course, had no goals that involved reinforcing physics concepts; instead, the course focused on goals that were unique to lab environments, such as developing scientific practices and expertlike views of experiential physics, with a particular focus on measurement and uncertainty [44].
In redesigning the course for remote instruction, we strove to retain its focus on goals unique to lab environments. However, in the absence of a physical lab space, the exact nature of these goals needed to change. Instead of focusing on measurement and uncertainty, we decided to develop experimental research skills, such as reading published literature, working with code, developing a research plan, performing data analysis, and engaging in peer review.

Group work
During Spring 2020, many physics laboratory instructors also transitioned from primarily group work to primarily individual work [55]. In our group's prior survey, the most common motivation given for this change toward individual work was equity concerns [55]. Because of the sudden nature of the transition, many instructors could not, or did not want to, require students to attend labs synchronously [55]. However, in recent studies, it has been suggested that the COVID-19 pandemic has put college students at a higher risk for loneliness during the pandemic than usual [59] and led to an increase in worry and grief [60] among students. In a study published in March 2020, Thomas et al. found that fostering a sense of community can significantly lower these levels of loneliness among incoming college students [61], but group work in the online classroom can pose an array of new challenges to students and instructors alike [62]. Nonetheless, group work can have overwhelming benefits, including increased motivation, creativity, and reflection [63], and development of teamwork and collaboration skills is often emphasized as important for lab work and professionalization [29,32]. For example, the AAPT Recommendations for the Undergraduate Physics Laboratory Curriculum [29] suggests that one of the goals for students in physics labs should be to develop "interpersonal communication skills" through "teamwork and collaboration." Between the potential benefits for the students during the semester and the value for their future careers, we decided to make "teamwork" a major focus of the remote version of the lab.

Developing a unique and motivating experience
An explicit goal of the pre-COVID version of our lab course [44] was that students should have a positive attitude about the course and about experimental physics in general. These goals are challenging to achieve in physics labs even outside of remote instruction [64,65]; however, we were motivated by evidence suggesting that a CURE might be particularly well suited to achieving them. For example, it has been shown that engaging in authentic practices, like those of a CURE, may be a critical part of improving students' beliefs around the nature of experimental physics [66]. Furthermore, a study by Hanauer et al. produced a model of the psychological outcomes of CUREs that showed improvement in student self-efficacy, science identity, understanding of scientific community values, and networking, all of which resulted in increased persistence in the sciences [10]. Given these factors, we believed a physics CURE would provide a unique and motivating experience for students during a trying time, and could ultimately achieve some of our key learning goals by providing an authentic and positive experience in experimental physics.

B. Constraints
As we pursued these objectives, there were many constraints that influenced the development of the CURE, including the limited course development time, the large class size, access to relevant expertise, accessibility for students, and the fully online teaching environment.

Time
Although we had two faculty members and a postdoc focused on the course design, development time itself remained major constraint in choosing the project. The entire curriculum development was completed between May and August in 2020, while the two faculty members were also engaged in teaching and research. Typically, transformations like these would take multiple years to develop, test with students, and compare to previous semesters [44]. The course development process included finding a research partner and a research question for the project, determining if the research was accessible to introductory level students, planning the course schedule or structure, and creating all the course materials. Running the course itself required approximately the same amount of time and effort from the instructors and TAs as past non-CURE versions of this course. However, the research question for this CURE cannot be used for more than a few semesters before it needs to be modified. Therefore, we would expect that, long term, this type of CURE would be significantly more work for the instructional team since they would need to redesign the materials every few years.

Large class size
Physics-1140 enrolls approximately 400 to 700 undergraduates each semester who are primarily second-year students interested in engineering (see Table I). One of the primary challenges we faced was being able to coordinate a CURE in such a large class with limited instructional staff (two course instructors and 20 TAs). Unlike previous physics CUREs and authentic project-based labs [24,25], we could not manage students having substantially different projects from one another. There needed to be clear grading expectations and consistent procedures across all the groups each week. Still, a key aspect to a CURE is providing students with a sense of ownership over their experience. We found inspiration from large biology CUREs such as the Phage Genomics project [36], where introductory biology students isolate novel bacteriophage from the environment and characterize them using electron microscopy and DNA analysis. Each pair of students has their own unique sample to analyze, but they follow the same analysis procedure as all of their peers. Likewise, we hoped to find a large dataset in which each student team could analyze their own unique data using relatively consistent, introductory-level data analysis techniques. Unlike many CUREs, we were less constrained by the need for long-term sustainability of the class, because our course was always intended as a temporary solution until pandemic conditions allowed a return to in-person instruction. However, the dataset ultimately needed to be large enough so that this class could be run for three semesters (1200-1800 students in total) while classes remained remote. In addition, like the Phage project, it was essential that the student analyses could not be trivially performed by computers or machine learning algorithms, to ensure that the students were making a genuinely valuable contribution to scientific knowledge that could not have been easily obtained without them.

Expertise
We faced two major challenges of expertise when designing the course: (i) we had never designed a CURE before and (ii) our areas of physics research did not immediately provide a research question which would be best solved by 1200-1600 undergraduate students. Given our compressed timeline, we needed to tap in to our local experts to help with both of these obstacles.
We set up meetings with professors at CU Boulder who had run CUREs previously in biology and astrophysics and who had been leaders in CURE education research. These meetings helped us better understand the key features of CUREs (see Sec. II) and the technical details of managing a large number of students and TAs while conducting real research. In addition, we connected with researchers at the Laboratory for Atmospheric and Space Physics (LASP) at CU Boulder for insights into possible research questions, in part because we knew their work often involved large open-source datasets. We were introduced to Dr. James Mason, a research scientist at LASP, who specializes in solar measurement satellites and solar physics. After a few meetings, we were able to identify a research question to answer that would be mutually beneficial to Dr. Mason and to the students in the Physics-1140 class: within the (scaffolded) capabilities of our students, and yet not something that could easily be addressed without their help. For more details on the research see Sec. III C.

Accessibility
Another key piece of our course design was ensuring that the needed technology was accessible to all students. Based on the research question chosen, we knew that coding in PYTHON was going to be a substantial component of this class. To help make code compilation available to all students, even those accessing the class via tablets, chromebooks, and phones, we needed to provide an outside server. In addition, we wanted to limit the amount of downloading and configuration needed to set up the coding environment, and needed to ensure access and compatibility across platforms. Given the time constraints and lack of budget, we decided to use Google Colaboratory [67] (Google Colab). Google Colab allows students to write and execute PYTHON in their browser, with no configuration required, provides free access to Google's GPUs, and enables easy sharing amongst classmates. The Colab notebook environment lets students write and execute code and markdown files similar to a Juypter Notebook [68]. Students were able to access all the PYTHON libraries needed in order to perform the data analysis for this class. Nevertheless, while it ultimately proved to be the best available solution for our course, we found that Google Colab still had some downsides; for example, it is difficult to save the Colab notebook as a PDF and the collaboration aspect is not as seamless as, for example, Google Docs. As a result, there are major version control issues if students work on the same document at the same time.
When the course began, we administered a survey at the beginning of the semester and determined that all TAs and students registered for the course had access to internet, a device to attend synchronous Zoom meetings, and Google products like Colab. Consequently, we were able to proceed confident in the fact that students could all participate meaningfully in the data analysis and the teamwork aspects of their remote lab course.

Online teaching
A final major constraint was the need to teach the course completely online. To assess the particular impacts of this constraint, we began by asking students about their past experiences with the online environment in spring 2020 in a precourse survey. The question stated: "Remote classes have their own unique advantages and challenges. Our goal is to help you learn as much as possible in this class. How can we help you be successful in a remote environment?" Two common themes we saw from the students were (i) consistent communication and (ii) clear expectations. One student wrote, "I believe that consistent communication will enable me to succeed in this class. In addition to that, having professors and TAs who I can talk to sometimes when I have questions will help me get the most out of this class." Another said, "Being clear on the due dates of assignments, as well as providing multiple clear and accessible opportunities for help when needed outside of class would be invaluable in a remote learning environment, as well as understanding and leniency towards technical difficulties." The latter concern was particularly challenging for this course, since each week had very different types of assignments given the evolving nature of research.
To address both the need for consistent communications and clear expectations, we developed a Canvas page for the course that provided easy coordination, explicit weekly checklists, and multiple locations for feedback throughout the semester.
The Canvas page was designed such that the course activities for each week could be found on an individual weekly "page." These pages were linked to the homepage of the course via a button (see Fig. 1). Each week's page was divided into clear sections: first a checklist for the week, then links to the asynchronous lecture, then materials needed for the lab, and finally, assignment due dates and links to the assignment submission pages (see Fig. 2). After the course, when asked to "Please rank your agreement with the following statements, as compared to other classes you took this semester," 63% of students reported that this course was "more" or "much more" organized.
In general, we sought to give our Canvas page a "professional" and "intentional" appearance, consistent with the idea that students were engaged in a real, professional research project. Interactive asynchronous lecture videos were structured similarly to the prelab videos used in the pre-pandemic version of the course [69], and, in general, embedded content on the course page was designed consistent with best practices for multimedia instructional materials, in order to maximize student comprehension and positively address the affective domain [70].
Finally, we sought to provide multiple opportunities for feedback including weekly postlab reflections, an anonymous feedback link, and numerous office hours. Beyond reading the feedback, we would summarize the main points from the students and address the feedback during the lectures.

C. The research
The Colorado PHysics Laboratory Academic Research Effort (C-PhLARE) is a project seeking to help answer a long-standing question in solar physics: What mechanisms are behind the coronal heating? The temperature of the Sun's corona is millions of kelvin greater than that of the photosphere, despite being much further from the center of the sun [71]. Several theories have been proposed to explain the coronal heating and it seems likely that the corona is heated by a multiple interrelated mechanisms [71]. However, it is still possible to narrow the list of contributing mechanisms, and perhaps identify which contributions dominate. For example: is the coronal heating primarily caused by many small flares (nanoflares) occurring all over the sun? Or is it dominated by a different process like magnetohydrodynamic waves? One way of answering this question is by looking at the slope of the solar flare frequency distribution (occurrence rate versus energy) [72][73][74]. A steeper slope would imply that small flares are more common-these small flares may not produce much energy, but if they occur with enough frequency, their energy in aggregate may be a substantial mechanism behind coronal heating. The flare frequency distribution (FFD) is given by a power law, where n is the number of events, E is the radiated flare energy, A is an overall constant, and α is the critical exponent. In the most common presentation of these data on a log-log plot, A becomes an offset constant, and α appears as the slope. It has been shown in a study by Hudson et al. [72] that if α < 2, then nanoflares are not frequent enough to be the main mechanism behind the coronal heating. Therefore, determining an accurate α value is critical to addressing our research question. In addition, it is key to examine how α might change depending on the solar cycle or the time within a solar cycle (i.e., solar maximum versus solar minimum). There have been some past studies calculating this α value [73,[75][76][77][78], but most have been within other solar cycles (i.e., not solar cycle 24, from December 2008 through December 2019, which we are studying), did not look at data across the entire solar cycle, did not cover as large a range of solar flare energies (i.e., from Cto X-class flares), and/or calculated α using far fewer flares than our students were able to collectively analyze.
One of the challenges of doing this type of calculation is that each flare analyzed needs specialized treatmentbeginning and ending times of the flare need to be chosen and appropriate baseline correction needs to be appliedbefore the total energy of the flare can be determined. These decisions, although relatively simple for humans, are quite difficult for automated computer algorithms. For example, in a related algorithm proposed for analyzing coronal mass ejections, 70% of candidate events could not be automatically analyzed because a pre-flare baseline could not be established [79].
The problem is therefore ideally suited for our largeenrollment introductory-level CURE. Working in small teams, a large class like ours could identify and determine the total energy of hundreds of individual flares, which collectively give a picture of how frequently various energy levels occur. Each individual flare analysis is straightforward enough that, with some scaffolding, it is within the capabilities of a student with some introductory physics and calculus knowledge to perform the analysis and form a meaningful understanding of the work they have done. On the other hand, because these analyses could not easily have been obtained by an automated system, students can recognize that their work represents a meaningful contribution to scientific knowledge that would not exist without them. Indeed, with the help of 1200-1800 students, we are able to determine α from a wide range of solar flares and examine potential variations in α between solar minimum and maximum for a much larger dataset than would otherwise be possible [73,[75][76][77].

D. Team formation
Throughout the course, students worked in teams of three or four, which remained fixed (barring drops, etc.) from week to week. Because so much of the course involved group work, our team formation process was particularly intentional. Students were asked a variety of questions in a precourse survey, including comfort using video on Zoom, prior coding experience, coding confidence, time as a student at CU, declared major, gender identity, and time zone. All of these factors influenced team creation; however, there were two particularly significant factors considered: gender identity and coding confidence.
First, we avoided groups in which only one student did not identify as a man. This choice was based on prior group dynamic research that found women had lower performance and poorer social cohesion in male-dominated groups [80] and that a gendered division of roles may be more likely to develop spontaneously in unstructured physics lab like ours compared to traditional labs [81].
Second, we tried to form teams whose members reported a similar level of coding confidence. Because of a concatenation of factors-the introductory level of the course, challenges with remote collaboration, and constraints using Google Colab, we believed it was important that team members had similar coding experience, so that one student with more experience would not dominate the others.

E. Assignment structure and grading
Grades for the course were based on both group and individual work. "Individual" assignments were required to be written up by each student alone. The author of the assignment was allowed to discuss ideas and questions with instructors, TAs, and teammates; however, the final product needed to be their own work. On the other hand, "team" assignments were required to be written up collectively by the assigned lab team. The team members were told they should contribute equally to the assignment and that if they did not contribute substantially they should not expect to be listed as an "author" on the assignment submission. Each member of the team was required to submit an identical copy of the assignment to receive credit.
Overall, the class was split fairly evenly between team and individual assignments with individual assignments comprising 51% of the points and team assignments accounting for 49%.

IV. COURSE SCHEDULE
The PHYS-1140 CURE is a 15 week course in which students engage in an authentic solar physics research project. In addition to 1 h and 50 min labs held weekly, synchronously over Zoom [82], there were 7 asynchronous, approximately 15 min, prelab lecture videos. These lecture videos were contained embedded questions via PlayPosit [83] that were worth credit for students to answer. The lectures welcomed students into the course and prepared students for key labs such as developing the analysis plan, peer review, and calculating the alpha value from the whole class data. Messaging throughout these lectures was intentional and consistently reinforced the authenticity of the research project. For example, before the students develop their analysis plan the lecture video discusses "general process for grant funding" and how research proposals are written for funding agencies such as the Department of Defense, the National Science Foundation, NASA, and the Department of Energy. These grant templates are then directly compared to the template provided for the student analysis plan explaining the importance of each component (e.g., background and significance or broader impacts).
Here, we provide further details about the six phases of the synchronous lab meetings that students attended through video conferencing tools. Some example course materials can be found in the Appendices.

A. Project on-boarding
The first three weeks of class were dedicated to acquiring skills and foundational knowledge important for this research project. The first week was focused on teamwork training. Students were broken into breakout rooms on Zoom and assigned a teamwork "scenario" to read and discuss. The scenarios were based on challenges that may arise when engaging in teamwork in general or specifically in the remote environment. The scenarios were also created to show different student perspectives, see Appendix A.
In the following week, students were assigned to their semester-long teams (details on how the teams were assigned can be found in Sec. III D) and engaged in a literature review. Students were asked to watch two videos on the design and launch of the GOES-P (later called GOES-15) satellite, which collected the data that they used for the research in this course. In addition, students had four reading assignments: a note from the principal investigator, excerpts from the full technical document describing the data recorded by the GOES satellite [84], a graph of the "flare frequency distribution [85]" with a brief description (an example of the flare frequency distribution plot showing the class data is shown in Fig. 3), and excerpts from a 2012 paper by Aschwanden and Freeland [75], which describes some previous research similar to this project (in the later two semesters, this was replaced by excerpts from a paper by Shibata et al. [85] as we adapted the readings to our observations of the students' level of comprehension).
Students then discussed guiding questions (see Appendix B) with their team during the lab. In the end, they created a list with jargon terms and their definitions and three remaining questions they still had after the readings for the principal investigator. The instructors read through all of the questions students submitted, grouped them into common themes, and chose exemplary questions to ask the principal investigator during a live, synchronous research meeting for the whole class in the subsequent week; examples of those questions and answers are shown in Appendix C.
During the following week, students began working on an independent assignment (see Sec. III E for more details on independent versus team assignments) to develop or reinforce their PYTHON coding skills. This assignment introduced students to programming in the .ipynb environment through Google Colab. The main topics covered progressed from making a markdown cell and running a coding cell, to using arrays and dataframes, then plotting and log-log plotting, and finally basic fitting as relevant to the research needs of the project. In each section, the students were not expected to generate novel code from scratch. Instead, they were presented with example code and asked to explain its function, and then asked to make some modification to the details of code to put their understanding into practice (for example, adjusting the binning of a histogram to better suit the data presented). The overall arc of the assignment used an example problem of computing a frequency distribution of the sizes of craters on the moon-a task specifically designed to use many of the same concepts and data analysis tools as the solar flare research topic, but in a simpler context that did not yet require novel scientific research.
The full "coding packet" assignment is available as Google Colab notebook (.ipynb file) in the Supplemental Material [86]. See Sec. V B 2 for a brief discussion of the philosophy behind this approach as it relates to the explicit learning goals of the course.
Thus, although students were participating in this project as a course in which they had enrolled, this phase was meant to provide students with the authentic experience of joining a research group: learning the group norms and meeting the other members of the lab, reviewing the relevant literature, meeting with the PI, and learning the data analysis tools used by the group.

B. Research plan development
The following three weeks were spent on developing a research plan and practicing the steps with a test flare. Students began by gathering the irradiance data of a large X-class flare that occurred on September 10th, 2017 from the Space Weather Data Portal [87] (each team worked with the same test flare at this stage of the analysis). Students worked with their team in Google Colab to calculate and plot the energy versus time of the flare from the irradiance data. The next week, students developed a written research plan as a team. To give students a more authentic experience of the process of developing a research plan, the template for the research plan assignment was modified from the National Science Foundation Proposal Outline. It contained four sections: (i) Overview, goal, and objectives. Students were asked to "succinctly and specifically state the question your plan will address. Briefly discuss how your proposed research will help build, and/or expand scientific knowledge." (ii) Background and significance. Students were asked to "Briefly overview the background of this research by discussing the current state of existing knowledge and identifying the gaps that the project intends to fill." (iii) Analysis plan. Students were asked to "Describe the work you will do to meet the objectives set forth in the first section. Include clear statements of the research activities to be undertaken, including methods and procedures (e.g., include any mathematical methods, formulas or PYTHON functions you will be using) and how you will process and analyze the data. If there are obstacles you expect to face, state them clearly and describe how you will address them. Include any plans for future collaboration with other teams." (iv) Broader impacts. Students were asked to "Discuss the broader impacts of the proposed research. These can include possible impacts to society and/or development of skills and knowledge of the team members." Finally, students implemented their analysis plans to test their procedure for baseline correction and total energy calculation by applying them to the "test" flare from September 10th.
A central component of these three weeks was iteration. As described in Sec. II, it is key that students repeat aspects of their work during a CURE so that they can troubleshoot, problem solve, and progress the research. Allowing students to practice their analysis plan using a test flare showed the iterative processes of science and allowed them to get feedback on their methodology before engaging in the actual analysis. This process is not only helpful for student learning, but also for maintaining the integrity of the research.

C. Data analysis
In the data analysis phase, teams spent two weeks analyzing their own flares for the research project. Teams were assigned a flare class (X class, M class, or C class) and asked to find a flare of that class on the Space Weather Data Portal from within the relevant time period to analyze for the project. Teams were required to analyze at least one flare, but were encouraged to analyze additional flares if time permitted. These additional flare analyses were not for credit, but students were told that "our collaboration needs as many flares as possible to be analyzed to increase the size of the dataset and improve our determination of the power law." Ultimately, the majority of teams analyzed one or two additional flares. After the analysis was completed in these two weeks, the Colab Notebooks were deidentified and set out to individuals from other teams for peer review.

D. Peer review
Confidential peer review is an essential part of authentic scientific research. After teams determined the total energy and produced a flare analysis in a Colab Notebook, they engaged in the same double-confidential peer review process that many experts use to provide constructive criticism of a scientist's work and ensure its validity. Students were instructed, "when writing your review, consider what type of feedback you would like to receive. For example, it helps to receive feedback which is very specific, acknowledges both the parts that are well done, as well as the ones that need some changes, and of course, is written in a professional and respectful tone." Following the model of many peer reviews in the field of physics, our peer review had five main sections: (i) Overview. Summarize the methods used by the authors whose work you are reviewing. Include information about the flare, describe the baseline correction in detail, describe the method of integration, include relevant PYTHON functions used, and report the total energy that was calculated. (ii) Merits. Describe the merits of the work. What was done well? Is it clear what was done? Do they explain and justify their approach? (iii) Critiques. Describe places that could be improved and provide suggestions for improvement. Be specific and provide actionable advice. (iv) Overall recommendation. During the scientific peer review process, the reviewer is typically asked to make an "overall recommendation" on whether a paper or report should be published. Here, we ask you to finish by doing the same thing, using the following scale: No revisions are needed, needs minor revisions, needs major revisions. (v) Conclusions. In a few sentences, summarize your review and briefly justify what overall justification you choose.
In both the merits and critiques sections, students were asked to consider a series of questions shown in Table IV in  Appendix D. After a team received feedback on their work in the form of a peer review, they were given an opportunity to revise their work and resubmit it. Based on the authentic revision and resubmission practices common in the field of physics, the team was required to provide a cover letter along with their revised work that demonstrated that they read the peer reviews carefully, outlined the revisions they made in response to the reviewers comments, and/or defended their reasons for not making the changes the reviewers suggested. Students then submitted their final values for the flare total energy and the flare peak irradiance to the collaboration for collective analysis.

E. Calculating alpha
The final phase of the research project involved combining the class data and determining α for the dataset [see Eq. (1)]. This step was done over two weeks. First, students used a database with all the recorded solar flare peak irradiances for the past 20 years to find a relationship between peak irradiance and flare frequency, which follows the relationship where n is the number of events, I p is the peak irradiance of the flare, and in log-log plotting, B is the offset constant, and β is the slope. From this equation for the frequency of flares per irradiance dn=dI p , we obtain an equation for the frequency of flares per total energy dn=dE via the chain rule: Teams calculated dn=dI p for themselves, and the instructors calculated a value of dI p =dE for the students from the larger dataset. In the second week, students combined these values to calculate the energy of the flares as a function of frequency and then fit the plot to determine an α value.

F. Documentation and reflection
The final week of class featured the second, and last, synchronous meeting with the principal investigator. The principal investigator summarized the class findings and put them in the context of previous studies. In addition, the principal investigator posed new questions such as "does the value of α vary depending on solar minimum or solar maximum?" to motivate the continuation of the research in future semesters. Following the meeting, students were assigned two independent assignments-a "flare archive entry," which documented all the important information about the flare they analyzed and a "memo to future researchers," an informal letter discussing their experience and giving recommendations to the students continuing the project the following semester. We found that in this framing, in which students were explicitly encouraged to write for an audience of students who would be joining the project by taking the course in the subsequent semester, elicited particularly deep and thoughtful insight about the experience of the course, the biggest challenges they faced, the skills that they gained, and the keys to success.
At the conclusion of the course, students were reminded that their results would ultimately be written up and submitted for publication, and given information about how to opt-in to authorship and track the progress of the paper once the data gathering was complete.

V. COURSE OUTCOMES
In evaluating the impact of the course, we focus on the three categories of desired outcomes that motivated the course design: namely, that students should learn skills used in authentic experimental research (including becoming more comfortable with coding as a tool), engage in teamwork and recognize its importance, and feel motivated by, and interested in, the experimental research process. This work provides an overview of our analysis of the course; we intend to follow this work with subsequent research providing further details of each of the three major outcomes. For each of these, we analyzed our outcomes based on some or all of the four types of data sources described in Sec. VA.

A. Research methodology
For our findings discussed in this paper, we rely on a mixed-methods approach based on four data sources: (i) closed responses from the precourse survey, (ii) closed and open responses from weekly reflection questions, (iii) quotes from the memo to future researchers assignment, and (iv) closed responses from the postcourse survey, which was modified from the PITS [10] and URSSA [88]. All closed responses from both surveys and reflection questions were asked on a Likert scale. We present the students' responses to the Likert-scale questions as a percentage and the uncertainty is calculated using a 95% binomial confidence interval. The open responses were coded using a thematic coding scheme to find common trends in student responses. We used an a priori codebook containing seven codes: affect, authenticity, coding, community, identity, learning, and teamwork, which reflected our motivations for the course. Specific quotes were chosen based on their exemplary nature to highlight trends seen within these themes. All data presented here is drawn from the first semester in which the course was taught (Fall 2020). Further analysis including the subsequent two semesters will be presented in future work.

Authentic research practices
Throughout the course, students engaged in many authentic practices of research, such as peer review and writing a research proposal. The personal gains questions, a subset of questions from URSSA [88], assess student confidence, comfort, and general self-efficacy with conducting research and working on a research team and in a lab [88]. It has been reported that these gains are related to students' readiness to take on the role of scientist or science professional [88]. In Fig. 4, we see that the majority of students self-reported "moderate," "good," or "great" gains from this course for their confidence in their ability to contribute to science, comfort discussing scientific concepts, care in conducting research procedures, confidence in their ability to do well in future science courses, patience with the slow pace of research, and understanding of what everyday research is like. These self-reported gains from this course are only slightly lower to the gains seen in traditional undergraduate research experiences (non-CUREs); see Table II [88,89]. Likewise, at the end of the semester, students were asked to rank their agreement on a scale of 1 (none) to 5 (a great deal) to the statement: "During your research experience in this course how much did you engage in real-world science research?" Students' mean response was 3.94 AE 0.05 with a standard deviation of 0.90.
Students' belief that they developed meaningful research skills and engaged in an authentic scientific research experience are further echoed in the memos to future researchers. One student wrote, This lab was honestly one of the most interesting STEM classes I've had the pleasure of taking. I've worked in two separate research labs and I still managed to learn a lot about scientific processes. I've even used much of the knowledge and skills I developed in class within my position as undergraduate research assistant.
This response emphasizes that, even though this was a large, introductory lab course taught entirely online, the student felt that they learned similar research skills to a traditional undergraduate research experience. Another student expressed that this course was particularly interesting because of the authenticity of the work: …the thing that makes this course interesting is the fact that you're working on actual data that was collected from real-life instruments in space. Therefore, there will always be new data to analyze and the more data we collect, the more accurate results and conclusions we can arrive to. Participating in this research gives you a sense of your contribution to science and how are actual science researches conducted.
This student says that they learned "how are actual science researches conducted" echoing the findings seen in Fig. 4. They also imply that the authenticity of this experience gives you "a sense of your contribution to science," in other words, that the research conducted in the class positively contributed to scientific knowledge. A majority of the students (68.6%) felt similarly-that this research helped solve a problem in the world and was important to the scientific community shown in Fig. 5. In addition, most students felt that they were responsible for the outcomes of the research (81.4%), that their findings were important to the scientific community (80.4%), and the research itself gave them a sense of personal achievement (63.6%) (Fig. 5).
We believe consistent messaging (i.e., using the word "research" instead of "project" in all of the course materials) and the steps taken to consistently engage students in authentic practices contributed to these beliefs.
For example, in response to the reflection questions during the peer-review processes, the vast majority of the class found the peer-review processes to be helpful for their research and an authentic scientific practice. The vast majority students agreed that peer review is an essential part of the scientific process (98.7%), that the act of peer reviewing others helped guide their own research (92.0%), and that the peer-review comments provided useful information to help them revise their analysis (88.8%) shown in Fig. 6.
Overall, these elements combined together to create a large, course-based research experience that has only slightly lower gains in learning and beliefs surrounding research practices of a traditional undergraduate research TABLE II. Student reported gains of research skills, confidence, and understanding from the URSSA survey [88] with 1 = no gain; 2 = a little gain; 3 = moderate gains; 4 = good gain; 5 = great gain. The Phys-1140 CURE is compared to student responses from two previous studies of students in traditional undergraduate research experiences.
Items. How much did you gain in the following areas as a result of your most recent research experience? experience (Table II). As one student wrote, addressing the next semester's students in their memo, Overall, remember that this class is kind of a "How to Science" class. The most important things you learn aren't how a solar flare works or how to write code. It's how to conduct good and meaningful science. It's how to ask scientific questions and work collaboratively. Just remember that this class is here to help you become a better scientist and to set you up for success in any scientific or engineering discipline that you may pursue.

Coding skills
In addition to the research skills already discussed, students' experiences with coding during the course deserve special analysis because work with PYTHON code featured so prominently in their week-to-week assignments. Although this was explicitly not a programming course, and while learning the details of PYTHON coding itself was not a learning goal, we did have a goal to familiarize students more generally with how to use coding as a tool for research. We drew inspiration for this distinction from our observation that many active FIG. 5. Student responses to the postcourse survey questions measuring attitudes toward their research experience. Students responded to a 5-point Likert scale from "strongly disagree" to "strongly agree." The responses presented in the graph have been collapsed to a 3-point scale. The uncertainty was calculated using the 95% binomial confidence interval with n ¼ 404.
FIG. 6. Students ranked agreement with statements about peer review following their own peer reviews and their corrections based on the reviews. Students responded to a 5-point Likert scale from strongly disagree to strongly agree. The responses presented in the graph have been collapsed to a 3-point scale. The uncertainty was calculated using the 95% binomial confidence interval with n ¼ 398.
researchers find themselves using programming languages in which they have little-to-no formal training, but which they have picked up by tinkering with existing code and drawing from online documentation as needed. Hence, while we never expected the students to write a complete program from scratch, our goal was that by the end of the course, they would be able to read, understand, and modify code examples, look up and apply basic functions and libraries, and document their code well enough that it could be easily understood by another researcher.
There was a significant increase in coding confidence gains after completing the coding packet assignment in the third week of class, see Table III. This was particularly true for the students who had no confidence in coding prior to the course. Additional analysis shows that of the students who marked "no confidence" in coding prior to the course 9.0 AE 3.0% were still "not at all confident" after the coding packet, 59.6 AE 7.7% moved to feel "somewhat confident" and 28.8 AE 4.7% reported feeling "confident" after the coding packet.
Without prompting, 66 out of 426 students discussed learning coding in their memo to future researchers. Below are a few exemplary quotes from students: Personally, I am not an expert with the coding language but have obtained a tremendous amount of knowledge on how and when it would be beneficial to utilize. I had never worked with PYTHON before and I felt that this course allowed me to complete work in PYTHON while simultaneously teaching me as I went.
Using skills you'll learn by coding in Google Colab, which uses the coding language PYTHON, you can write blocks of code that read, analyze, and display different aspects of the data.
These three students focus on the utility of coding in PYTHON. Although none of them claim to now be expert coders, the first student says that they have a better understanding of "how and when" using programming would be useful in scientific research. The second student felt that they learned by doing-which is common among professional scientists and researchers. And the last student emphasizes some of the useful tools that coding in Google Colab provides. Echoing these ideas, albeit in a more critical tone, one student wrote, Most of the coding you'll be doing is from looking at examples of functioning code from the weekly guide, and adjusting it for your specific data. This streamlines the research process, but it also (unfortunately) doesn't help much with learning PYTHON itself. This class will give you a very basic understanding of PYTHON, but you'll have to work on your own time to figure out how to write code on your own.
Although this student was disappointed that coding in PYTHON was not more explicitly taught in the course, they highlighted aspects which exactly reflected the actual course goal: "looking at examples of functioning code… and adjusting it for your specific data" so that it "streamlines the research process."

C. Teamwork
Teamwork and collaboration was another major area of emphasis in the course. The first week of class featured teamwork training to both set the tone-emphasizing the importance of teamwork in this lab-and provide a space for students to think of responses to common teamwork challenges before they occurred. Overall, students found the training to be helpful. Out of 387 students who responded to the reflection questions that week, 15% said the training was "very helpful," 32% responded "somewhat helpful," 32% responded "moderately helpful," 11% said it was "slightly helpful," and only 7% said the training was "not at all helpful." In the reflections after the teamwork training, students were asked to "Describe one strategy that you learned today that would help you during future team collaborations." Students discussed a variety of strategies, but a common theme was communication. One student wrote: Communicating openly and honestly among the group members is one of the key norms we established during the lab. It helps us all in the group to be on the same page and work in the most productive manner.
We believe that this emphasis on the importance of teamwork in the beginning of the course set a framework for regulating common teamwork challenges students might have faced along the way. Overall, students reported that they overwhelmingly enjoyed the teamwork aspects of this course with over 80% agreement to all the postcourse TABLE III. Student self-reported "coding confidence" before beginning the course and after completing the coding packet assignment in week 4 of the course. The uncertainty was calculated using the 95% binomial confidence interval. survey teamwork related statements, see Fig. 7, and felt that they were able to contribute to their team's success. When asked at the end of the semester in the final reflection question, "Overall, how effectively did your team work together on this project?" 88.4 AE 9.6% of students in the course reported that it was "very" or "extremely" effective.
In the memos, many students reflected on teamwork in the course and how it was helpful for learning. A few students expressed that working as a team helped them better understand physics concepts: When others on your team get stuck or are confused about something you can help them out which will further improve your own understanding of the concepts and formulas. Working in teams served as a great tool to better my understanding and knowledge of physics and engineering concepts and real-world application.
However, far more students discussed the impact working as a team had on learning coding skills. One student wrote, I had terrific teammates who worked together with me to learn how to effectively use PYTHON and it made the process easier than I could have imagined.
Seemingly, this student found that working as a team made learning and completing the work easier.
Lastly, many students discussed learning collaboration skills through their teamwork experience.
I think the main goal of this research for us is to learn how to collaborate. The coding isn't crazy hard, and for most of us, this research doesn't have much use in the future, but the skills you learn working together over zoom and with classmates was essential to the success of the group.
This student reflected that they did not learn much about concepts or coding, but found the collaboration skills learned in this course both helpful and practical for future success. These kinds of sentiments were commonly seen in their reflections, since most students in the class were not physics or astrophysics majors (Table I). Other students reflected on having to learn how to collaborate with students who worked differently than they did: I also learned how to coordinate work with other people that have different expectations of what level of work is good work. Also how to work with others that may or may not want to work together or work hard. I'm also an astrophysics major where all my partners were engineering majors. It was interesting to see how I approached the project versus how they approached it. I think this will definitely inform how I interact with engineers in my career.
Working with others can be challenging, especially if they have different standards of work, as discussed in the first student's quote, or different ways of thinking, as in the second quote. However, navigating these challenges is an important skill valued in many fields [32]. FIG. 7. Students ranked agreement with statements on teamwork at the end of the course. Students responded to a 5-point Likert scale from strongly disagree to strongly agree. The responses presented in the graph have been collapsed to a 3-point scale. The uncertainty was calculated using the 95% binomial confidence interval with n ¼ 404.
Overall, students in the course seemed to overwhelmingly value their teamwork experience and felt that it helped them succeed in this course.

D. Unique and motivating experience
Throughout the course, we hoped to develop a unique and motivating experience for the students such that they would leave with a positive attitude about the course and about experimental physics in general. To this end, we asked students a series of questions about affect in the postcourse survey to measure student enjoyment, engagement, and participation in the course. A majority of students agreed that they found the research project exciting (56.4%), enjoyed the class (57.8%), and found the research project interesting (72.5%), see Fig. 8.
Furthermore, students strongly recommended the course to future students in the memos-particularly because they felt that they were able to contribute to the scientific community: The results that will be generated from the analysis will be most likely be published in a scientific journal. All the students who will be participating in this project will be considered co-authors of the published scientific journal. Therefore, this is a very interesting and beneficial project that is worth undertaking. In addition, the project will be on a virtual online lab and not a traditional lab. Personally, I really enjoyed this class. I was expecting a typical physics lab however, I was very excited to learn I would be partaking in an actual scientifical [sic] research on flares.
Other students mentioned that the teamwork aspect of the course was both motivating and fun, This class was very fun and engaging for two main reasons. First, it feels like you are doing "actual science," doing useful work rather than repeating a simple experiment that has been done a million times. Secondly, it lends itself very well to teamwork and collaboration.
A student in the class expressed that this course was an opportunity to make friends who they hope to meet in person: I went into this class not knowing anyone as an international student and finished this class with three great friends that I hope to meet one day after this pandemic is over and I am back in the U.S.
The ability to collaborate might have been particularly refreshing in this course when teamwork was often not emphasized in other online courses: Being able to collaborate with new people was definitely a nice change of pace given the current state of the world, and I would have taken what we did in this class over performing small physics experiments in Duane [the physics department building] any day. In addition, some students wrote that they hoped courses like these continue in the future, I hope projects like these don't go away once covid is over because I believe that this research project was a better experience than the normal physics lab ever could be. If the projects do continue then why stop at solar flares, I think experimental physics should be a course where you get a guided introduction into any physics research projects.
Responses like these demonstrated that students saw the value in having a course which taught experimental physics research practices.

VI. FUTURE IMPLICATIONS FOR INSTRUCTION AND RESEARCH
The findings presented in this work show that our primary motivating goals were achieved to a substantial level: 1. Students gained research skills-students selfreported gains in research skills in this course were similar to traditional undergraduate research experiences, and reported gaining confidence and experience in using coding as a tool for research. 2. Students had productive and enjoyable teamwork experiences-students reported overwhelmingly positive teamwork experiences in the class and voiced that teamwork helped them learn. 3. Students valued the course and found the research interesting-students reported that they enjoyed the course and many expressed that they hoped a course like this would continue in the future. As the first large-scale, introductory, online physics CURE, this course may serve as a road map for developing and implementing courses like these in the future. We plan to continue our research to more deeply understand in what ways did students view their research experience to be authentic, how students engaged in teamwork, changes in interest in scientific research, and which elements of the course led to these successes. In addition, we note that this course was conducted entirely during a global pandemic, which greatly disrupted the educational system. Future work is needed to understand how the pandemic environment impacted students perceptions of the course; what additional challenges and difficulties due to the pandemic did students overcome in order to participate in this course? Many of these additional challenges may be tied to the online nature of course. However, other external factors such as increased food insecurity, financial hardships, a lack of social connectedness and sense of belonging, uncertainty about the future, and access issues that impede their academic performance and well being-often disproportionately impacting students of color and low-income students-may have lead to obstacles related specifically to the pandemic era [93].
Large-scale remote CUREs, regardless of field, may open more opportunities for nontraditional students and students with disabilities to have access to research experiences [2]. Remote CUREs could be very beneficial, especially if they are able to have similar gains to in-person undergraduate research experiences as shown in this work. In addition, there have been no previously reported cases of large-scale physics CUREs. Increasing CUREs in physics could help provide physics research opportunities to a broader range of students.
However, one overarching challenge in designing a CURE for a large-enrollment introductory physics course is sustainability. As stated in our course motivations, longterm sustainability of this course was not a priority in our design because it was created to fill a temporary need for an online laboratory course. If this course were to continue, we would quickly reach the point where additional flare data would not be expected to change the overall results, and the relevance of the research to the scientific community would quickly diminish. New research questions and, therefore, new course materials, would need to be created every few years-a major burden on faculty and departments. Similarly, finding research questions in physics that authentically require the support of hundreds of students, but have relatively simple data analysis is likely to be difficult. However, given the positive outcomes demonstrated in course, we encourage the physics community to continue to explore ways to implement CUREs in their curricula.

ACKNOWLEDGMENTS
We thank Dr. James Mason for leading the C-PhLARE research project as well as all the students in the course who took the time to share their experiences with us. In addition, we would like to thank Lisa Corwin, Don Woodraska, Courtney Peck, Allison Youngblood, John Keller, Pam Harvey, Kristin Oliver, and the CU Boulder Labs PER group for their vital contributions. The data collected by students was accessed via the University of Colorado's Space Weather Technology, Research, and Education Center's Space Weather Data Portal which relies on GOES data that are processed at the NOAA Space Weather Prediction Center and the NOAA National Centers for Environmental Information provided by NCEI. This work is supported by STROBE National Science Foundation Science Technology Center, Grant No. DMR-1548924.

APPENDIX A: EXAMPLE TEAMWORK TRAINING SCENARIOS
In the following scenario the protagonist might have been viewed as a "social loafer" [94] from their teammate, but the root of the problem stemmed from the two lab partners struggling to reconcile the differences in their prior coding knowledge: Scenario 1: You have recently started a lab, but your lab partner seems much more experienced than you. They go ahead and get started working on the project quickly while you struggle to even load the data. You asked them to explain what they did and they go on to tell you some complicated technique that they used to complete the lab. When they were finished with the explanation, they asked if you had any questions. You didn't want to tell them that you were struggling with loading the data so you just said no. At the end of the semester, your lab partner ended up doing most of the work for the project and you both got a good grade. Your lab partner never complained and seemed happy doing the work themselves, but you felt like you weren't able to work collaboratively nor to learn.
Likewise, in this second scenario, it is common to feel frustration when one or more teammates do not have the same standards for work: Scenario 2: You have just spent the last month working with your lab partner. You both get along well and have been working productively. For the final project, you decide to split the work. You decide to make all graphs and your partner says they will write the descriptions of the results. You both work for over a week and then come back together to go over the final product. On Monday morning, they hand you their draft. After reading through the draft, you are forced to conclude that they do not understand what was done in the figures you made and it needs to be completely rewritten.

APPENDIX B: LITERATURE REVIEW GUIDING QUESTIONS
Students were prompted with guiding questions for the readings: 1. What are the motivating factors behind this research project? 2. What questions do we hope to answer with this research? 3. What is unique about this project that differentiates it from past research? 4. For each graph you encounter, consider the axes.
What are they showing? 5. What pieces of "jargon" do you encounter in these readings? (Jargon is special words or expressions used by particular groups that can be difficult for others to understand) 6. What other questions or points of confusion do you have about the project? Make a specific list so you can discuss with your TA and labmates.

APPENDIX C: RESEARCH MEETING WITH PRINCIPAL INVESTIGATOR
Below are three examples of questions students asked and the responses from the principal investigator during the meeting. These examples represent the three most common categories of student questions, which were (i) broader impact, (ii) "why us?", and (iii) technical questions: Question: How will the research we conduct affect the scientific community and future research? Response: For one, the whole way we have this set up may be a model for other universities to create a class to do something similar, regardless of the scientific field of study. The science here will let us see how the sun compares to other stars and to other sources of flares (like the violent accretion disks around black holes). We have an idea of where the sun and our results should fit in, but the question is: does it? If yes, that's good! It tells us that our ideas about how energy builds up and suddenly gets released are probably on the right track. If no, that's even better. Isaac Asimov: "The most exciting phrase to hear in science, the one that heralds new discoveries, is not "Eureka!" (I found it!) but "That's funny …" Question: Why are undergraduate students participating in this research rather than graduate students or professional researchers? Response: Most scientists do start their research careers as undergrads. They usually start out on projects like this-things that are tricky for computers to do alone but are somewhat well defined and have a clear end result. As time goes on, start asking questions that have murkier paths to an answer -Ph.D. students spend 3 or 4 years trying to answer just one question usually. These kinds of projects are total win-win situations: undergrads (you) get an introduction to the science topic, learn how to use and develop new tools to answer questions, get a taste of real research to see if it's something you'd like to peruse career-wise, how many scientists learn to program for the first time. Professionals (me) get help doing the work to answer a question that would be impossible to do on our own. Question: How do you tell the difference between a small flare pointed towards us and a large one pointed away? Response: It's nearly impossible to detect flares that occur on the opposite side of the sun without a satellite over there to see them (we've actually flown some out there: see STEREO, Parker Solar Probe, and Solar Orbiter). You can see some of the light reflected back to us from the distant side of the solar system "the heliopause"-our bubble around the sun. IV. Questions students were prompted to consider when engaging in the peer-review process for the course.

Q1.
Was the subclass of the flare reported correctly? Q2.
Was the date and time of the flare peak reported correctly? Q3.
Did the flare have potential contributions from other flares? Q4.
Was this accounted for by the authors in their report? Q5.
Was the approach taken clearly explained? Q6.
Did the authors thoroughly describe the approach they were taking? Q7.
Are the details clear, and the steps justified? Q8.
Was the baseline correction reasonable? Q9.
Was the baseline biased by other flares that occurred during the pre-flare signal? Q10.
Did the authors use enough information to make a reasonable estimate of the baseline? Q11.
Was the method of integration reasonable? Q12.
Were the limits of integration correct? Q13.
Did the authors integrate with respect to time, rather than the default index units? Q14.
Were the units correct? Q16.
Did they report the final total energy in ergs? Q17.
Was the Colab notebook clear? Q18.
Were comments and markdown descriptions used to describe their process? Q19.
Were there sufficient figures and plots in order to understand their process and the appropriateness of the analysis?