Undergraduate student experiences in remote lab courses during the COVID-19 pandemic

The 2020 – 2021 academic year was a unique time for many instructors who had to adapt their courses to be conducted remotely due to the COVID-19 pandemic. This was especially challenging for physics lab courses, which usually emphasize hands-on experiments. Although many courses have now returned to in-person teaching, the possibility remains of future disasters necessitating similar remote courses. It is important to understand how undergraduate students experienced remote physics lab courses during the pandemic, including what aspects of the courses contributed to positive student outcomes. To investigate this, we surveyed over 5000 students from 24 different institutions, asking how the students engaged with their physics lab courses during the 2020 – 2021 academic year. Here, we describe the frequency with which the students performed various class activities, aspects of the course environment, challenges the students faced, aspects of the courses the students found enjoyable, and some student outcomes. We further study the impact of the course activities and course environment on four of the outcomes (self-reported learning of lab skills, self-reported learning of concepts, course enjoyment, and development of a sense of community). We find that students who were provided clear expectations, had enough time for their coursework, frequently worked in groups, and frequently had access to guidance from their instructors were more likely to report positive outcomes. This work demonstrates the importance of certain aspects of lab courses for several desirable outcomes in remote lab courses during a pandemic, with findings that may transfer to in-person or remote lab courses in the future.


I. INTRODUCTION
From March 2020 to summer 2021, many colleges and universities were forced to conduct courses remotely. Instructors had to quickly alter the format of their previously in-person courses, which was often a large challenge, particularly for those teaching lab courses due to the common use of hands-on experiments [1,2]. Although at the start of the COVID-19 pandemic, the switch to remote classes was sudden, by fall 2020, many instructors had several months to prepare for teaching remote courses. Courses taught remotely during the 2020-2021 academic year were therefore different both from the emergency remote teaching in the spring of 2020 and from courses initially structured to be remote or online [3]. We investigate remote lab courses during the 2020-2021 academic year, where we define remote labs as courses that were labs prior to the pandemic and that were conducted during this time with all students and the instructor not in the same physical location as each other.
Although many courses have now returned to in-person teaching, there are still reasons to investigate what lessons can be learned from the remote teaching during the pandemic. Not only can these lessons contribute to future long term studies of the effect of the pandemic, but there exists the possibility of institutions needing to return to remote instruction for extended periods of time due to future outbreaks of COVID-19 or other unforeseen circumstances, such as man-made or natural disasters (e.g., wildfires and hurricanes). After all of the remote teaching caused by the COVID-19 pandemic, infrastructure is already in place at many institutions to teach courses remotely. It is important for instructors to know what has been effective in similar situations to optimize student outcomes during any possible transitions back to remote teaching.
Additionally, lessons from labs during the pandemic may help improve current lab courses, whether initially designed to be remote or in-person. The amount of planned online courses has been growing in the decades preceding the pandemic, with benefits such as increasing the accessibility of science education [4,5]. Many people believe that colleges and universities will not be the same after the pandemic, now that people have seen that remote education is a possibility [6]. Online courses have the potential to improve accessibility of physics labs to those with disabilities inhibiting them from participating in traditional labs, those with dependents or other responsibilities outside of the course who need a more flexible schedule, and those wanting to experience physics labs in locations where lab equipment is not available [7,8]. During the 2020-2021 academic year, many more instructors than in previous years had to think about creative ways to engage their students with remote lab activities, often finding cheap and widely available options. Lessons from that year may prove useful in answering the important questions of which student outcomes can be successfully achieved in remote lab courses and how to optimize those outcomes. There are also similarities between the remote lab courses and their in-person counterparts, so some of the findings from studying remote labs may carry over into in-person labs. In particular, our results suggest that the features that made the remote labs the most successful are features of the lab courses that do not depend on the labs being remote.
The goals of this work are twofold: to document student experiences in remote lab courses during the 2020-2021 academic year and to extract possible lessons for the future based on the reported student experiences. We use the term experiences to broadly encompass anything that happened to students in their courses, from what they did to how they felt about it. First, we detail what happened in remote physics labs by examining the frequency of course activities, the course environment, and student affective and learning outcomes. We focus on this time period because future remote instruction would be most similar to times during the pandemic in which the instructors already had some experience with remote teaching. Second, we investigate the impact of the ways the students participated in their courses on four student outcomes: self-reported learning of concepts, self-reported learning of lab skills, enjoyment of the course, and development of community. Although there are many kinds of learning that occur in lab courses, concepts versus lab skills is a coarse-gained delineation that has been useful in prior research characterizing lab courses [9,10]. Additionally, students' enjoyment of their courses and sense of community are important outcomes because they can contribute to students' persistence to remain in the field [11,12]. To address these goals, we ask the following two research questions: RQ1: What were student experiences (e.g., activities performed and student affect) in remote lab courses during the 2020-2021 academic year? RQ2: Which activities or parts of the course environment had an impact on whether or not the students reported learning concepts, learning lab skills, enjoying the course, and developing a sense of community? In order to answer these research questions, we analyzed student responses to questions appended to the Colorado Learning Attitudes about Science Survey for Experimental Physics (E-CLASS) [13]. During the 2020-2021 academic year, over 5000 students from 24 different institutions responded to these additional closed-response questions. We first present the survey results in a descriptive manner and then perform logistic regressions on the data. The regressions model how the student outcomes depend on student experiences with the course activities and course environment, as demonstrated by their answers to the closed-response questions. Although there were many external factors affecting the students while they participated in their courses (e.g., studying at home and general stress caused by the pandemic), we focus on the impact of course-related factors that could inform future instruction.
This paper is organized as follows. Section II discusses prior studies on remote lab experiences, with a focus on the remote lab courses taught during the COVID-19 pandemic, in order to situate our work within the field. We then discuss our methods for data collection and analysis in Sec. III. In Sec. IV, we answer RQ1 and present the results of the entire survey divided up into course modality, course activities, course environment, and student outcomes. We further analyze the data in Sec. V by presenting logistic regression models demonstrating the impact of the course activities and environment on four outcomes, answering RQ2. We further discuss these results in Sec. VI, bringing together the themes we see in the data as lessons instructors can take away from this work and directions for future research. We summarize and conclude our work in Sec. VII.

II. REMOTE LAB COURSES
Since the start of the COVID-19 pandemic, there have been many new studies about students' and instructors' experiences with remote teaching (for example, Refs. [2,). These studies cover a wide range of topics, including changes in students' conceptual learning gains after the switch to remote teaching [14][15][16], student motivation and attitudes towards physics in remote lecture courses [17], and recommendations for how instructors can optimize their remote lecture courses [18][19][20][21]. Teaching remote lab courses has also been an area of interest [2,[21][22][23][24][25][26][27][28][29][30][31][32][33][34][35], due to the difficulty of conducting hands-on experiments in a remote format. The rest of this section focuses on previous studies of remote lab courses, both prior to, and during, the pandemic. We separate them into these sections because there is a fundamental difference between well-developed remote labs that were intended to be conducted in that format and lab courses during the pandemic that were only conducted remotely due to necessity [3].

A. Prepandemic remote labs
Even before the COVID-19 pandemic, there was an effort to determine ways to offer remote physics labs for the growing number of students enrolled in online courses. Many approaches to remote labs have been taken including using simulations, performing experiments at home, remote-controlling experiments, and having students view pre-recorded videos of experiments [7]. Remote labs may be beneficial for students with certain disabilities or with other commitments that do not allow them to attend regular lab sessions [8]. They additionally reduce the equipment cost for the institutions, reduce the risk of accidents, and allow more flexibility for the students [39,40]. Here, we are not advocating for removing in-person labs, which offer some kinds of student learning not available through remote options, but only pointing out the benefits offered by remote labs.
There has been a long-standing discussion in science education about whether hands-on lab activities are necessary or if the same learning goals as for in-person labs can be accomplished with simulations or virtual labs [41][42][43][44][45]. In physics classes in particular, studies have shown that students' conceptual understanding can be at least as good when hands-on lab experiments are replaced with either simulations or virtual labs [46][47][48][49], and that students' epistemologies and views of help seeking are similar between remote and in-person labs [50]. Other studies have focused on the unique affordances offered by inperson or virtual labs. For example, students perform different experimentation strategies when using simulations compared with hands-on experiments [51], and virtual reality labs offer new possibilities for modeling and experimental design [52]. Most of these studies demonstrate that remote courses (or parts of courses being done virtually) can be effective for many desirable outcomes when planned out properly. Nevertheless, some lab goals, such as gaining experience with safely operating specialized lab equipment, can only be achieved by working with the equipment itself, as occurs in in-person labs. More research is needed to fully understand the different learning opportunities offered by remote versus in-person labs.

B. Remote labs during the pandemic
Many papers since March 2020 describe ways instructors changed their physics lab courses during the transition to remote teaching, providing examples for other instructors of possible ways to conduct remote lab courses. One approach was to enable students to perform experiments at home, for example, by mailing the students experiments kits [22,23], choosing experiments that use only common household materials [24], or using Interactive Online Lab (iOLab) [25]. Other instructors adjusted the lab equipment so it could be controlled by the students remotely [26]. Several instructors also used the change to remote learning as an opportunity to emphasize inquiry-driven lab activities and projects [27][28][29], and observed that the conversion from traditional labs to open-ended projects led to increased student agency and enjoyment of the course [30]. Other work compared student learning and interest in different methods of remote labs within a single course [31] and compared social connectedness and self-efficacy between students participating in remote versus in-person labs at a single institution [32].
Other studies have surveyed multiple different institutions in order to understand the range of ways courses adapted to being remote. A precursor to the current work investigated both instructor and student perspectives from many different courses and institutions in the spring of 2020 with a focus on the varied ways instructors adapted their labs for the remote format [33]. Other research focused on the instructor perspective of the transition to remote teaching, describing instructors' perceived successes and challenges [2], as well as the resources the instructors used and their anxiety towards, and perceptions of, how well they taught online [34].
Some surveys have focused on the student experience, looking for trends that were common across many courses. An analysis of student E-CLASS scores found that students in the fall of 2020 had similar scores to in previous years, indicating there was no net effect on student views about experimental physics due to the remote modality of the lab, although there were slight improvements in specific survey items [35]. A study of five universities in Austria, Croatia, and Germany showed that students were more likely to report they had learned concepts and skills in lab courses where they gathered data themselves (e.g., from videos), instead of being provided data or simulating it [21]. A recent American Institute of Physics survey showed that the majority of physics departments held both remote and inperson labs during fall 2020 and that women were more likely than men to be less confident in their ability to perform well in labs during the pandemic compared with before [53]. Unlike in the other lab studies surveying multiple institutions, in this work we focus on the entirety of the 2020-2021 academic year and investigate the experiences of over 5000 students. This work provides a bigger picture view of student experiences in remote lab courses during the pandemic, allowing us to investigate a variety of student outcomes and find similarities across courses from different types of institutions.

III. METHODS
In our study, we examine closed-response questions from a survey investigating students' experiences in remote lab courses. In this section, we discuss the creation of the remote lab survey questions, the data cleaning process that allowed us to use the data for logistic regressions, the demographics of the student population studied, an explanation of our analysis methods, and limitations and ethical considerations of this work. Additional details about our methodology are included in the Supplemental Material [54].

A. Survey design and dissemination
Our data come from remote lab survey questions appended to the end of the E-CLASS. The E-CLASS is an online assessment administered as pre-and post-tests in lab courses via Qualtrics [13]. This survey examines changes in students' views about experimental physics after participating in a lab course. During the 2020-2021 academic year, instructors administering this survey were provided the opportunity to include extra questions at the end of the post-test related to how the students engaged with their course. All of these additional questions were optional for the students. A first version of these remote lab survey questions was added to the E-CLASS in the spring of 2020, and the questions were then edited to allow us to better interpret students' responses in fall 2020. We use this second version of the survey in this analysis. This final version for the 2020-2021 academic year can be found in the Supplemental Material [54].
The remote lab survey questions include a variety of types of questions with different response scales. The survey begins with a set of questions asking how the students participated in the lab portion of the class (e.g., remotely, in person, a hybrid, etc.) and whether or not the students had a choice in that modality. The next set of questions asks the students how often they performed a variety of activities in the lab portion of their class, with frequency scale responses of never, rarely, sometimes, often, and always. Examples include how often the students used simulations or completed lab activities as a group. Some of the frequency scale questions have follow-up questions with possible responses of yes or no, which are given to the students if they did not respond never to the frequency scale question. For example, a student who used simulations with any frequency (i.e., who responded rarely, sometimes, often, or always) would then be asked whether or not they created their own simulations.
Other sets of questions investigate student affect and other course outcomes. One set of questions asks the students to rank their agreement with a variety of statements to which they can respond with a five-point Likert scale of strongly disagree, disagree, neither agree nor disagree, agree, or strongly agree. For example, one of these questions asks whether the students enjoyed their lab course. The last set of questions ask the students what they enjoyed and found challenging about their course. These consist of lists of options where the students can select as many options as they want.
All of the questions in the survey ask about the experiences of individual students, not the course as a whole. For some of the courses, we saw variation across student responses within a single course, even for the questions about the frequency of activities. This also includes responses to the question about course modality, as there may have been situations where a single student in a course needed to be remote (e.g., because they were immunocompromised) even though all other students in the course attended in person. It is unsurprising that students within the same course may have had different experiences from one another, especially during a pandemic. We chose to use the student perspectives for all of our analyses in order to validate the student perspective. Nonetheless, the survey questions investigate variables that could be impacted by the instructors' choices, so the results can inform future instruction.

B. Data cleaning
We went through an extensive data-cleaning process due to the large number of remote lab survey questions as well as some incomplete responses from the students. Because our research questions examine remote lab courses, we first retain only the students who participated in their lab courses entirely remotely. For RQ1, we describe what happened in the courses from the students' perspectives, so we present the results as answered by the students on the closed-response survey. However, for RQ2 (the regressions), without any data cleaning, we would have too many variables, some of which are highly correlated with each other. In order to use the data for a regression, we combine similar questions and collapse response scales. We describe here the essence of the approach to provide suitable background for the reader to interpret the results, with the full details given in the Supplemental Material [54].
To begin the data cleaning process, we divided the questions into what we call "inputs" and "outputs" and considered only the survey questions that were intended for use in the regression models. The questions investigating how students interacted with the course, including how often they performed different activities and what the course environment was like, are used as inputs to the regressions. All of the inputs relate to aspects of the courses that the instructors were able to influence to some degree. Other questions ask about outcomes of the course, and we use four common course objectives (learning of concepts, learning of lab skills, course enjoyment, and development of community) as outputs for the regression models. Other outcomes, such as whether the students participating in group work found it to be productive and the list of challenges the students encountered, are more specific and were intended to be purely descriptive because the question format is not suitable for regressions.
There still remained some questions that were highly correlated with each other, so we combined questions with similar meanings into a single question. Some of these came from frequency scale questions and their follow-up binary response questions. These include questions about how students engaged with watching videos of experiments and performing simulations. We also combined the frequency scale questions about students designing their own experiment and designing an experimental procedure because the majority of the students gave the same response for the two questions. The exact details of how we combined these questions are provided in the Supplemental Material [54]. After this process, we were left with 15 input variables.
We additionally collapsed some response categories for the regressions. We combined agree with strongly agree, disagree with strongly disagree, rarely with sometimes, and always with often. For the rest of this paper, we use the phrase "with low frequency" to encompass responses of both rarely and sometimes and "with high frequency" to encompass responses of always and often. In order to refer to the four responses other than never, we use the term "with any frequency." Once we finalized the questions to use as regression variables, we retained only the students who responded to all of those questions (see Table I for exact numbers). This is a necessary step because incomplete responses cannot be used in a regression. We chose this method of listwise deletion because it is a standard practice and only 5% of the students were missing a response to at least one of the relevant variables, so imputing the data would not have had a significant impact on the regression analysis [55]. To ensure we were looking at the same number of students throughout the different analyses in this paper, we used only the students who responded to all of the regression variable questions for the results presented in both Secs. IV and V.

C. Students and courses in the dataset
In all of our analyses, we divide up the students enrolled in first-year (FY) and beyond-first-year (BFY) courses. Prior research has shown that these courses are distinct from one another, often with different course goals, kinds of activities, and student majors [10,56,57]. In Sec. IV, we present the results for both FY and BFY courses to see similar trends between the two without applying statistical tests, since our goal is not to compare them. In Sec. V, we present results for only FY courses because there are too few students in BFY courses to obtain sufficient statistical power.
Our final dataset contains 4565 students participating remotely in FY courses and 250 students participating remotely in BFY courses. The large difference is primarily due to the relative size of the courses, with FY courses generally being much larger. Part of the difference is also due to BFY courses being more likely to be taught in person (see Sec. IVA). Table II shows the demographic information of the students.
These students were enrolled in 46 FY courses at 23 institutions and 15 BFY courses at 12 institutions. Information about these institutions is provided in Table III. Some of the institutions offered multiple sections of the same course, either in the same or different terms throughout the academic year. Taking that into account, the students come from 34 unique FY courses and 12 unique BFY courses. The data for FY and BFY courses are each TABLE I. Total number of students who responded to (i) at least one E-CLASS question ("some E-CLASS"), (ii) at least one of the remote lab survey questions ("some remote questions"), (iii) all of the remote lab survey questions that we use as inputs and outputs for our regression models ("complete responses"), and (iv) all of the remote lab survey questions that we use as inputs and outputs for our regression models and who participated in their course entirely remotely. Each of these categories is a subset of the previous ones and the number of students is divided into students in first-year (FY) and beyond-first-year (BFY) courses.  [58] and students were allowed to select multiple options, thus allowing columns' sums to exceed 100%. All demographic questions were optional for the students, leading to some columns adding up to less than 100%. For the student majors, we grouped together responses of similar majors. We combined physics, engineering physics, and astrophysics into one category. The other science category includes chemistry, biochemistry, biology, astronomy, geology or geophysics, math or applied math, computer science, physiology, and other science. We additionally grouped together nonscience major with other option or undeclared. dominated by a single course from the same large research institution; 17% of the students in FY courses were enrolled in the largest unique FY course, and 34% of the students in BFY courses were enrolled in the largest unique BFY course. Removing these courses does not substantially qualitatively change the data or the claims in this paper. The exact numbers in Sec. IV change slightly, especially for BFY courses, and a couple of additional variables gain significance in the regressions when the largest course is removed, as discussed in Sec. V.

D. Statistical analyses
In order to answer RQ1, we present the values reported by the students along with 95% confidence intervals. The confidence intervals are either binomial or multinomial, depending on the response scale of the question. We use the collapsed response scales for the multinomial data as that is how we discuss them. We do not apply further statistical analyses since the goal is to see the general trends.
To answer RQ2, we perform one-vs-rest logistic regressions because the research question asks which parts of the course had an impact on whether or not the students reported the different possible student outcomes. The one-vs-rest binary logistic regressions allow us to compare students who agreed to an outcome with all the other students (those who were neutral or disagreed), as well as those who disagreed with all other students (those who were neutral or agreed). We present a set of odds ratios with their corresponding p values along with 95% confidence intervals. For categorical independent variables, the odds ratios give the ratio of the odds of the outcome occurring (e.g., a student agreeing that they enjoyed their course) for students with a particular response (e.g., always or often) to the independent variable compared with students with a different response to that variable (e.g., never). The comparison response is referred to as a reference category.
In each of the regressions, there are a large number of variables, so we account for the multiple comparisons by correcting the p values. We use models with all 16 input variables because our research question is broad and we want to minimize the bias coming from omitted variables [59]. We present models without interaction terms because all of the correlations are small [60] and our initial models including interactions indicated overfitting. There are still many different statistical tests in each regression, so we apply the Holm-Bonferroni method [61] to correct the p values for each model independently. Although there is a debate about when and how to correct for multiple comparisons [62], we chose this method as a way to balance retaining sufficient statistical power and limiting false positives. We use a significance threshold of 0.05 and present statistical significance for both the corrected and uncorrected p values.

E. Limitations and ethical considerations
The main limitation of this study is our sample size and the way our sample may not be representative of the population of physics students in the United States. The demographics of the students in our study (see Table II) do not perfectly match with the demographics of physics students overall [63], as we are overrepresenting white and Asian students. The selection of courses in our study are also not representative of all lab courses in the United States since the instructors had to opt-in to implementing the E-CLASS in their courses, which indicates those instructors cared about assessing the impact of their course on student attitudes about experimental physics. Additionally, both the FY and BFY data are dominated each by a single large-enrollment course.
Another limitation is that the students most affected by the pandemic might not have had the time, energy, or internet connection required to respond to this survey. Although we tried to make the remote lab survey questions quick and convenient to answer, it is possible that we are losing the voices of the students most affected by the pandemic. This could be because they were not able to attend their courses, they were not able to fill out the survey, or for other reasons. We keep this in mind while discussing our results, but we have no reason to believe our response rate was any worse than in prior years. In fact, there are indications that the E-CLASS response rate in the fall of 2020 was similar to the prior year [35].
Although we were concerned about the ethics of studying students during a pandemic, we believe this study had minimal negative impact on the students and some potential benefits. We minimized the burden of this study on the students by adding these questions on to an already existing part of the courses (the E-CLASS) and making responses to all questions optional. All of the questions were directly about the courses and not investigating additional stressful topics. Documenting student perspectives from lab courses during the pandemic could help instructors better understand student experiences. This has the potential to benefit other students in the future, including other remote lab courses that have the possibility to provide learning opportunities for students not able to participate in traditional lab courses.

IV. STUDENT EXPERIENCES WITH REMOTE LABS
In this section, we answer RQ1 by describing how students perceived their remote lab course experiences during the 2020-2021 academic year. Each section contains a different set of questions the students answered, which were grouped together by topic during the analysis. We present the general trends of FY and BFY courses next to each other to easily see commonalities between the two. All of the results for students in FY courses have 95% confidence intervals of 2%-4% and the results for students in BFY courses have 95% confidence intervals of 7%-15%.

A. Course modality
Before focusing in on remote lab courses, we investigated the percent of students who participated in their lab courses remotely. This enabled us to gain a broader understanding of lab courses during the pandemic and to know what fraction of total surveyed students we were studying. This section is the only part of the data presented in this article for which we include students who participated in their courses in person or in a hybrid approach. Additionally, in this section, we did not remove any students with missing responses for questions other than the ones about course modality. For the data presented in Fig. 1, there were 5070 responses from students in FY courses at 24 different institutions and 433 responses for students in BFY courses at 22 different institutions.
The majority of students in both FY and BFY courses reported completing their courses entirely remotely, yet many more students in BFY courses participated in their courses in-person or in a hybrid approach. In FY courses, 94% of the students completed their courses entirely remotely, whereas only 61% of students in BFY courses were remote. The difference between FY and BFY courses may be attributed both to the courses having different goals and the number of students enrolled in BFY courses being much smaller, so it was likely easier for those courses to facilitate social distancing measures. Compared with the recent American Institute of Physics survey [53], we found a higher percentage of students in our sample to have participated in labs remotely; however, our results are by student instead of by department and we did see some differences within our data between FY and BFY courses at the same institution that would show up at the department level. Of the students in our study, the majority of them (89% of students in FY courses and 81% of students in BFY courses) expressed that they did not have a choice in their course modality.

B. Course activities
For the rest of this article, we present results of only students who participated in their lab courses remotely (see Sec. III C for demographics of these students), beginning with a description of how students conducted hands-on experiments or activities in the remote environment. Figure 2(a) shows that the majority of students conducted hands-on experiments at some point during their remote courses. For students in FY courses, 48% conducted handson activities with high frequency (a response of always or often) with 33% never conducting hands-on activities. For students in BFY courses, 61% of students reported conducting hands-on activities with high frequency with only 11% reporting never doing any hands-on activities. There are several reasons why students in BFY courses may have been more likely to conduct hands-on experiments. First, BFY courses are generally smaller, which may have made it easier for the instructors to distribute the equipment necessary for hands-on remote labs. Second, there may be slight differences in goals between FY and BFY lab courses with FY courses typically having a stronger focus on learning concepts and BFY courses typically having a stronger focus on learning lab skills [10].
Of the students who performed hands-on experiments at home, there was a variety of ways they acquired equipment, as shown in Fig. 2(b). The options for acquiring equipment include students being given equipment by their institution, being required to purchase equipment, using household materials they already had, and using sensors on their smartphone. These categories are not exclusive. Students in BFY courses who performed experiments at home primarily were given equipment, used household materials, and used smartphone sensors, with fewer being required to purchase materials themselves. Compared with students in BFY courses, the students in FY courses were similarly likely to use household materials and smartphone sensors, but were less likely to be given equipment and more likely to be required to purchase equipment. Another way students participated in their remote lab courses was by watching others conduct experiments or activities through live or pre-recorded videos. Figure 2(a) shows that 42% of students in FY courses watched others over video with high frequency and 24% never did. Students in BFY courses responded very differently with only 23% watching others over video with high frequency and 37% never doing so. These differences may again be due to the different goals of FY and BFY courses or the different amount of opportunities for the students to perform hands-on experiments, since we presume that watching experiments over video was commonly used as a replacement for hands-on experiments.
There was a variety of ways students watched others conduct experiments over video. One option we explicitly asked the students about was whether they controlled the data collection or procedure remotely by communicating with an instructor, teaching assistant (TA), or other students. The second option we investigated was whether the students interacted with a video in order to collect data, for example, by measuring times and positions of objects in the video. Both of these activities were commonly done by students in FY courses, as shown in Fig. 2(c). For students in BFY courses, more communicated with others over video than collected the data by interacting with the video, and there were some-the other category-that did neither but still responded with anything other than never to the frequency scale question. These students could, for example, have watched an instructor-made video of an experiment without interacting with it.
The students were also asked about other kinds of activities in which they participated with varying frequencies. Figure 3 shows the responses to many of these frequency scale questions, including activities related to using simulations, designing experiments, working on complex questions, and writing or communicating about their work.
Students reported using simulations in different ways, with differences in use between students in FY and BFY courses. Of the students in FY courses, 51% used simulations with high frequency, whereas only 12% of students in BFY courses did so. This may be due to FY courses often focusing on learning concepts, since simulations have been shown to be useful for supporting students' conceptual understanding [64]. Follow-up questions examined how the students used simulations, and we found that 24% of students in FY courses and 43% of students in BFY courses created their own simulations. This left 62% of students in FY courses and 20% of students in BFY courses who used simulations in their courses without creating them. They may have used simulations to collect data or played around with simulations to understand the behavior of a physical system. Anecdotal evidence of some of us  found that some students may classify what is and what is not a simulation differently than their instructors, such as by considering any programming to be a simulation [35]. The students were also asked how frequently they designed their own experiment and their own experimental procedure. The overall responses to the two separate questions were very similar to each other and many, although not all, students responded to them identically. A total of 80% of students in FY courses and 90% of students in BFY courses designed an experiment and/or procedure with any frequency.
Another question asked students whether they worked over an extended period of time (two or more weeks) to investigate and respond to a complex question, challenge, or problem. The goal of this question was to investigate what many instructors refer to as projects. Many of the students, in both FY and BFY courses, did work on such a problem. For this question, along with the ones asking how frequently students gave presentations or wrote reports, we do not know how students interpreted the frequency scale-if a single project that extended over a large period of time was counted as rarely because it was one activity or sometimes, often, or always because the student worked on it for a long period of time.
Additionally, many students indicated that they had documented or presented their work in some way: through writing reports, giving presentations, or writing in a lab notebook. Giving a presentation (either oral or poster) was one of the least frequently done activities with approximately half of students in both FY and BFY courses never giving one. Many students in both FY and BFY courses kept lab notebooks, and over 90% of them wrote lab reports or proposals with any frequency.

C. Course environment
We also examined other aspects of what we refer to as the course environment. These include whether clear expectations for earning a good grade were provided, whether the students had enough time to complete their coursework, how often the students had access to guidance or mentorship, how often the students encountered challenges with technology, and how often the students completed lab activities as a group. These factors are partially influenced by the instructors, but also depend on factors outside of the instructors' control. Student responses to these questions are shown in Fig. 4.
Overall, the student responses were relatively favorable despite the ongoing pandemic. Over three quarters of the  students (both in FY and BFY courses) had access to guidance or mentorship from an instructor or TA with high frequency. More than half of the students agreed or strongly agreed that clear expectations for the course were provided and that they had enough time for their coursework. On the other hand, many students encountered challenges with technology that were a barrier to their participation in the course. We do not know whether or not this was something the instructor could control, such as a choice in analysis software as opposed to a poor internet connection. There were small differences between responses for students in FY and BFY courses, with students in FY courses being more likely to agree or strongly agree that clear expectations were provided and they had enough time for their coursework. It is unknown whether this is due to the pandemic or is a common difference between FY and BFY courses. The last question in Fig. 4 shows that a large majority of students in both FY and BFY courses worked with at least one other student with high frequency. This contrasts with studies looking at the spring of 2020, where many lab courses that had done primarily group work until the transition to remote instruction switched to individual work afterward [2,33]. It seems that with additional time for planning, instructors were able to incorporate group work into their remote lab courses.

D. Student outcomes
We classify the remaining questions in our survey as student outcomes, including whether or not the students reported that the course helped them learn concepts and lab skills and whether their learning was affected by the COVID-19 pandemic. All of these outcomes are selfreported by the students. Student responses to these questions are shown in Fig. 5. Although many students thought their learning was negatively affected by the pandemic, the majority still reported that they learned both concepts and lab skills. In order to differentiate between  The largest difference between FY and BFY courses was seen in the self-assessed learning of lab skills, which may be partly caused by developing lab skills being a goal for a larger fraction of BFY courses [10]. The course outcome with the most mixed student responses was whether the students developed a sense of community in the course. This is rarely an explicit learning goal of physics courses, but still may be one of the instructors' objectives. Since the courses studied were remote and the students were not physically together, it might be expected that students would not develop a sense of community, as was the case for many nonlab courses during the transition to remote teaching in the spring of 2020 [20,34]. However, in our study, more than 40% of the students agreed or strongly agreed that they developed a sense of community during the course.
We additionally investigated whether the students enjoyed their lab courses overall and which aspects of the courses the students enjoyed the most. Figure 5 shows that more than 50% of students in both FY and BFY courses enjoyed their lab courses, with only around 20% of each disagreeing or strongly disagreeing with that statement. It is impressive that even in the remote setting forced by the pandemic, the majority of students enjoyed the lab courses. Figure 6(a) shows the percentage of students who reported enjoying a variety of aspects of the courses. For both FY and BFY courses, the top three most enjoyable aspects were working in a group, working at the students' own pace, and learning concepts. To learn more about student experiences with group work, we asked the students who reported completing lab activities in a group whether they found group work enjoyable and productive. Of those students, 78% from FY courses and 81% from BFY courses found group work enjoyable, and 85% from FY courses and 82% from BFY courses found it productive.
On average, each student enjoyed many different aspects of their course. The median number of aspects found enjoyable was four for students in FY courses and five for students in BFY courses. For the students in BFY courses, almost 2 3 of the options given were enjoyed by close to half of the students. For the students in FY courses, there is more variation between percentages of students enjoying the different aspects, with working in a group being enjoyed by significantly more students than all of the Rank agreement with: Course helped me learn physics concepts.
Course helped me learn lab skills.
I developed a sense of community in my course.
I enjoyed my course.
My learning in this course was negatively impacted due to COVID-19. other options. Working in a group may have been particularly enjoyable for the students during this isolating time because it gave them a chance to interact, albeit virtually, with others. Note that some of the aspects of the courses that have the smallest percentages were likely not done by many students. For example, choosing a research question was enjoyed by the fewest number of students in FY courses, and this may be because it is not a common activity for FY courses. We also investigated the challenges faced by the students, as shown in Fig. 6(b). The largest challenge faced by students in both FY and BFY courses was the large workload with 37% of students in FY courses and 66% of students in BFY courses indicating that as a challenge. For students in BFY courses, this was a significantly larger percentage than for the other challenges; however, for students in FY courses the percentage was only slightly larger than for many of the other challenges. It is not clear whether the large workload being the biggest challenge is typical or whether it is due to the pandemic (either because of the remote course format or because of all the other factors in students' lives during a pandemic).
There were many aspects of these lab courses that the students overall found challenging, even though most students only mentioned facing a few challenges each.
Most of the possible challenges listed in the survey were found challenging by at least 20% of the students (for both FY and BFY courses). We investigated whether a small portion of the students found everything challenging or whether most students found some subset of these aspects challenging, and found it to be the latter. About half of the students in FY courses and about 40% of the students in BFY courses marked zero to two of the listed aspects as challenging, and the median number of challenges marked was two for students in FY courses and three for students in BFY courses. Thus, it seems that there were many potentially challenging aspects and each student found different parts of their course challenging.

V. IMPACT OF COURSE STRUCTURE ON STUDENT OUTCOMES
After gaining an initial understanding of how students experienced their remote lab courses, we further investigated how some of these experiences were connected with each other-in particular, how some of the student outcomes depended on the course activities and course environment. To do this, we applied a logistic regression to four common course outcomes: self-assessed learning of concepts, self-assessed learning of lab skills, enjoyment of What students enjoyed during course What challenges students faced during course the course, and development of a sense of community. We used the course activity and environment variables as inputs to the models. Because of the lack of statistical power coming from the small number of students in remote BFY courses in our sample, we present only the results for the students in FY courses. In this section, we present the odds ratios coming directly from the logistic regressions by student outcome; these results may be particularly useful for instructors who have specific course goals in mind. These results will be further discussed in Sec. VI where they are grouped by the course activity and course environment variables, so we can understand the overall effects of how students engaged with their remote lab courses in the context of prior research. For each outcome, we performed two separate one-vs-rest logistic regressions: students who agreed with the outcome compared with all other students and students who disagreed with the outcome compared with all other students. This allowed us to investigate the variables that both contributed to and hindered each student outcome. There was a strong similarity between the results for the two regressions for each outcome, so in this section we include only the plots of the odds ratios for the agree-vs-rest regressions. We briefly mention additional variables that were significant in the disagree-vs-rest regressions if they did not also appear in the agree-vs-rest regressions. The exact values of the odds ratios and 95% confidence intervals for all variables, as well as plots of the odds ratios for the disagree-vs-rest logistic regressions, are included in Appendices A-B.
The plots of the odds ratios (Figs. 7-10) list only the variables that have uncorrected p values less than 0.05. The variables that are statistically significant after the Holm-Bonferroni correction are shown in bold and discussed in this section. The plots have a vertical dashed line where the odds ratio equals one because it delineates an increase from a decrease in the odds. An odds ratio greater than one indicates that the outcome has a higher likelihood for students who report a given response to a variable compared with students who report the reference category response to that variable. For all of the independent variables, the reference category was set to be no, never, or neither agree nor disagree. In the text, we continue referring to responses of always or often as with high frequency and responses of rarely or sometimes as with low frequency. We use the phrases "agree" or "disagree" to encompass responses of both (dis)agree and strongly (dis)agree.

A. Self-assessed learning of concepts
The first outcome we considered with a logistic regression was whether or not the students reported that they learned concepts in their lab course. Figure 7 shows the odds ratios from the logistic regression model for students in FY courses where we see that the variable with the largest odds ratio is having clear expectations provided for earning a good grade. After that, other variables that contributed to students being more likely to respond that they learned concepts are creating own simulations, using simulations without creating them, having enough time for coursework, interacting with videos to collect data, conducting hands-on activities with high frequency, and keeping a lab notebook with high frequency. Having access to guidance with high frequency and designing an experiment or procedure led to students being less likely to disagree that they learned concepts (see Appendix A), even though these are not statistically significant for students agreeing that they learned concepts. When the students from the largest course were removed, controlling a procedure via video by communicating with another person became statistically significant.
Some variables also led to students being less likely to learn concepts. One of the most surprising outcomes from this regression is that writing a report with low frequency decreased the odds of students agreeing that they learned concepts by almost a factor of two. However, this variable did not change the odds of students disagreeing that they learned concepts (see Appendix A). When students from the largest course were removed, giving a presentation with high frequency instead became significant for decreasing the odds of students learning concepts. Further work would need to be done to better understand these results. Not having clear expectations also led to students being less likely to agree that they learned concepts and students who disagreed that they had clear expectations or enough time for coursework were more likely to report they did not learn concepts (see Appendix A).

B. Self-assessed learning of lab skills
The second outcome we modeled is whether or not students believe they learned lab skills in their course. Figure 8 shows that the four variables with the largest contribution are having access to guidance with high frequency, having clear expectations provided, conducting hands-on activities with high frequency, and having enough time for coursework. The frequency of conducting hands-on activities mattered, with a higher frequency leading to a higher likelihood of students having learned lab skills, even though conducting hands-on activities with low frequency still had a positive effect. Other variables also had a small effect including keeping a lab notebook with high frequency, designing an experiment or procedure, investigating a complex problem with high frequency, and controlling the procedure over video by communicating with someone else.

C. Course enjoyment
As shown in Fig. 9, the largest contributions to students agreeing that they enjoyed their remote lab courses came from access to guidance, clear expectations, and enough time for coursework. Students who had access to guidance or mentorship with high frequency had the largest odds ratios. The next two largest odds ratios are having clear expectations provided and having enough time for coursework. Students who agreed with these two variables were more likely to agree that they enjoyed the course, and students who disagreed that clear expectations were provided were less likely to agree that they enjoyed the course. Similarly, students who disagreed that they had clear expectations or enough time for coursework were also more likely to disagree that they enjoyed the course (see Appendix A). Thus, having clear expectations and enough time for coursework led to student enjoyment, and not having either of those hindered student enjoyment as well.
There are several other variables that also increased the odds of students enjoying the course by small amounts. These include working in a group with high frequency, controlling the procedure over video by communicating with another person, keeping a lab notebook with high frequency, and conducting hands-on activities with high frequency. In addition, students who investigated a complex problem were less likely to disagree that they enjoyed the course (see Appendix A), so that variable may have helped students be at least neutral about their enjoyment of the course.

D. Development of sense of community
Working in a group had the largest effect on whether the students developed a sense of community during their lab course. Figure 10 shows that students who worked in a group with high frequency had a three times higher odds of agreeing that they developed a sense of community compared with students who never worked in a group. Students who worked in a group with low frequency were still more likely to agree that they developed a sense of community than students who never did. In this context, group work means that the students completed some lab activities with at least one other student; it does not mean that the instructor intentionally assigned work that had to be done as a group. It is not surprising there is a connection between working in a group and developing community, since people often build community while working together.
The two variables with the next largest odds ratios were students agreeing that they had enough time for coursework and that clear expectations for earning a good grade were provided. These students had more than twice as large of odds of developing community as the students who were neutral about those variables. Students who disagreed that they had clear expectations or that they had enough time for coursework were similarly more likely to disagree that they developed community (see Appendix A).
Four other variables had statistically significant yet smaller odds ratios that contributed to students being more likely to agree that they developed community. These are investigating a complex question over an extended time with high frequency, giving a presentation with low frequency, designing an experiment or procedure, and keeping a lab notebook with high frequency. We found these four activities to be weakly correlated with each other. Additionally, students who created their own simulations or controlled the experimental procedure via video by communicating with another person were less likely to disagree that they developed community even though these variables were not significant for students agreeing that they developed community (see Appendix A). Lastly, students who conducted hands-on activities with high frequency became significant when the students from the largest course (which did not use hands-on equipment) were removed, so performing hands-on activities may have contributed to the development of community in some courses.

VI. DISCUSSION AND IMPLICATIONS FOR FUTURE LABS
In Sec. V, several variables had a significant impact across multiple outcomes (self-assessed learning of concepts, self-assessed learning of lab skills, course enjoyment, and development of community), so here we discuss the regression results by variable. This allows us to focus individually on some of the aspects of a course that an instructor can influence, thereby providing lessons for future lab instruction. We begin with the variables with the largest odds ratios: clear expectations for earning a good grade provided, enough time for coursework, access to guidance, and working in a group. All four of these variables describe the course environment-aspects which are probably relevant across all courses-instead of specific activities, which vary from course to course. We then discuss some of the activities that were also significant, including various ways students watched experiments over video and different forms of communication (keeping a lab notebook, writing a report, and giving a presentation). Some of these results raise questions for future research.

A. Clear expectations and enough time for coursework
Two of the variables that were statistically significant and had strong positive effects for all four outcomes were having clear expectations for earning a good grade and having enough time to complete the coursework. Students who agreed with these variables were more likely to also FIG. 10. Logistic regression odds ratios for students who agreed or strongly agreed that they developed a sense of community compared with all other students. Only variables with uncorrected p values less than 0.05 are shown. Bold variables are those that are statistically significant after the Holm-Bonferroni correction. Error bars show 95% confidence intervals. The odds ratios are on a log scale to make the positive and negative effects be of approximately equal size. agree with these outcomes and students who disagreed were more likely to disagree with the outcomes. Thus, it is important for instructors to not only consider intentionally incorporating these aspects into their classes to improve student outcomes, but to also realize that not providing clear expectations or enough time can significantly hinder these outcomes. For some outcomes, these two variables had comparable size effects to each other, but for selfassessed learning of skills and concepts, having clear expectations provided had a larger effect than the students having enough time for their coursework. These two variables may be related to each other. An instructor who provides clear expectations may have received feedback from the students on how much time it will take them to do their coursework or students may be able to better plan out time to do their coursework when they know how much work is expected of them.
Prior work has shown that providing clear expectations to students can lead to improved motivation and student learning [65,66]. Having clear expectations allows students to feel in control of their own learning and believe that they are able to achieve good grades if they do what is expected by the course, leading to improved motivation [65]. Students who understand the expectations of the courses also perform better academically, especially when the students may not yet know what is expected of typical college classes, as is the case for firstgeneration students in first-year courses [66]. In the spring of 2020, remote labs were rated by students as slightly worse at providing clear expectations than inperson labs [33], but we do not know if this improved by the fall of 2020. The results we present here align with prior work [65,66] and suggest that providing clear expectations is important for student learning, enjoyment of the course, and development of community, when the course is occurring in an unusual format, as was the case during the pandemic.
Whether or not students have enough time for their coursework has affected student outcomes in courses both during and prior to the pandemic, although the size of the impact may have varied. The amount of time students have for their coursework depends both on the amount of assigned work and the course structure as well as on students' personal circumstances that determine whether or not they can focus on their school work. Even before the pandemic, prior work has shown associations between appropriate workloads and student satisfaction and academic achievement [67]. Many students in the spring of 2020 reported spending more time on coursework after the transition to emergency remote teaching than before [18,19], and one study showed that the large workload was a cause of student stress [30]. In our dataset, the large workload was the most commonly faced challenge by the students (see Fig. 6). It is important for instructors to realize how large of an effect a large workload can have on student enjoyment of the course, students' self-reported learning of concepts and lab skills, and even development of community. Instructor consideration of the amount of time students have to dedicate to a course may be particularly beneficial for students with outside responsibilities or jobs.

B. Access to guidance
The other variable with particularly large odds ratios for enjoying the course and self-assessed learning of both lab skills and concepts was whether the students had access to guidance or mentorship from their instructor or TAs. Having access to guidance with high frequency was significantly better than never having any, and having access to guidance with low frequency was somewhere in between. However, having access to guidance with low frequency was not statistically significant when looking at the corrected p values. This may be due to the small number of students who responded Never, the reference category, to that question, which may also have caused the confidence intervals to be large. Having access to guidance was not significant for whether or not the students developed a sense of community.
One way to consider the effect of interactions with instructors on students' learning is through the framework of self-regulated learning. Self-regulated learning is a process where the learners themselves are able to carry out the metacognitive, motivational, and behavioral processes necessary for learning [68,69]. Students who have strong selfregulation skills learn better and more efficiently than students who do not [70]. This is especially true in remote courses where there is less structure and students have more control over their own learning [71][72][73]. A strong correlation between self-organization skills (a part of self-regulation) during the pandemic and learning achievement was found in Ref. [21]. Instructors can help their students better regulate their learning through co-regulation by providing guidance to them at the appropriate time [69].
It is possible that having access to guidance is also correlated with other aspects of the course, both other variables studied here and external factors outside the scope of this work. For example, instructors who are more available for their students may have more time for teaching, which could additionally lead to them providing clear expectations or helping students feel motivated because the students believe their instructors care about their learning. Other work has similarly found that the quality and frequency of interactions between students and instructors or TAs can affect academic performance [74] and student engagement [75]. The amount of studentinstructor interactions may additionally depend on demographic factors. One study found that women and nonwhite students were more likely to have less frequently sought help from an instructor during the pandemic compared with before [53]. There are a variety of ways students can receive guidance from their instructors even in a remote environment, and more work is needed to understand exactly how student-instructor interactions lead to learning and other outcomes [76].

C. Working in a group
Working in a group with high frequency was the variable with the largest odds ratio for students developing community, and it also had a small effect on students enjoying their course. Students who participated in group work with high frequency were much more likely to develop community than students who never did, and students that did group work with low frequency were somewhere in between. Thus, the more students worked in groups, the more likely they were to develop a sense of community, even in a remote course environment. Note that we still found gains from a small or moderate amount of group work, which could be beneficial in situations where it is not possible for students to work in groups frequently. Additionally, students who worked in a group with high frequency were slightly more likely to enjoy the course. Frequently working in a group did not show up as significant for self-assessed learning of either concepts or lab skills, in contrast with a recent study that showed how frequently working in a group led to student conceptual learning gains in pre-pandemic introductory physics courses [77].
There are many documented benefits of group work [78,79], including in online courses [80], with the potential to build a sense of community being particularly beneficial to students during a pandemic. Although developing community is typically associated with in-person interactions, it has been shown that it is possible to intentionally build a sense of community in online courses, with small group activities being one contributing factor [81]. One study of remote physics lab courses during fall 2020 emphasized this need for formalized structures to help the students develop a sense of community [32]. We do not know how much the courses participating in our survey intentionally focused on building community or working in groups, but we found that more than 40% of the students developed community in their remote lab courses (see Fig. 5). This is particularly important during a pandemic, when students are at higher risk for loneliness [82], which can be mitigated by a strong sense of community [83].
It is not clear whether or not the impact of working in a group on course enjoyment was primarily due to the pandemic. Our survey does not differentiate between structured group work (e.g., instructors requiring students to complete certain activities as a group) and students choosing to work with each other on their own (e.g., informal study groups). It is possible that any work done with other students (whether assigned or not) could take the place of some of the ad hoc interactions students typically have with each other during in-person education (e.g., interactions in between classes). Group work being associated with a higher likelihood of students both enjoying the course and developing community could be due to these course interactions being the primary outlet students had to interact with others during the pandemic.

D. Other significant variables
There are other variables that had small but positive significant effects for at least one of the four outcomes. These are specific activities (e.g., different methods of using videos of experiments, keeping a lab notebook, and writing reports or giving presentations) and not broader aspects of the course environment. Compared with the four variables in the preceding sections, these variables had smaller odds ratios (close to or less than two), which may be because fewer courses in our study performed each of these different activities. Lab courses are complex spaces, so student outcomes may depend on many smaller elements of the course adding together. Some of these significant variables have been found by other studies to lead to beneficial student outcomes (for example, there are many documented benefits to incorporating experimental design in lab courses [56,[84][85][86]), while others prompt new research directions.
One area that merits further investigation is understanding the most effective ways to use videos of experiments in remote lab courses. During the pandemic, instructors incorporated videos of experiments in their lab courses in a variety of ways [33]. In our study, students who interacted with the video to collect data themselves were more likely to report they had learned concepts, whereas students who controlled the procedure by communicating with others over video were more likely to report learning lab skills, enjoying their course, and developing community (or at least being less likely to disagree that they developed community). It is not clear if instructors' choice of how to engage with videos in their courses contributed to the student outcomes or whether the form of engagement chosen was only correlated with the course goals and thus the student outcomes. Various studies have looked at different aspects of this question. One study, conducted during the COVID-19 pandemic, found that students reported learning more concepts and skills when collecting their own data (for example, from a video) instead of being given data [21]. A prepandemic study found that students learned more from watching videos of demonstrations instead of live demonstrations [87]. Another study directly compared students working with hands-on apparatus and those working with a video of the same experiment. They found slight variations in students' mental states and enjoyment between the two groups [88]. Future work could investigate the different benefits that arise from various forms of engagement with videos of experiments.
Another significant variable that is not well studied is keeping a lab notebook. In our results, students who kept lab notebooks with high frequency had slightly higher odds of agreeing with all four outcomes. There has not been much work within physics education research looking at the effects of lab notebooks, but prior work found that many students did not learn authentic scientific documentation practices in their courses [89]. Other work has shown that students found electronic lab notebooks easier to use and better for collaboration than paper notebooks [90]. We do not know what fraction of the students used electronic lab notebooks instead of paper lab notebooks during their remote labs. Future work could investigate if keeping a lab notebook in in-person courses has the same positive effect on these outcomes, whether or not there is a causal connection, and if so what is the mechanism behind it.
We additionally find the surprising result that writing a report with low frequency had a small negative effect on students' self-reported conceptual learning. When the students in the largest course were removed from the dataset, giving a presentation with high frequency instead caused students to be less likely to report that they learned concepts. Both writing reports and giving presentations are variables where the frequency scale on its own may be misleading because the scale of the activity is also relevant. For example, some courses might have a single report or presentation at the end of the course which accounts for the majority of the students' grades. Thus, a student who wrote a report or gave a presentation with low frequency might still spend a lot of time on those components of the course, possibly even more so than students who frequently gave very small presentations. This could lead to students who reported doing these activities with low frequency having more meaningful experiences than students who reported doing them with high frequency. It is not obvious why writing a report would be associated with students being less likely to learn concepts, so future work is needed to investigate the benefits and drawbacks of incorporating different forms of communication or presentations in lab courses [91,92]. The importance, and not just the frequency, of these activities should be taken into account.

VII. CONCLUSIONS
In order to find what lessons could be learned from remote lab courses during the 2020-2021 academic year, we surveyed students about their course experiences and modeled four outcomes for students in first-year courses with logistic regressions. Even though the majority of students thought their learning was negatively impacted due to COVID, many students still enjoyed their courses and reported learning concepts and lab skills. Some students even developed a sense of community in the remote environment. Students who agreed that they were provided clear expectations for earning a good grade and had enough time for coursework were much more likely to agree with all positive outcomes than students who did not. Additionally, students who worked in groups were the most likely to develop community and students who frequently had access to guidance from their instructor or TA were likely to enjoy their course and report learning both concepts and lab skills. While these conclusions are subject to the limitations of our methodology (Sec. III E), we believe that these most impactful variables are aspects that could improve any course, independent of there being a pandemic. The world changed a lot during the studied year, but the principles of teaching and learning did not.
Many other activities or aspects of the course environment contributed smaller amounts to the different outcomes, raising several new questions for follow-up studies. One such area is studying the benefits of incorporating different kinds of writing and presentations in lab courses. Further work could aim to understand both how students engage with these activities through more than a single frequency variable and how the possible forms of engagement contribute, either positively or negatively, to different outcomes. The best ways to engage students with videos of experiments is another area that requires further investigation. A better understanding of which aspects are crucial for different student outcomes will help improve remote lab courses, thereby allowing opportunities for students to learn important parts of physics even when they cannot physically work with lab equipment. Some of these results may also suggest opportunities for additional professional development for instructors running remote labs.
Although the focus of this work has been on remote labs, we are not advocating for the removal of in-person labs; the two modalities provide very different affordances. We hope this work contributes to the broad goal of understanding the distinct possible benefits for students from each modality so the appropriate choice can be made for any given course based on the situation and the desired student outcomes [93].

ACKNOWLEDGMENTS
We thank the students who responded to our survey, the CU PER group for useful conversations and feedback, and Kenna Castleberry and the students in the JILA writing workshop for feedback on the manuscript. This work is supported by NSF Grants No. DUE-2027582 and No. PHY-1734006. Figure 11 shows plots of the statistically significant variables for the disagree-vs-rest regressions, similar to those in Sec. V for the agree-vs-rest regressions. For each of the four outcomes (learning concepts, learning lab skills, enjoying the course, and developing a sense of community), we plot the odds ratios of the variables that have uncorrected p values less than 0.05 and indicate with bold the variables that remain under 0.05 after the Holm-Bonferroni correction. All of these variables are compared with the same reference categories as for the agree-vs-rest regressions, that is a response of no, never, or neither agree nor disagree, depending on the possible question response options.

APPENDIX B: DETAILED REGRESSION RESULTS TABLES
Tables IV-VII show the odds ratios, 95% confidence intervals, uncorrected p values, and Holm-Bonferronicorrected p values for all variables for the regressions for the four outcomes (self-assessed learning of concepts, self-assessed learning of lab skills, course enjoyment, and development of community). The agree-vs-rest and disagree-vs rest regression models for each outcome are shown side by side for ease of comparison. FIG. 11. Logistic regression odds ratios for students who disagreed or strongly disagreed that they (a) learned concepts, (b) learned lab skills, (c) enjoyed the course, and (d) developed a sense of community. Only variables with uncorrected p values less than 0.5 are shown. Bold variables are those that are statistically significant after the Holm-Bonferroni correction. Error bars show 95% confidence intervals. Note that the odds ratios are on a log scale to make a decrease in the odds be of an approximately equal size to an increase. Regression results for students in FY courses reporting whether or not they developed a sense of community in their course. The agree-vs-rest regression compares students who responded agree or strongly agree to all other students and the disagree-vs-rest regression compares students who responded disagree or strongly disagree to all other students. The corrected p values use the Holm-Bonferroni correction, as described in the main paper. Agree-vs-rest