Challenge of engaging all students via self-paced interactive electronic learning tutorials for introductory physics

As research-based, self-paced electronic learning tools become increasingly available, a critical issue educators encounter is implementing strategies to ensure that all students engage with them as intended. Here, we first discuss the effectiveness of electronic learning tutorials as self-paced learning tools in large enrollment brick and mortar introductory physics courses and then propose a framework for helping students engage effectively with the learning tools. The tutorials were developed via research in physics education and were found to be effective for a diverse group of introductory physics students in one-on-one implementation. Instructors encouraged the use of these tools in a self-paced learning environment by telling students that they would be helpful for solving the assigned homework problems and that the underlying physics principles in the tutorial problems would be similar to those in the in-class quizzes (which we call paired problems). We find that many students in the courses in which these interactive electronic learning tutorials were assigned as a self-study tool performed poorly on the paired problems. In contrast, a majority of student volunteers in one-on-one implementation greatly benefited from the tutorials and performed well on the paired problems. The significantly lower overall performance on paired problems administered as an in-class quiz compared to the performance of student volunteers who used the research-based tutorials in one-on-one implementation suggests that many students enrolled in introductory physics courses did not effectively engage with the tutorials outside of class and may have only used them superficially. The findings suggest that many students in need of out-of-class remediation via self-paced learning tools may have difficulty motivating themselves and may lack the self-regulation and timemanagement skills to engage effectively with tools specially designed to help them learn at their own pace. We conclude by proposing a theoretical framework to help students with diverse prior preparations engage effectively with self-paced learning tools.


I. INTRODUCTION
Effective use of electronic learning tools for self-paced learning can provide a variety of students an opportunity to learn using an approach that allows each student to make progress at a pace that is commensurate with their prior knowledge [1][2][3][4][5]. Many instructors provide web-based tools to their students to encourage "self-study" outside of class, even in brick-and-mortar classes. These tools have the potential to help students learn to think in an expertlike manner while engaging in problem solving and can expose students to concepts in a way that scaffolds learning [6][7][8][9][10].
Some tools available to students are adaptive in that they adapt to students with different prior knowledge and skills. For example, an adaptive learning tool may provide more scaffolding support to a student who is struggling than others in the same course. Moreover, interactive electronic learning tools that are designed via research can be particularly beneficial because they can help a variety of students with different prior preparations and allow them to learn at their own pace [11][12][13][14][15].
However, an issue that instructors often encounter is achieving appropriate student engagement with these selfpaced learning tools, especially among those who are struggling with the course material and are in need of remediation. In particular, many students may have difficulty motivating themselves; they may not have clear goals and may lack self-regulation and time-management skills, which are critical for effectively engaging with self-study tools [16][17][18]. They may also lack self-efficacy or belief in their ability to be able to learn using self-paced learning tools and their beliefs about the nature of learning may also negatively influence how they engage with these learning tools and learn from them [19]. Students may also struggle due to social and environmental factors, e.g., with outside demands on their time from other competing activities such as other classes, work, or family obligations. They may lack a supportive environment, including encouragement from family, and may also lack the guidance and support of other mentors and advisors in order to make effective use of the self-paced learning tools. Without sufficient help for developing these skills and incentives to motivate them and strategies to alleviate the negative impact of social and environmental factors, students may not engage effectively with the self-paced electronic learning tools. It is therefore important to investigate whether students engage with selfpaced learning tools effectively and various factors that can deter or incentivize their use in order to develop a holistic learning environment to help students with diverse backgrounds benefit from these tools. Additionally, it is valuable to examine and compare the effectiveness of selfpaced learning tools when implemented in a controlled environment in which students must effectively engage with the tool vs an environment in which students are free to use the tool in whatever manner they choose. A framework for understanding the factors that can support or hinder effective use of self-paced learning tools would be helpful in developing and implementing self-paced tools in environments conducive to learning in the future.
Here we first describe an investigation of the effectiveness of interactive electronic learning tutorials as a selfstudy tool to help introductory physics students learn mechanics by analyzing their performance on similar problems administered as quizzes in recitation classes. Typically, the student population in the introductory physics courses is very diverse in terms of students' prior preparation, goals, motivation for engaging with the material in the course and other characteristics. These tutorials were developed using research in physics education and were refined through an iterative process including feedback from students and instructors [12][13][14][15]. They are designed to aid students with diverse backgrounds via a guided approach to learning, in which the guiding questions provide scaffolding support to help students learn physics concepts and develop problem-solving and reasoning skills. The effectiveness is examined via implementation of these tutorials in both controlled and uncontrolled environments to determine the difference in effectiveness brought about by various constraints inherent to the implementation of self-paced learning tools. Reflecting on the findings from the tutorial implementations, we propose a self-study for engaged learning framework (SELF) that takes into account the characteristics of the student, the social and environmental factors that influence student learning, the properties of the self-paced learning tool, and how the tools are implemented. The framework can be used to guide the development and implementation of future self-paced learning tools. Our goals are as follows: • Determine the effectiveness of the electronic learning tutorials for a diverse group of introductory physics students at a large research university in one-on-one interviews in which researchers ensured that the tutorials are used as intended. • Determine the effectiveness of the same tutorials as self-study tools for a diverse group of introductory physics students in brick-and-mortar classes in which researchers had no control over how the tutorials were used by the students. • Compare the performance of the students who worked on the tutorials in a one-on-one interview vs those who used them as self-study tools in the traditional, brick-and-mortar introductory physics courses. • Develop a theoretical framework that can be useful for effective implementation of self-paced learning tools in the future by hypothesizing possible reasons for the differences in the performance of students who worked on the self-paced tutorials in a one-on-one interview situation vs those who used them at home as a self-study tool to improve their performance on homework and quizzes in a traditional introductory physics course. We first describe how the interactive tutorials were developed via research. Then, we describe how their effectiveness was investigated in one-on-one implementation via individual interviews with students. Next, we discuss how their effectiveness was probed as a self-study tool in large brick-and-mortar introductory physics courses. We then compare their effectiveness in one-on-one interviews vs as a self-study tool in large introductory physics classes. Finally, reflecting back on our findings, we propose the SELF, a theoretical framework that synthesizes various factors that are critical for helping students with diverse prior backgrounds and preparations and with different social and environmental constraints benefit from selfstudy tools. The framework proposes that lack of social and environmental factors conducive to learning and inadequate incentives and support to help students engage with them, the self-study tools may not have a positive impact on learning even if they are developed via research and effective in one-on-one controlled implementation.

II. DEVELOPMENT AND STRUCTURE OF THE TUTORIALS
The development of the electronic learning tutorials is guided by a cognitive apprenticeship learning paradigm [20,21] which involves three essential components: modeling, coaching, and weaning. In this approach, "modeling" implies that the knowledge and skills that students should learn (e.g., how to solve physics problems systematically) is demonstrated and exemplified appropriately to help students learn. "Coaching" means providing students opportunity, guidance, and practice so that they are actively engaged in learning the skills necessary for good performance. "Weaning" consists of reducing the support and feedback gradually so as to help students develop selfreliance. The cognitive apprenticeship learning paradigm is effective in helping students develop expertise in the knowledge-rich domain of physics because the "modeling" and "coaching" aspects explicitly bring out the novice vs expert differences in an authentic learning context [20,21]. When students are actively engaged in practicing the skills, the "coach" can give immediate feedback on their performance in order to help them learn physics content and develop problem-solving and reasoning skills. After obtaining feedback, students reflect on their learning experiences and are providing further support as needed to continuously improve. The feedback is gradually reduced until students develop self-reliance.
The electronic learning tutorials are developed to model and coach students to learn physics content and develop problem-solving and reasoning skills and wean students as they develop self-reliance. Each tutorial starts with an overarching problem that is quantitative in nature. Figure 1 is an example of one of these overarching problems. Before working through a tutorial, students are asked to attempt the problem to the best of their ability. The tutorial then divides this overarching problem into a series of subproblems, which take the form of research-guided conceptual multiple-choice questions. These subproblems model effective steps for successfully solving a physics problem, e.g., analyzing the problem conceptually, planning the solution and decision making, implementing the plan, and assessing and reflecting on the problem-solving process. The alternative choices in these multiple-choice questions elicit common difficulties students have with relevant concepts found via research. Incorrect responses trigger immediate help based upon the student response and coaching is provided by directing students to additional scaffolding (students are directed to help sessions in which they are provided suitable feedback and explanations both conceptually and with diagrams and/or appropriate equations to learn relevant physics concepts). The feedback and explanations given are meant to coach students in order to help them repair, organize, and extend their knowledge structure and develop problem-solving and reasoning skills. Correct responses to the multiple-choice questions advance students to a brief statement affirming their selection followed by the next subproblem.
In addition to the tutorial problem statement in Fig. 1, the investigation described here was conducted on two other tutorials. In the Newton's second law tutorial, students are provided a set of three blocks on an inclined plane connected via strings to each other and being pulled up the incline. They are asked to determine the acceleration of the middle block and the tension in all strings. In the conservation of mechanical energy and work-energy theorem tutorial, students are provided a problem in which they must use both conservation of mechanical energy and work-energy theorem in two subproblems of the problem. In this problem, they are asked to determine the safety of a stunt in which a man is shot out of a spring-loaded cannon and onto an airbag. For the first subproblem involving mechanical energy conservation, students are provided the initial compression of the spring, and various heights in order to be able to figure out the changes in the gravitational potential energy so that they can find the speed of the person right before he falls on the airbag. For the second subproblem involving the work-energy theorem, they are provided the thickness of the airbag and the average force the airbag exerts on the person in order to figure out whether the person stops before the airbag is fully compressed (if that is the case, the person is safe). Figure 2 shows examples of subproblems in two of the electronic learning tutorials. The top two images in Fig. 2 are from the Newton's second law tutorial and provide students an opportunity to determine which free body diagram is correct for a system of three blocks that are in contact resting on an inclined plane with a force applied upwards along the plane. The tutorials coach students in an adaptive manner in that they provide suitable feedback and help to students if they select a particular incorrect answer to a subproblem and focus on providing immediate support to help students reflect upon and overcome their difficulty. This structure is suitable for students with diverse levels of prior preparations and backgrounds since it provides additional support to underprepared students while allowing more prepared students with adequate knowledge of the physics concepts to move more quickly. For example, in Fig. 2, if students select option A, they are provided with help that focuses on the fact that the plane applies a normal force. If they select option C (as shown in Fig. 2), the help focuses on the fact that the normal force should point perpendicular to the surface, and, similarly, if they select option D, the help focuses on the fact that the gravitational force points vertically down, instead of pointing in the direction perpendicular to the surface. The bottom two images in Fig. 2 are from the conservation of energy tutorial. Students are asked to determine which forms of energy the spring-Dave-earth system possesses before the spring is released. If students select option A, they learn that the system possesses no kinetic energy because Dave has zero initial speed. If they select option B, they learn that the system does possess spring potential energy but that Dave also started at some initial height above the reference height so there is some gravitational potential energy. Similarly, if students select option C, they learn that the system possesses gravitational potential energy but the spring was initially compressed so there is nonzero elastic potential energy also. The coaching and feedback students obtain when they select the correct answer (option D) is shown in Fig. 2 and helps them learn that the system has both elastic and gravitational potential energy at the moment in question.
After students work on the subproblems, they answer several reflection subproblems. These reflection subproblems focus on helping students reflect upon what they have learned and apply the concepts learned to different contexts. If students have difficulty answering the reflective subproblems, the tutorial provides further assistance and feedback in the form of a review of the effective problemsolving approach. Thus, this interactive electronic learning tool does not merely model or exemplify a systematic approach to problem solving, it engages students actively in the use of this systematic approach and provides feedback and guidance (coaching) based on their need. The tutorial will provide less scaffolding if students become more proficient in solving the later subproblems on their own without help.
Each electronic learning tutorial problem is matched with paired problems that use similar physics concepts but which are somewhat different in context. Students can be given these paired problems as quizzes so that they learn to decontextualize the problem-solving approach and concepts learned via the electronic learning tutorial. The paired problems also play an important role in the weaning part of the learning model and ensure that students develop complete self-reliance and are able to solve problems based upon the same concepts without any guidance. Students' performance on the paired problems after they work on the tutorial was used to evaluate the effectiveness of each tutorial. One of the paired problems posed in association with the tutorials is as follows: A 20 kg boy stands on a small stationary (at rest) merrygo-round near the edge of the merry-go-round. The total moment of inertia of the system of merry-go-round with the boy on it about the center is 120 kg m 2 . The boy at the edge of the merry-go-round (radius of 2 m) jumps off the merry-go-round in a tangential direction with a linear speed of 1.5 m=s. What is the angular speed of the merry-go-round after the boy leaves it?
The other paired problem associated with this tutorial and the ones related to the other two tutorials used in this investigation are available for download online [22,23].
Although twenty such electronic learning tutorials were developed, which cover many topics in introductory physics related to mechanics, electricity, and magnetism, in this investigation, we focus on the effectiveness of three of these tutorials focused on introductory mechanics described earlier. The three interactive electronic learning tutorials selected for this research study were developed to improve student understanding of physics principles which are central in an introductory mechanics course: application of Newton's second law, conservation of energy and workenergy theorem, and conservation of angular momentum. All three interactive tutorials were developed using the protocol discussed above. First, a quantitative problem that requires use of these physics principles was selected. Each tutorial problem was chosen to be somewhat more difficult than a typical introductory level physics homework problem on the same physics principle (these problems were used for quizzing in introductory physics courses at the same university so their difficulty level was known). This level of difficulty was chosen so that the problems could not be solved using a plug and chug approach and would have enough depth to be able to help students learn an expertlike problem-solving approach. Then, a cognitive task analysis was performed by three graduate student researchers and one professor (all physics education researchers) to break down each tutorial problem into a series of subproblems dealing with different stages of problem solving that must be answered to solve the tutorial problem. Each subproblem was then posed as a multiple choice question. The incorrect options for each multiple choice question were chosen to emphasize common difficulties uncovered by having introductory physics students solve similar problems in an open-ended format. Explanations for each multiple choice option were written and refined, based on one-on-one student interviews, to reinforce student understanding of the reasoning behind the options given and to aid them in repairing their knowledge structure when students selected an incorrect option. Using this approach, the initial drafts of the electronic learning tutorials were created. Each initial draft was revised several times based on interviews with introductory physics students and feedback from graduate students and several professors who were asked to work through them and provide feedback to ensure that they were comfortable with the wording of the subproblems and progression of the tutorial. During this refinement process, the fine-tuned versions of the electronic learning tutorials were implemented in one-on-one think aloud [24,25] interviews with introductory physics students and were shown to improve student performance on the paired problems that were developed in parallel with the tutorials.
Comparing the lengths of the three electronic learning tutorials, we note that the Newton's second law and conservation of energy and work-energy theorem tutorials were made up of 17 and 19 subproblems, respectively, while the conservation of angular momentum tutorial is made up of 7 subproblems. Based upon prior research, it is possible that the more complex problems may be more effective in helping students learn a systematic approach to problem solving and connect different physics concepts [26]. Newton's second law was complex because it involved several blocks and conservation of energy and work-energy problem was complex because it was a context-rich problem and involved two physics principles. However, since angular momentum conservation is challenging to conceptualize, the researchers collectively decided to investigate how a short tutorial that focuses on why angular momentum is conserved in a given context and how to apply the angular momentum conservation principle in one problem helps students transfer their learning to an isomorphic paired problem.

III. RESEARCH METHODOLOGY
Below, we describe the methodology for the implementation of the three electronic learning tutorials in oneon-one implementation with student volunteers and as a self-study tool as part of traditional brick and mortar large introductory physics courses at the University of Pittsburgh (which is a large, typical state-affiliated university in the U.S.) to evaluate their effectiveness.

A. Deliberate one-on-one implementation
One of our goals was to determine the effectiveness of these electronic learning tutorials in one-on-one implementation. Therefore, they were administered individually to introductory students in deliberate one-on-one think-aloud interview settings [27] so that researchers could monitor whether students were using them as intended. The thinkaloud process was useful for ensuring student engagement during the interview process. The students recruited had grades ranging from A to low C in their introductory physics class focusing on mechanics and were recruited via an email that was distributed late in the semester so that all pertinent physics concepts covered in the tutorials had been introduced. The students were paid for their participation and all students that responded to the email were interviewed. Informal questioning of the students regarding their performance in their introductory physics class exams, homework and quizzes revealed that the students were typical introductory level students with several performing particularly well in the class and several struggling. In these courses, typically 10% of the students receive a grade of C-or lower, so these recruited students form a representative group from the class. These interviews were audio recorded.
In this deliberate one-on-one implementation, students were observed by a researcher as they worked on the electronic learning tutorials but they otherwise followed the same instructions that were given to the students in the large introductory physics courses, who used them as a self-study tool. However, in one-on-one implementation, the researchers made sure that students adhered to the guidelines. For example, students had to first attempt to outline the solution to the tutorial problem to the best of their ability and only then asked to start the tutorial and attempted each subproblem in the appropriate order. As noted, throughout this one-on-one implementation process in which a student worked on the electronic learning tutorials, the student was asked to think aloud while being audio recorded and a researcher made further record of his observations of each student's interaction with the tool. This process was repeated with each student for each tutorial.
Twenty-two 2-to 3-hour long, one-on-one, think-aloud interviews were conducted with volunteers who were either in an algebra or calculus-based introductory physics course. We note that although several interviews were conducted with students individually during the development of the electronic learning tutorials, we conducted these 22 additional one-on-one interviews with students who had been exposed to a typical classroom treatment of Newton's second law, conservation of energy and work energy theorem, and conservation of angular momentum. In 17 of these interviews three tutorials were covered, and in the remaining 5 only two tutorials were covered due to the interviewed student working slowly. In each case, the order in which the tutorials were presented was changed resulting in approximately 20 individual interviews for each tutorial (roughly half of which were with students in the algebrabased physics course and the other half were with students in the calculus-based physics course). Throughout this process, a researcher was present to provide materials, and the interviewer ensured that the students explicitly followed the instructions provided and made effective use of the tutorial. Students were asked to think aloud so that researcher could understand their thought processes and the researcher remained silent while the students worked unless they became quiet, in which case the researcher prompted students to keep talking. After working through the entire electronic learning tutorial, the students worked on the corresponding paired problem.

B. Large scale implementation of the tutorial as a self-study tool
We also investigated the effectiveness of the electronic learning tutorials for a diverse group of introductory physics students in brick-and-mortar classes in which researchers provided guidelines but otherwise had no control over how the tutorials were used by the students. The tutorials were implemented as self-study tools in two traditional, large introductory physics courses. The first course was an algebra-based first semester introductory physics course with roughly 385 students (split into two sections). These students came from varied backgrounds in math and science with a majority of them pursuing bioscience or neuroscience majors. The second course was a calculus-based first semester introductory physics course with roughly 350 students (also split into two sections). These classes were the same two introductory classes that individual interview participants were recruited from but during a different semester. The students in this course were almost entirely physical science, mathematics, and engineering majors.
Each of the three tutorials was posted on the course website as a self-study tool after students had received classroom instruction in relevant concepts. They were intended to be used at students' discretion after the associated physics concepts and principles were introduced in lecture but before students had the opportunity to do the associated homework problems. The links to the tutorial were uploaded on the course website but the amount of time each student spent working through them could not be tracked. Students were aware that no points would be awarded for completing the electronic learning tutorials, but announcements were made in class, posted on the course website, and sent via email informing students that the tutorials were available when relevant concepts were covered in class. The incentive that the instructors gave to their students for engaging with these selfpaced tutorials was that they would be helpful for solving assigned homework problems and in-class quiz problems (paired problems) for that week.
The paired problems associated with each tutorial were given to students during their regular weekly recitation class. These quizzes with paired problems were given after students had been given access to the associated tutorial for an entire week. Each paired problem was administered in the week following instruction in a particular concept. All students had sufficient time to complete the quizzes. Students were given a grade based on their performance on these paired problems as their weekly quiz grade. At the top of each of the paired problem quizzes administered in the recitation, students were asked the following questions and assured that the answers to these questions would not influence their grade: • Have you worked on the corresponding online tutorial? • Was the tutorial effective at clarifying any issues you had with the problem covered in the tutorial? • If the tutorial was ineffective, explain what can be done to make it effective? • How much time did you spend on the tutorial?
To compare the performance of the students who worked on the electronic learning tutorials in a one-on-one interview vs those who used them as a self-study tool, we compared student performance on the paired problems. Rubrics were developed by three graduate students and a professor for each of the paired problems. Once the rubric for grading each paired problem was agreed upon, 10% of the paired problem quizzes were graded independently by three graduate students and a professor with the finalized version of the rubric. When the scores were compared, the interrater agreement was better than 90% across all graders. After interrater reliability was established for the rubric the remaining problems were graded by one of the researchers.

A. Deliberate one-on-one implementation
The purpose of conducting the 22 individual interviews was to gauge the effectiveness of these electronic learning tutorials when administered in a controlled environment in which a researcher can monitor that they are being used as intended compared to their use as a self-study tool in large enrollment classes. Table I shows the average performance of students along with the standard deviation in the one-onone interview group. Table I shows that students in a oneon-one interview setting had an average score of above 80% on all the paired problems.
B. Large scale implementation of the electronic learning tutorial as a self-study tool In this section, we discuss the average performance of students on the paired problems in the large scale implementation of the tutorials as a self-study tool. Before working on the paired problem, students were asked whether they had worked on the tutorial as a selfstudy tool and how much time they spent working on it. Also, students were asked to write down on the paired problem if the tutorial was effective at clarifying any issues they had with the tutorial problem. They were told that they should be honest because the answer to the question would not impact their grade. Only approximately 60% or less of the students reported that they worked through each of the tutorials in both the calculus-based and algebra-based classes. Table II shows that a majority of students thought that the tutorials were effective at clarifying issues they had with the problem. Students were also asked to write down whether anything can be done to make the tutorials effective if they thought it was ineffective. Most students provided no comments and a few students who provided comments generally noted that perhaps they can be made shorter so that they can quickly browse over them.
In Table III, students in the large introductory physics classes are divided into the "tutorial" or "nontutorial" group based upon self-reported data about whether they worked on the tutorial regardless of how much time they had spent working on it. Table III shows that two of the three tutorials that were given as a self-study tool resulted in a statistically significant increase in average student performance (compared to the nontutorial group) on the paired problem in the algebra-based group although the average scores of the tutorial group are very low on all paired problems and range from 47% to 54%. The only tutorial that resulted in a statistically significant increase in average for both the algebra-based and calculus-based groups is the conservation of angular momentum tutorial. Table III also shows that the effect size between the tutorial and nontutorial groups is small for almost all tutorials for both algebrabased and calculus-based classes. Only the conservation of angular momentum tutorial (which is the shortest of the three tutorials with only 7 subproblems) had a medium effect size for the calculus-based students, but even in that case, the paired problem average for the tutorial group is only 69%. Furthermore, one possible reason why students in the tutorial and nontutorial groups have no significant differences for the conservation of energy tutorial is that this tutorial is the longest involving two distinct subproblems and without deep engagement with the subproblems in the tutorial, students are unlikely to learn from it.  Table IV compares the performance of students in oneon-one implementation of the electronic learning tutorials with those who claimed they had used them as a self-study tool. Table IV shows a considerably higher average score for students in the one-on-one implementation group compared to those in the large scale implementation as a self-study group for all three tutorials. Thus, a noteworthy observation is that the one-on-one implementation group, composed of 12 students from large enrollment algebrabased courses and 10 students from calculus based courses, scored considerably higher than both large scale self-study implementation groups. This higher score is statistically significant between all self-study groups and interview groups for all tutorials with the exception of students in the calculus-based self-study group for the Newton's second law tutorial. We do not separate the algebra-based and calculus-based groups in Table IV since there were only 22 students including both groups in the one-on-one implementation group (due to the time required for each interview). The fact that the students in the calculus-based courses in a particular group generally outperformed those in the algebra-based courses (see Table III), it is likely that if we had sufficient number of students in the interview group from only the calculus-based courses, the comparison between the interview group and the self-study group for the calculus-based students for the Newton's second law tutorial would have been statistically significant. Also, the standard deviations are notably lower for the interview group than either self-study group. This is likely due to a combination of a ceiling effect in the interview group's paired problem scores and the varying degree to which selfstudy students engaged with the tutorials unlike those in the interview group.

V. DISCUSSION
We evaluated the relative effectiveness of the researchbased electronic learning tutorials when students worked on them as a self-study tool at their own discretion without supervision in large enrollment introductory physics classes as compared to in a deliberate one-on-one setting. Students making use of the tutorials in a deliberate one-onone interview situation worked on them with a researcher monitoring the students so that they used them as TABLE III. Comparison of the performance on the paired problem and standard deviations for students who used the tutorials as a self-study tool and those who did not use them in large, brick-and-mortar introductory physics courses. prescribed. The students were prompted to think aloud while working on them but otherwise were not disturbed. The think-aloud process was important for ensuring that students were engaged with the tutorial. In the self-study implementation, although students were instructed to follow the same guidelines for effective learning, they could potentially take a short cut and skip subproblems if they decided not to adopt a deliberate learning approach while using these research-based tools [28]. We found that many students in the self-study implementation group did not work through the tutorials, and if they did, the increase in average performance is not impressive with respect to the effects expected from these research-based electronic learning tutorials (evidenced by the performance on the paired problems of those in the individual one-on-one implementation group). This dichotomy between the performance of the self-study group and the one-on-one implementation group suggests that research-based tutorials, when used as intended, can be an excellent learning tool for introductory physics students across diverse levels of prior preparation, experience and mathematical background but getting students to engage with them effectively as a self-study tool can be challenging. Many factors pertaining to students' characteristics, social and environmental factors and the manner in which the self-study tools are implemented and incentivized can impact how effectively students engage with them. For example, one possible reason for the significantly better performance on paired problems among the one-on-one implementation group as compared to either the algebrabased or calculus-based self-study groups is the ineffective approaches to using the tutorial as a self-study tool due to individual, social and environmental factors. As noted, students were made aware (by way of emails, announcements on the course web-page and a description of the tutorials given to them verbally during their regularly scheduled class time) that working on the self-paced interactive tutorials posted on the course website does not contribute directly to the grade but that working through them deliberately will help them learn and improve their homework and quiz performance. Upon examining student comments and other data gathered with their response to the paired problems in the self-study group, it appears that some students who claimed to make use of the tutorials may not have used them effectively. Some students explicitly commented that they "skimmed" or "looked over" the tutorials but that type of engagement with the electronic learning tool may not help them learn physics. A detailed look at the performance of students enrolled in the introductory physics courses on the paired problems indeed suggests that many students may have memorized certain equations by browsing over the tutorials, expecting that those equations may help them in solving the in-class quiz problems, instead of engaging with the self-paced tools as instructed in a systematic manner. The tutorials strived to help students develop a coherent knowledge structure of physics concepts, practice effective problem-solving approaches, and transfer their learning to a different context, but some students in the selfstudy group focused instead on the equations that they thought could be useful on the in-class quiz problems. This approach of focusing on the equations used instead of deep learning when working through the tutorial is consistent with the approach many novice students use when classifying physics problems-they often classify physics problems based upon the surface features of the problem as opposed to the deeper concepts [29]. Interestingly, in a survey given at the end of the course to students who used the tutorials as a self-study tool, a majority noted that they thought that they were effective even though their performance on the paired problems reflected that they had not learned significantly from them.
Students' estimates of how much time they spent working on the tutorial also suggests that they were not productively engaged while working on the tutorial. For example, in one-on-one interviews, most interviewed students spent 15-30 min on the conservation of angular momentum tutorial. On the other hand, in the self-study implementation in the large introductory physics classes, some students reported that they spent up to 1.5 h on the same tutorial. However, the students in the one-on-one interview setting had an average score of over 80% on the angular momentum paired problem. This score is 45% higher than the average score of students in the self-study group in brick-and-mortar introductory physics courses. This dichotomy indicates that students in the self-study group may have engaged with the tutorials in a manner not conducive to learning (even though they reported spending a longer time working on the tutorial). If their self-reported time spent is accurate, the dichotomy may be due to possible lack of self-regulation, time-management, and focus among other things while working through the tutorial. Moreover, adverse social and environmental factors (e.g., students' other workload and/or social engagement, support and encouragement from family, and guidance of other mentors and advisors) and lack of sufficient incentive to engage with the self-study tools can also prevent students from using them effectively.
This study suggests that even when research-based selfstudy tools take into account students' prior knowledge and skills and provide appropriate scaffolding support to a variety of students, many students may not follow the guidelines for effectively using them. The haphazard use of these research-based self-paced tools can reduce their effectiveness significantly. The significantly lower performance of students in the self-study group compared to the controlled group in this investigation supports the hypothesis that major challenges in implementing research-based tutorials as self-study tools are likely to be issues such as students' level of motivation, focus, self-regulation, and time management to engage with them, along with the social and environmental factors governing students' lives and the manner in which these self-study tools are implemented and incentivized [30][31][32]. Many students have difficulty internalizing that much of the value to be gained from these self-paced tools depends on them interacting with them in a prescribed manner. For example, students who explicitly reported having skimmed through the tutorials most likely did not engage with each of the individual subproblems as they were prescribed to do. Additionally, they may not have attempted to solve the tutorial problem on their own without the scaffolding provided by the tutorial as they were asked to do before starting to work on the tutorial. Although the instructions for effective usage of these self-study tools were provided to students through several channels, many students may have interacted with the self-study tools only superficially. Even among students observed in deliberate one-on-one interviews, some had to be prompted several times to make a prediction for each subproblem and articulate their reason for selecting an answer before selecting an answer rather than randomly guessing an answer. We hypothesize that in addition to providing help to develop self-management and time-management skills and incentives to motivate students to engage with the self-paced tutorials more effectively, additional supports that take into account the social and environmental factors governing students' lives are needed.
The fact that the tutorials were ineffective as a self-study tool (even though they were effective in deliberate, one-onone administration for a variety of students) attests to the difficulty in making any research-based online learning tool effective for a diverse group of students. The data from online courses suggest that the students from low socioeconomic backgrounds have a high attrition rate. Moreover, not only do online courses [33][34][35][36][37] use electronic learning tools, but the instructors in a typical brick and mortar course often integrate web-based components as learning tools similar to the study described here. Many instructors realize that self-paced, out-of-class learning tools are critical even for a traditional, brick-and-mortar course, especially since these self-paced tools address some of the challenges involved in educating students with diverse motivation, goals, prior preparation, and backgrounds in a given course [38][39][40][41][42]. These instructors may aim the lectures at an average student in the course and assume that the students below the class average at a given time will catch up using the self-study tools they prescribe. However, many students may not engage with them as prescribed, as suggested by our investigation.
Apart from the self-regulation and time-management skills necessary to hold oneself accountable for learning from the self-study tools, social and environmental factors and motivation can play a critical role in whether students take advantage of these tools. It is therefore important to contemplate different facets of learning from self-paced learning tools in order to provide learning environments with appropriate support and incentives for students to benefit from them, especially if they are research-based tools that have been found effective in one-on-one implementation. Below, we propose a theoretical framework for this purpose.
The framework, which we call the SELF, is proposed to create learning environments to help students with diverse backgrounds benefit from self-study tools. Our framework consists of four quadrants as shown in Fig. 3; all of them must be considered holistically in order to help a diverse group of students learn effectively from self-study tools. The left two quadrants focus on the characteristics of the self-study tool and the right two quadrants focus on the characteristics of the user, or student. The self-study tool or user can have "internal" or "external" characteristics that can support or hinder engagement with the tool and learning. For example, in the upper left hand quadrant, the internal characteristics of the self-study tool pertain to the tool itself-how a research-based design of the selfstudy tool based upon a cognitive model and appropriate discipline-specific attributes can help students learn. In the lower left hand quadrant, the external characteristics of the self-study tool pertain to how the tool is implemented and incentivized-how additional characteristics of the implementation of the self-study tool can motivate and help students effectively learn from the tool. In the upper right hand quadrant, the internal characteristics of the user (student) that can impact learning from self-study tools involve the students' prior knowledge, skills, affective characteristics, and epistemological beliefs. In the lower right hand quadrant, the external characteristics of the user that impact learning from self-paced tools pertain to social and environmental factors-how the user-environment interaction supports or hinders effective usage of self-study tools.
The factors listed in the four quadrants in Fig. 3 can be thought of in terms of affordances or constraints [43,44]. Affordances support effective interactions between an individual and a tool, whereas constraints hinder effective interactions between the individual and tool. For example, in physics and other domains, such as chemistry, computer science, and engineering, different representations offer different affordances and can either help or hinder effective problem solving and learning [45]. Also, in these disciplines, learning and developing competency in problem solving includes a deep conceptual understanding and the constitution of the representations used to present problems and to solve them have been argued to be critical aspects of interactive learning [45]. An effective use of mathematical symbolism can afford logical reasoning in a precise form, whereas visual representations can afford spatial relationships. Moreover, representations such as written language, pictures, and diagrams may also afford more interactive engagement than spoken language, gestures, and facial expressions [45]. Providing opportunities for students to learn how to use various representations effectively and recognize connections between different types of representations can also afford interactive engagement with learning tools and promote expertlike problem solving [45]. Affordances and constraints can also be classified into internal and external categories. Simon notes that a tool can be thought of as a meeting point between an "inner environment" (the characteristics of the tool and user) and an "outer environment" (the surroundings in which the tool and user operate) [44]. For example, in order to land a plane safely, the internal affordances could refer to the features of the plane itself while the external affordances could refer to the features of the runway, and weather on a particular day when a pilot is landing the plane. In our framework, the internal and external characteristics of the tool and user can be viewed as affordances or constraints depending upon whether they support or hinder learning from self-study tools. With regards to the framework in Fig. 3, a researchbased self-study tool can provide affordances such as appropriate scaffolding support and elements of efficiency and innovation that support student learning. However, a self-study tool that is not designed carefully via research can constrain student learning, e.g., if it does not take into account students' prior knowledge and does not have adaptive features to cater to helping a variety of students learn, it can hinder learning. Similarly, a student's level of motivation can be viewed as an affordance if the student has the requisite motivation to succeed in the course. On the other hand, a student's motivation can also be viewed as a constraint-if the student does not have the requisite motivation to engage with the self-study tools.
Most of the research-based self-paced electronic learning tools developed so far have mainly focused on the upper two quadrants of the framework in Fig. 3. In the upper-left quadrant, the focus is on the self-study tool characteristics that directly focus on knowledge and skills to be learned via the tool. For example, the cognitive apprenticeship paradigm [20] can inspire the development of adaptive selfstudy tools that provide coaching and promote mastery for a variety of students [46]. These self-study tools, when developed carefully via research, can provide appropriate scaffolding support to a diverse group of students. In order to make the self-study tools effective, educators often consider the user characteristics in the upper-right quadrant [47][48][49][50][51][52][53][54]. The various models of learning lead to similar conclusions about how to connect user characteristics with the characteristics of the self-study tools (i.e., how to connect factors I and II). For example, Schwartz, Bransford, and Sears' [55] preparation for future learning Factors that promote self-regulated learning Tool characteristics User Characteristics

Internal Characteristics
Factor I. Self-study tool characteristics (internal)pertaining to how the tool focuses on knowledge / skills to be learned • Develop adaptive tools based on "cognitive apprenticeship model" to promote mastery of material for a variety of students model emphasizes that in order to develop adaptive experts, there should be elements of both efficiency and innovation embedded in the instructional tools and design. One interpretation of this model is that if the students are asked to engage with learning tools that are too efficient, they will get bored and disengage. On the other hand, if the learning tools are too innovative, students will struggle so much while engaging with them that they will become frustrated and give up. Thus, the learning tools and instructional design should blend elements of both efficiency and innovation, allowing a variety of students to engage and struggle productively while learning, and to develop adaptive expertise [56,57].
In addition, effective self-study tools should have formative assessment built into them so that students can receive feedback and evaluate their own learning as they make progress. Since student characteristics within a particular class vary, carefully designed interactive self-study tools can develop adaptive experts by providing appropriate balance of innovation and efficiency for a variety of students [58][59][60][61].
Students who are lacking some elements of prior knowledge can benefit from a carefully designed self-study tool that involves formative assessment, allows students to make mistakes but learn from them and try again, and scaffolds their learning [62][63][64][65][66][67][68][69].
In the study described here, the research-based tutorials included considerations of factors I and II. For example, the tutorials provided an opportunity for productive struggle-they specifically encouraged students to work on each tutorial problem before starting to work on each of the subproblems. The act of struggling with the tutorial problem can help students connect what they are learning with their prior knowledge and aid in learning. Additionally, struggling with the tutorial problem before engaging with the tutorial may increase students' motivation to engage deliberately with the electronic learning tool as prescribed. However, the tutorials could also be improved based upon consideration of factors I and II. For example, the longer tutorials were more complex since they either involved application of more than one physics principle or application of the same principle (Newton's second law) in different contexts. These longer electronic learning tutorials are useful for helping students develop both content knowledge and skills to solve complex problems. However, since many students may have disengaged with the longer tutorials while using them as a self-study tool, finding better ways to keep students motivated throughout while working through them should be a high priority rather than only developing shorter tutorials focused on one physics concept or principle [70]. Based upon considerations of factors I and II, one strategy that may make them more effective is to break the multiprinciple tutorials into single-principle tutorials. After working through the single-principle tutorials, students can then work through a subsequent multiprinciple tutorial that combines the learning in those single-concept tutorials.
Since students would have been exposed to the individual concepts in various single-concept tutorials, they will be more likely to effectively engage with the multiprinciple tutorial that consolidates those principles into a more complex problem. However, research-based self-paced learning tools that take into account students' prior knowledge will not necessarily help them learn if students do not take advantage of the self-study tools to learn in an effective manner. While the top two quadrants in our framework are often considered in the development of self-study tools, the lower two quadrants of our theoretical framework for learning using self-study tools have mostly been ignored while designing and implementing most self-study tools even if those tools are carefully developed. But the study presented here suggests that these lower two quadrants are likely to play a critical role in whether students, who are especially in need of remediation via self-study tools, take advantage of these self-study tools. The lower right quadrant or factor IV focuses on external student-environment interaction characteristics, e.g., how students interact with their surroundings and how they manage their time and regulate themselves. For example, if the students get bogged down with unimportant activities (e.g., communicating with friends on social media), they are unlikely to make time for activities that are important (e.g., learning from selfstudy tools). Factor IV also involves the encouragement and support that students may receive from their environments such as help from family, advisors, mentors, and counselors, e.g., to manage their time better and engage in learning using self-study tools effectively. In our study, students' engagement with the self-study tools may have been impacted by social and environmental factors including their external workload and social engagement and whether they have self-management skills, time-management skills, family encouragement, and support from other advisors and counselors.
The question then boils down to whether there are external affordances that can be provided during the implementation of the self-study tools to assist students who otherwise may not engage with them effectively due to personal constraints. This external additional support from educators for self-regulation and effective use of the selfstudy tool is included in the lower left quadrant (factor III) and focuses on providing motivation and support for engagement, taking into account the user characteristics and user-environment interactions. Consideration of the various types of support in quadrant III during the implementation of the self-study tools is critical to ensure that most students engage with the self-study tools effectively. In our study, the only component of factor III that we took into account was that the self-study tutorials were connected to in-class and out-of-class assignments, i.e., students were told that the self-study tutorials would be helpful for solving homework problems and in-class quiz problems for that week. However, the connection between the self-study tools and the in-class and out-of-class assignments may not have been explicit to students. The students may have engaged more effectively with the electronic learning tutorials if more elements from factor III were included in the implementation of the self-study tools. For example, making explicit connection between self-paced learning and other in-class lessons or out of class assignments and assessments may help students engage with the self-study tools more effectively. Furthermore, self-study tool developers or implementers can consider embedding modules that focus on motivating students to engage with the self-study tools effectively and strive to get buy-in from students by having them think carefully about why they should engage effectively with these tools and how they can help them in the long term. Similarly, the students who are struggling to manage their time well can be provided some modules to guide them in making a better daily schedule which includes time to learn from the self-study tools (for example, once students have made a schedule that includes time slots for learning from selfstudy tools, electronic notifications can remind them of their schedule as needed).
One factor (see factor III of the framework) that may help students engage with these tools effectively is participation in learning communities consisting of students, all of whom are expected to learn from the self-study tools, and then having them engage in some follow up activities in a group environment (this group work can be done online or in person depending on the constraints of the class). In this way, individual students may feel more accountable to their group members and effectively use self-study activities to prepare for the group activities. For example, in the study discussed here, encouraging and incentivizing students to work in these types of learning communities could have aided students in engaging with the self-paced electronic learning tutorials more effectively. In particular, if students knew that they were assigned to work with a group on a complex physics problem, they may have had more motivation to work through the electronic learning tutorials individually in order to prepare for the group work.
Moreover, having more effective grade incentives [71,72] to learn from the self-study tools is another external factor that can also increase student engagement (see factor III of the framework). For example, to help students engage effectively with the electronic learning tutorials, an instructor could incentivize participation in learning via grade incentives to ensure that students work on them as prescribed. Also, if students work systematically on them and are engaged throughout, they are unlikely to have cognitive overload [73,74] since learning is scaffolded throughout and one subproblem builds on another. One motivating factor would be to award course credit to students based on their answers to each subproblem with decreasing score if they guess multiple times. This strategy might be more successful at motivating them to answer each subproblem carefully (as opposed to randomly guessing an answer) while working through the electronic learning tutorial. On the other hand, the inclusion of grade incentives that decreases based on attempts may have the unintended consequence of making highly grade conscious students less willing to make mistakes and learn from the provided scaffolding and may drive some to engage with the tutorials in ways that ensure that they receive full credit but do not engage with the material for understanding and learning. In addition, it is possible that if students in the study described here were asked to submit a copy of their answers to each subproblem of the electronic learning tutorial and explain why each alternative choice to each subproblem is incorrect as part of their homework, it may have increased their motivation to engage with these selfstudy tools (especially because students have many conflicting priorities for their time and they may not engage with self-study tools if working through them is not directly tied to the grade).
Moreover, students who have difficulty engaging with the self-study tools due to lack of self-efficacy or unproductive epistemological beliefs [75][76][77][78] about learning can be guided to help them develop self-efficacy [19] and productive epistemological beliefs. For example, a short online intervention has been shown to improve student self-efficacy significantly [19]. Similarly, students who have unproductive epistemological beliefs (such as physics is just a collection of facts and formulas, only a few smart people can do physics, and they should just memorize physics formulas and regurgitate them) are unlikely to productively engage with the self-study tools designed to help them develop expertise in physics. It is important to address these issues in order to ensure that students who are most in need of learning using self-study tools actually benefit from them and retain what they learn [79][80][81][82][83][84][85][86][87][88][89].
We note that, in our framework, factor III may also impact factor IV. When students are motivated to think about the importance of using self-study tools, given credit to work through the self-study tools, work in learning communities that keep each student accountable while providing mutual support, and can discern the connections between the self-study tools and in-class assignments, homework, and quizzes, they may manage their time more effectively. Connecting self-study tool content to real-world applications can also increase student motivation to learn from these self-paced tools. It is also important to note that factors I and III can impact factors II and IV so we cannot disentangle any of these factors. Students who are lacking prior preparation may also have difficulty in managing their time effectively. But there are often students who are prepared to learn using self-study tools but lack time management skills. Other students may not have good prior preparation but they may have good time management skills. In all these cases, in order to help students learn effectively from the self-study tools, the affordances should outweigh the constraints. Therefore, consideration of factors III and IV, which are often ignored by educators developing self-study tools, is critical.

VI. SUMMARY AND CONCLUSIONS
We compare the effectiveness of three electronic learning tutorials when used as a self-study tool in large enrollment classes with their effectiveness when they were used by students in a closely monitored, deliberate, one-on-one setting. The students who used the electronic learning tutorials in a one-on-one setting were instructed to work on them following the same instructions as those provided to students using these tutorials as a self-study tool, but they were monitored, i.e., they had to follow the instructions and could not skip any part. After working on the tutorial (either as a self-study tool or as a deliberate one-on-one electronic learning tool), students' knowledge of the associated physics concepts was evaluated via their performance on an associated paired problem that involves the same physics principle or concepts as the electronic learning tutorial problem. We find that students in the deliberate, one-on-one implementation group significantly outperformed those in the self-study group on the paired problems.
The fact that students had to follow the correct protocol in the deliberate one-on-one implementation (i.e., start by solving the problem without any help and then work on the tutorial as intended) may have contributed to the success of the tutorials in deliberate one-on-one interviews. These students may have benefited from being isolated from outside demands on their time for the duration of the interview allowing them to focus and engage properly with the tutorial. On the other hand, the lack of effectiveness when students used the tutorials as a self-study tool is likely due to students engaging with the tutorial in ways other than those outlined for them or outside demands on the student's time from other classes, work, and family or social commitments, resulting in the tutorials being deprioritized and thus not properly engaged with.
We propose a framework called SELF, to create learning environments to help students with diverse backgrounds benefit from self-study tools. Our framework consists of four quadrants which must be considered holistically in order to help a variety of students learn effectively from self-paced learning tools. Two of the quadrants of the framework focus on the characteristics of the self-study tool and two quadrants focus on the characteristics of the user. The internal or external characteristics of self-study tool or the user can support or hinder engagement with the tool and learning. This framework can be applied to other disciplines such as chemistry, computing, engineering, etc., to scaffold student learning. In particular, students in all of these disciplines interacting with even the best designed self-study tools are likely to do so in ways other than those prescribed explicitly, which can greatly diminish the tools' effectiveness. This limitation is inherent to self-study tools that have no means of regulating the ways in which the student interacts with them unless issues discussed in our framework in the lower left quadrant are explicitly incorporated. Future studies can further inform the SELF framework by investigating how students interact with self-study tools via interviews in which students can use the tool in whatever manner they wish (instead of as prescribed), and then asking them additional questions focusing on how they generally engage with self-study tools and why.
A lack of sufficient motivation, discipline, selfregulation, and time-management skills while engaging in learning using self-study tools as well as social and environmental factors including competition for student time from outside commitments may turn out to be the biggest impediments in implementing research-based electronic learning tools for use as self-study tools. The theoretical framework proposed emphasizes that in order for students with diverse backgrounds and prior preparations to benefit from self-study tools, educators must holistically consider various facets of student engagement with self-study tools and incorporate them in their development and implementation of those tools.

ACKNOWLEDGMENTS
We thank the U.S. National Science Foundation for Grant No. 1524575, and members of the University of Pittsburgh physics education research group for their help with the rubric development and grading to establish interrater reliability.