Framework and implementation for improving physics essential skills via computer-based practice: Vector math

We propose a framework for improving accuracy, fluency, and retention of basic skills essential for solving problems relevant to STEM introductory courses, and implement the framework for the case of basic vector math skills over several semesters in an introductory physics course. Using an iterative development process, the framework begins with a careful identification of target skills and the study of specific student difficulties with these skills. It then employs computer-based instruction, immediate feedback, mastery grading, and well-researched principles from cognitive psychology such as interleaved training sequences and distributed practice. We implemented this with more than 1500 students over 2 semesters. Students completed the mastery practice for an average of about 13 min =week, for a total of about 2–3 h for the whole semester. Results reveal large (>1 SD) pretest to post-test gains in accuracy in vector skills, even compared to a control group, and these gains were retained at least 2 months after practice. We also find evidence of improved fluency, student satisfaction, and that awarding regular course credit results in higher participation and higher learning gains than awarding extra credit. In all, we find that simple computer-based mastery practice is an effective and efficient way to improve a set of basic and essential skills for introductory physics.


I. INTRODUCTION
Learning how to solve problems is a critical goal for physics courses, and has thus been the focus of considerable efforts in research and instructional improvement (for reviews see Refs.[1,2]).While problem solving can be a very complex process, in this paper we will focus on the hypothesis that problem solving requires accuracy and fluency in relatively simple, elementary skills, such as vector addition and unit conversion.By investigating the example of basic vector math skills, we will argue that students frequently do not have these simple yet essential skills, or they are far from fluent in their use.In our review of the literature and in our professional experience, we have found very few examples (discussed below) that such essential skills are typically systematically or extensively practiced in introductory physics courses.One might speculate that is because many instructors are not aware of the need, believe that it is prerequisite knowledge for the course and, thus, is each student's individual responsibility, believe that it is sufficiently practiced in the course already, and/or do not have sufficient time to devote to the issue.Regardless of the reasons, we suggest that the lack of mastery or fluency in these elementary essential skills may be a significant obstacle to the process of learning physics and how to solve physics problems.
In this study, we provide a theoretical background, justification, and framework for the improvement of students' "essentials skills" via explicit practice, and we assess the implementation of an online instructional intervention in a real course setting for an example essential skill set, namely, the vector math skills necessary for solving physics problems in first year introductory physics courses.The development of the materials for the online vector essential skills units is a continuation of our recent work on training students on vector skills in an education research lab setting [3].Further, we are continuing work on essential skills units for a course on introductory materials science engineering [4,5].Here we will demonstrate, as part of a regular course, an instructional intervention to help students achieve and retain significant gains in mastery and fluency of a set of essential skills with a relatively small commitment of student time.
The essential skills practice assignments in this study are in an online format, which lends itself well to our goals, as described more in Sec.III.A considerable number of computer-based instructional components of courses have been developed and investigated in a variety of physics education research studies, but the combination of methods, goals, and focus of training on essential skills in this study is relatively unique.For example, in addition to numerous commercial systems, a number of online physics homework systems with varying levels of instant feedback have been developed that have proven to be useful instructional tools [6][7][8].However, these systems and studies have been aimed at providing practice and feedback for standard physics problems rather than more basic skills necessary to solve the problems of interest in this study, although these systems might easily be adapted to teach essential skills in the way studied here.
Recently, Schroeder et al. [9] and Gladdening et al. [10] have implemented something more similar to our work and focus, namely, computer-based mastery practice of specific skills in an education research lab setting and have shown it to be effective.The target skills in their studies tended to be specific to a particular physics problem type and subtopic (e.g., electric fields and potential for plane and spherical symmetries) and the studies focused on accuracy, while our target skills tend to be more broadly applicable (though not limited to this) and, as we will discuss below, we are aiming at accuracy, fluency, and retention.
Perhaps the most closely related work involves the (now commercial) online intelligent tutor ALEKS, which provides a system of mastery learning with instant feedback and adaptive practice sequences based on the student's responses on all prior questions answered in the system [11,12].The sophisticated adaptive-response design is based on a well-developed theory of knowledge spaces [13], and the system enjoys widespread use, especially in the K-12 realm.ALEKS is primarily designed to teach (for a fee) basic math, including arithmetic, algebra, trigonometry, and vectors, and has developed a unit on "math prep for college physics."Many of the goals of ALEKS to improve basic skills are similar to the goals of this study, including the topic of vector math.To our knowledge ALEKS has not been researched for effectiveness in a physics course context, nor is it based on other physics education research on physics student difficulties with essential skills relevant to physics, which is a critical component of the design process in this study.Further, our system is much simpler to construct and expand since it does not use adaptive response.
In future work, it may be worth comparing the relative effectiveness of the different systems and approaches mentioned above, but in all the results of these studies indicate that computer-based instruction may be an effective tool for teaching a variety of simple skills and knowledge for a physics course.Here we describe our specific design and implementation in a course setting, and report on assessments of its effectiveness.

II. ESSENTIAL SKILLS CHARACTERISTICS AND RATIONALE
We define an essential skill (ES) for a given course as having the following characteristics: • The skill is procedural and relatively simple, requiring at most a few steps.• The skill constitutes some steps necessary to solve commonly assigned problems.• The skills are largely automated in experts.When solving a physics problem, experts and novices alike must make effective use of a number of essential skills.However, the ways in which experts and novices interact with these problems are very distinct.An expert has largely automated these procedural steps, and thus proceeds further into the problem having expended relatively little time and cognitive load [14][15][16].Automated processes are rapid cognitive processes that occur at the subconscious level [17,18], and experts tend to automate a variety of tasks and skills through years of practice and application in their domain.
Novice students often have not automated the essential skills, resulting in increased time on task as well as increased cognitive load.Even if the novice correctly solves the procedural step-and they often do not-they do so slowly and with effort and are left with fewer cognitive resources available to attend to the rest of the problem and its conceptual underpinnings [19,20].That is, students often lack the necessary accuracy with the essential skills, and even students who are accurate with the skills are often not fluent enough in their use to be successful problem solvers.The term fluency in a skill is meant here as including a relatively short completion time, minimal cognitive load, and parallel (vs serial) processing (cf.Ref. [21]).
The central idea of this study is that students will benefit from both high accuracy and high fluency in essential skills.If essential skills are automated and students become fluent in their use, the reduced cognitive load presented to students will allow them to solve problems more efficiently and effectively.This idea is not new-others have emphasized the difference in cognitive load between experts and novices for commonly used skills, and that improving fluency in these skills is critical to expertlike performance in problem solving [21][22][23].Further, "procedural fluency" has been identified as one of the five strands of mathematical proficiency in the National Research Council report on mathematics learning for children [24].Nonetheless a general framework is needed to prescribe a procedure for developing ES practice for a given topic.This is described in the next section.

III. THE ESSENTIAL SKILLS DEVELOPMENT FRAMEWORK
In this section we propose a set of goals for ES improvement and an iterative ES development framework to achieve these goals.

A. General goals of essential skills instruction
There are four general goals for ES instruction, shown below.
(1) Maximize accuracy, fluency, and retention of essential skills.This is aimed at reducing cognitive load while solving problems (e.g., Ref. [20]).( 2 Improvements to self-efficacy are a critical component of STEM learning [25,26].In this study we were only able to collect anecdotal evidence for achieving this goal.This topic needs further examination in future studies.

B. Components of the essential skills framework
The ES development framework consists of four components:

Identifying essential skills
Naturally, one might imagine a multitude of elementary skills required to be successful in a physics course (reading, arithmetic, language mastery, etc.).Here we are interested in those basic skills in which a significant fraction of students have not attained a sufficient level of accuracy and fluency.Candidates for such skills can originate, for example, from discussions with experienced instructors, from direct observations, or from the education research literature, but ultimately empirical verification is needed.In pilot and formal studies in our own lab (some discussed below), we have found that it is not uncommon to find unexpectedly low levels of accuracy and fluency in even the most basic skills.Typically, we have found that essential skills topics involve applications and simple conceptual (and physical) understanding of basic calculation procedures, including basic math and interpretations of representations.Examples include vector math, interpreting graphs and diagrams, and simple arithmetic and algebra.

Identifying common difficulties and errors with essential skills
Similar to strategies used for improving conceptual understanding and problem solving in physics education (for a review see, e.g., Ref. [1]), we find that a systematic characterization of specific student difficulties with an essential skill is useful for designing targeted, effective practice.In our experience [4,5,27,28], we have found that difficulties with skills in a given domain (such as reading logarithmic graphs) tend to be multidimensional and interrelated, even for relatively simple skills, and exploring each dimension can lead to a robust set of practice examples that "triangulate" the issue by addressing the difficulty from different perspectives, dimensions, and representations.Of course incorrect student answers also provide useful material for distractors on practice questions.
Even if an instructor or researcher has experience with common student difficulties with a given essential skill, a systematic exploration of these difficulties is strongly recommended.In our experience we have always found the difficulties with skills to be more complex than first expected.For example, in the case of vector math, we found that some vector math skills (such as addition) must be broken further into subskills (e.g., finding trigonometric components of vectors), and in this case an exploration of difficulties with these subskills is critical [27,28].
The research literature often contains useful characterizations of student difficulties for a given physics topic, and there are a variety of methods used to identify and characterize difficulties (see a review, e.g., in Ref. [1]) Generally, difficulties are explored via an array of openended problems related to a given essential skill, using various representations and exploring different variables (dimensions) used in the skill.Typically, the problems are administered in written and interview format in order to observe the rich array of student responses, and this is followed up by investigations using multiple-choice questions administered to a large pool of students in order to determine the relative frequency of particular difficulties.Note that this process also can lead to the development of an assessment instrument to evaluate the effectiveness of essential skills units.

Employing research-validated methods in the design of ES practice
The essential skills practice units employed a number of research-validated principles and methods, described below.
Computer-based training with feedback.-Inorder to improve student skills with relatively little time (10-20 min=week) spent outside of class, we adopt the online, computer-based practice format.This format allows students to work towards accuracy and fluency on their own time and at their own pace.Furthermore, computer-based training with feedback has proven to be effective in a wide range of topics [29,30], allowing for an array of feedback formats, providing individualized feedback to each student.However, the effectiveness of feedback depends on several interacting factors, including the knowledge or skill to be learned, the type (e.g., complexity) of feedback, timing of feedback, prior knowledge of topic, student achievement, correctness of and confidence in responses, interest in topic, self-efficacy, and other attitudinal factors (for reviews, see Refs.[29][30][31][32][33][34][35][36]).
Here we will base the practice and feedback format on results from our recent clinical study which investigates the relative effectiveness of various forms of feedback on the same population as the current study (students enrolled in calculus-based introductory physics) and the same topic, namely, vector math [3].Following those results, the method used in the current study provides immediate feedback on the correctness of each response and also makes available a general explanation (via clicking a button if the student chooses to) of how to answer a particular question type.Note that this method uses only simple answer-based feedback and does not rely on more complex intelligent tutoring systems.Specific examples are provided in Sec.IV.
Mastery-based training.-Mastery-basedtraining-which involves allowing time on task to vary between students in order for everyone to achieve some predetermined standard of performance-is more efficient and flexible for a multitude of student backgrounds compared to conventional summative course assignments [37].Students already fairly fluent with a given essential skill can complete the task relatively quickly (typically within 5 min in this study), thus avoiding possible frustration, yet they are still given the opportunity to practice and potentially improve fluency.In contrast, students who need significant practice with an essential skill may have varying needs and typically require longer practice times (20-30 min) to achieve sufficient accuracy and fluency for a given skill.
Distributed and interleaved practice.-Distributingpractice repeatedly over the whole term-as opposed to practicing in one session-results in learning that is stable over longer time intervals [38][39][40].Such durability of learning is crucial for the essential skills domain, as these skills are frequently used for the entire course, and in future courses within the same discipline.Interleaving problem types (as opposed to blocking all problems types together) within a practice session improves learning, as well as student ability to distinguish between and appropriately apply the different types of similar skills [39].
Multiple representations.-Asmentioned in Sec.III B 2, the way in which essential skills such as vector addition are represented can affect student responses (for examples pertaining to vector math see Refs.[27,41,42]).If students work with only one representation of the essential skills, their learning may become tied to this representation.By mastering the use of multiple representations of individual skills and making explicit the connection between these representations, student gains on the essential skills are more likely to transfer to other aspects of course work, such as problem solving or conceptual understanding [43,44].

Iterative development of units
We employed the process of iterative development (in a cycle of implementation and assessment commonly used in physics education research, see, e.g., the review in Ref [1]) to improve the effectiveness of the units, and this was particularly helpful for more complex skills.For example, for some skills we found that after a round of implementation and assessment, many students still did not master a specific skill.Further analysis and development often resulted in breaking the skill down into subskills to be mastered.

IV. EXAMPLE APPLICATION OF THE FRAMEWORK: VECTOR MATH
In this study, we applied the ES framework to vector skills, namely, the topic of simple vector math operations necessary for introductory calculus-based physics.This included vector addition, subtraction, components, tilted coordinate systems, dot products, and cross products in both algebraic and arrow representations and sometimes in contexts similar to physics problems (e.g., simple free body diagrams).Vector math meets our required characteristics of an essential skill in that it is a relatively simple procedural task, is required to solve a number of physics problems, and is a fluent skill of experts.
In fact, simple vector math is so central to introductory physics that most, if not all, relevant physics textbooks include an early chapter on the topic, but it is clear that this is not sufficient.Specifically, the research literature contains a number of papers detailing student difficulties with various vector operations (e.g., Refs.[27,28,[48][49][50]), showing that typically-even postinstruction-only 50%-70% of calculus-based physics students can correctly perform these basic vector arithmetic operations.These student difficulties produce predictable and consistent wrong answers that can be used as realistic distractor options when designing training and test questions.In some instances, it was necessary to collect our own data to better understand student difficulties and common errors with specific vector skills (e.g., Refs.[27,28]).

A. Student population and study design
The study consisted of three experiments conducted at the Ohio State University, a large public university, over the course of several academic terms (Fig. 1).The purpose for the experiments was to establish baseline data (with no training), to investigate potential improvements in essential skills, to demonstrate replicability and retention of results, and to determine whether the method of awarding credit for the assignment significantly affected the results.Students in experiments 1 and 2 were enrolled in the first term of introductory calculus-based physics, in two separate semestersautumn and spring, consecutively-with no students participating in both experiments.While there may be small differences in the ability and preparation of two populations, our past experience with students enrolled in the two different semesters (and results from this study) have not shown any significant differences in performance or demographics between these populations.The courses were taught by faculty and senior lecturers in traditional format with 3 traditional lectures and one recitation (solving problems and taking quizzes) per week.The demographics were ∼80% male and <10% underrepresented minorities.The authors were not instructors for any of the course sections during the study.Of students who took the final exam for each course, a total of N ¼ 1056 students in experiment 1 and N ¼ 265 in experiment 2 completed both the pretest and the post test-not graded, course credit awarded for participation.These students were included in the study, representing roughly three quarters of all students in each course who took the final exams.Students in experiment 1 completed the assignments for 1% of their final grade, while experiment 2 students were given 1% extra credit for participating.In experiment 3, a subset of N ¼ 61 randomly selected experiment 1 students were brought in during the succeeding course to compare their performance with a sample of N ¼ 40 randomly selected students from the same course who had not previously been trained.Finally, in a preparatory experiment, baseline data from a total of N ¼ 194 participants from an autumn semester were collected on performance with vector skills near the end of the same first semester calculus-based course in which none of the students were assigned ES practice.These participants completed the task for partial course credit (∼1% of grade) and were randomly selected from all students taking the

B. Materials 1. ES practice online application
Students completed the ES practice assignments via an online application that we created and developed using the principles described in Sec.III.Table I shows the array of vector skills practiced, all chosen from the research discussed at the beginning of this section.Screenshots of example ES practice questions of three of the skills are shown in Fig. 2. The appendix also shows a variety of questions that were very similar to ES practice questions.
Each assignment consisted of 2-5 individual skills, and most skills were included on at least two assignments throughout the semester.Students were awarded credit for each skill by answering several (typically 3 or 4) consecutive questions correctly within a given skill to achieve "mastery."If a student missed a question, their score for that skill reset to zero.Students were able to retry indefinitely, with items randomly selected from a large pool of items, until they achieved the required number of consecutive correct answers.Note that the skills practiced were also interleaved such that, for example, the session FIG. 2. Example practice questions for addition (top right), components vasic (top left), and cross product direction (bottom) skills.Current student scores for each skill are recorded on the "progress wheels" to the left.Incorrect answer (red circle "x") results in the score returning to zero for that skill.
would consist of a practice question first from skill A, next from skill B, and so on, cycling through the skills until mastery of a skill was achieved, at which point that skill was excluded from the cycle.In Fig. 2, this would correspond to first getting a question on vector addition, then one on subtraction, then negation (finding the negative of a vector), then component basic (finding the x or y components of a vector), before cycling back around.As discussed more in Sec.V, we found that students typically take between 5 and 20 min to complete an assignment.
After submitting an answer, immediate answer-based feedback (correct or incorrect) was given after every problem.If a student answered the problem incorrectly, they could choose to try the same problem again up to 3 times in total.If a student submitted three incorrect answers to a single problem, the system allowed them to see the correct answer.Furthermore, after answering correctly or incorrectly, students could choose to view explanations, which popped up in another window of the application.The explanations included text and graphics and did not pertain to the specific problem or the student's submitted response, but rather outlined the correct procedure or rule for a given skill and could be applied to a class of similar problems.Each type of problem contained at least two explanation images.See Fig. 3 for an example.We have found this method of using explanation images to be highly effective [3].
The system made use of randomized questions that were either generated in real time via a mathematical algorithm or chosen randomly from a pool of similar sample questions.For example, a vector addition question would randomly generate two vectors to be added.
Many of the problem types included two answer modes (practiced separately): multiple choice or free response.The multiple choice options were generated via an algorithm that determined the correct answer for that question and incorrect answers known from research to be common.Often, the number of possible common incorrect answers was larger than the number of incorrect multiple choice options.Thus, the errors represented by the wrong answer choices were varied from problem to problem within the same type, minimizing the ability of students to simply pattern match.Free response options depend on the specific question asked, but they included typing in numbers and using a computer mouse to draw a resultant vector on a grid.Note that the practice questions used different representation of vectors (algebraic ijk format and arrows on grids) to provide practice in multiple contexts [27].

Assessment
Two assessments were used over the course of this study, both of which were administered online through the university's course management software.The first was a 26 item "vector skills assessment" of student ability to perform simple vector addition, subtraction, component, dot product, and cross product calculations.The questions in the assessment were similar-but not identical-to those included in training assignments.A pencil and paper version of the assessment is included in the Appendix.The reliability of the assessment is acceptable, with Cronbach's alphas of 0.73 and 0.77 for pretest students in experiments 1 and 2, respectively.Only 2 of the 18 items had pretest scores less than 0.3, and all remaining questions fell between 0.3 and 0.75.Point biserial correlations were also at an acceptable level; all values were between 0.2 and 0.6 for both experiments, typically on the higher end of that range.
The validity of the assessment is supported in four ways.First, most of the items were based on similar simple skills questions in textbooks and validated assessment instruments [3,48,49].Note that the Test of Understanding Vectors [48] overlaps significantly with our vector skills assessment, but the former is aimed more at assessing conceptual understanding, and the latter is more aimed at assessing accuracy in vector math skills.Second, many of the items had also been pilot tested and refined through testing in our education research lab, including student interviews [3,27,28].Third, pretest assessment scores were significantly correlated with grade in the course (r ¼ 0.36, 0.32; p 0 s < 0.001) for experiments 1 and 2, respectively.Finally, the assessment was constructed by instructors (the FIG. 3. Example explanation for the topic of simple vector components. authors) with extensive experience teaching the topic of vector math in introductory physics.
The second assessment, administered on the post test, was a series of Likert scale items about the effectiveness and fairness of the training, and students were given an opportunity to leave open-ended feedback.The items are discussed in Sec.V G.

A. Completion rates and practice time
A histogram of completion rates of weekly ES assignments is shown in Fig. 4. Completion of an assignment is achieved when the student successfully masters all of the 2-5 skills within that assignment.Of all students in experiment 1, 81% completed at least 10 of the 12 assignments.
In order to get a sense of the amount of time participants spent on the ES practice units, we present timing data in Figs. 5 and 6. Figure 5 shows a histogram of total time spent on the training units for units 4-12 (timing data are unavailable from the first three units).For Fig. 5, only students who attempted all 9 of the timed units are included in order to avoid underrepresenting the time commitment required.Figure 6 is a quartile box plot of time spent on each of the training units for which timing data is available, and includes all student attempts.To maximize the ease of viewing the heavily right-skewed distributions, only the middle two quartiles are shown.In experiment 1, the median and mean attempt times for units on which timing data are available were 9.4 and 13.4 min, respectively.Normalizing for the three units on which timing data are unavailable, this amounts to a median (mean) total training time of 113 (161) min over the span of the entire course term.This relatively small amount of time verifies one of the goals of ES practice, which is to add only a relatively small amount of time to the students' weekly workload.

B. Improvement of accuracy in vector skills
Participants completing the ES practice assignments demonstrated significant gains in accuracy in vector skills.This improvement can be shown in three ways.First, we compare pretest and post-test scores in the vector skills assessment.A casual inspection of the score distributions on the pretest and post-test for experiment 1 shown in Fig. 7 reveals a large overall gain in scores.To analyze this in more detail, we collected data on overall course grade and categorized students as above or below the mean course grade.A repeated measures ANOVA (high-low grade vs the repeated measure of pre-post vector assessment) reveals perhaps as expected that high grade students score better than low grade students [Fð1Þ ¼ 205.2, p < 0.001], and a significant increase in scores from pretest (41%) to posttest (69%) [Fð1Þ ¼ 2020, p < 0.001, d ¼ 1.5].An effect size of 1.5 represents a large effect, especially compared to the effect sizes typical of computer-based instruction, and it is important to note that this large gain came from only about 2 h of practice over the span of the entire course term.There was no significant interaction between pre-post gains and high-low grade [Fð1Þ ¼ 0.06, p ¼ 0.8], demonstrating that both high and low grade students are gaining equally.
Note that, of course, in this section we have only included data from students completing both pretests and post tests.Looking more broadly, of the 1313 total students who took the final exam in the course, 80% (1056) completed both the pre-and post test.Students that completed both the pre-and post test tended to have higher grades than those who did not complete both tests [tð1311Þ ¼ 9.451, p < 0.001, d ¼ 0.61].More detail about completion rates of ES assignments and course grade is in Sec.V F.
Second, we can at least partially address the claim that the gains in pre-post scores could be solely due to course instruction rather than any contribution from the ES practice assignments.Departmental and ethical constraints prevented us from performing a truly random-assignment control vs treatment study.Instead, we compared scores from experiment 1 to baseline data collected in earlier semesters.The baseline data were collected near the end of the term (i.e., post course instruction) from prior semesters of the same course that did not assign essential skills units.This allows for a comparison of post course instruction scores both with and without ES practice.The results are shown in Fig. 8.The results shown on the left half of the graph represent skills that are often explicitly taught and always used in the first semester course (e.g.vector addition, subtraction, components, tilted axes).The right half of Fig. 8 shows results for items representing skills (e.g., dot product, cross product) which, while they are commonly covered in textbooks, at our institution they are not commonly emphasized in the first semester course (thus technically may not be "essential," so future implementations may omit them).In particular, for vector skills commonly taught (such as vector addition), there is usually a significant difference between the pretest scores and post-test scores for instruction without training, while comparisons to post-test scores with training were always significant.Perhaps as expected, for vector skills that are rarely taught or emphasized (such as cross product), there is no difference between pretest and post scores for instruction without training, but there is a significant difference for instruction with training.See Table II for a summary of results.This provides further evidence that the training improves accuracy above and beyond any improvements from instruction.We note in particular that the scores on the item of computing components on an inclined plane is very low ∼20% even with instruction, and even with training this is only raised to ∼50%, indicating this is a difficult skill and more work is needed to improve student accuracy with this skill.
Third, to provide further evidence that ES practice contributes significantly to improved accuracy, a linear regression was performed with post-test score as the dependent variable and pretest score and number of ES units completed (mastered) as independent variables (see Table III).The results indicate that, independent of pretest score, the number of ES units completed significantly contributes to the variance of the post-test scores and is positively correlated (r partial ¼ 0.29, p < 0.001) with posttest score.In fact the regression analysis shows that on average, for every ES unit completed (out of 12 total), the post-test score increases by 0.14 standard deviations.While this does not prove that ES practice contributes to the post-test score (for example, there could also be selection effects influencing the number of units completed by each student), this result is consistent with the expectation that increasing the amount of ES practice increases ES performance.More discussion and support for this argument is presented in Sec.V F where we report on a comparison of  different course credit awards (and resulting participation rates) for the ES assignments.

C. Retention of skills
Experiment 3, a follow-up longitudinal study, was conducted in order to (i) investigate retention of gains in accuracy, and (ii) to further support the claim that a significant portion of the gains in accuracy are due to ES practice.In the semester immediately following experiment 1, we collected data from the subsequent course in the introductory physics sequence, electricity and magnetism.Many of the participants in experiment 1 were enrolled in this course, and a random sample of N ¼ 61 of these students, along with another randomly chosen sample of N ¼ 40 students in this course who were not enrolled in the previous semester of physics (thus not in experiment 1, and had no ES practice) were chosen to take a paper-and-pencil version of the assessment for partial course credit (other students in the class participated in other experiments so the random drawing is from the whole pool of students).This additional "delayed" post test was administered 11 weeks after the original post test from experiment 1, during the 5th week of the 15-week electricity and magnetism course.
The results are shown in Fig. 9.It is clear from this figure that the gain in post-test scores for the participants with ES practice is retained more than 11 weeks after the final practice session, since there is no significant change in scores between the post test and the 11-week delayed posttest scores [paired tð60Þ ¼ 0.47, p ¼ 0.64].
However, there is a significant difference between the 11-week delayed post-test scores of the participants with ES practice (76%) compared to participants who had no ES practice (66%) [tð99Þ ¼ 3.394, p ¼ 0.001, d ¼ 0.67].This result provides support for the claim that ES practice results in higher accuracy as well as the claim that the course itself also results in higher accuracy, consistent with Sec.V B.
One might argue that the difference between scores for the practice vs no-practice conditions may be due to a difference in populations since, for example, the ES practice condition consists of students taking the introductory physics courses in adjacent semesters, while the nopractice condition students took the first physics course at least one semester earlier.Thus, the two conditions may on average consist of different students-for example, the no-practice group may have forgotten some of the essential skills since they took the first course more than one semester prior to the test.To at least partially investigate this issue, we compared the second-semester physics course grades of the students in these two conditions and found no significant difference between the two conditions [tð96Þ ¼ 2.6, p ¼ 0.79, d ¼ 0.05].Note that since post-test score is somewhat correlated with final grade [r ¼ 0.24, p ¼ 0.02], one might expect that the practice condition, with higher post-test scores, would also have slightly higher grades.However, since the correlation is small, the post-test scores only account for a small amount of the variance of the final grade (r 2 ∼ 5%), thus a much larger sample size would be needed to see this effect.Therefore, we find no evidence of a difference between the populations, but clearly a controlled experiment with larger samples is needed to more formally resolve this potential confound.

D. Improvement in fluency
So far, we have only discussed improvements in student accuracy with essential skills.However, the goal of the ES Framework is not just accuracy but also fluency (i.e., speed and reduction of cognitive load).In this section, we discuss preliminary evidence suggesting that training helps students improve their fluency with the skills as well.
One measure of fluency of a target skill set is the amount of time required to complete a practice unit for that skill set.To quantify this, we compared completion times of two similar ES assignments.In particular, the 4th and 8th units, assigned in weeks 5 and 9, were identical in terms of skills content (type of practice questions).The only difference between the two units was that the 4th unit required only three consecutive correct answers for mastery, while the 8th unit required four.Despite the higher requirement for mastery, the median time for students who complete both units dropped from 13.6 to 9.9 min.Mean times showed a similar drop, from 19.2 to 13.7 min, a significant reduction by a paired-samples t test [tð805Þ ¼ 8.212, p < 0.001, d ¼ 0.37].In addition to the decrease in total time, student accuracy also improved; the average proportion of training problems eventually answered correctly (up to three tries) grew from 83.9% to 92.4% [paired tð805Þ ¼ 5.194, p < 0.001, Another measure of fluency is the amount of time needed to complete the assessment.The timing data for completion of the assessment were strongly right skewed; this is likely a result of students leaving the quiz window open while not actively working on the quiz, sometimes for multiple days.In order to remove such outliers, we only included students completing both the pretest and the post test in less than 2 h.A paired-samples t test showed that the mean time to complete the assignment decreased significantly [tð852Þ ¼ 9.319, p < 0.001, d ¼ 0.39] from 34 to 27 min (median time decreased from 30 to 22.5 min).It is worth noting that the pretest contained only 3 attitudes and beliefs Likert questions, while the posttest contained 11 Likert questions and four extended response questions for students to leave feedback about the training, meaning time on task decreased significantly, despite the additional survey questions.Therefore, it is reasonable to assume that the statistics above are conservative, representing a lower bound for the decrease in time to complete the assessment questions themselves.In any case, this provides more support that students were more fluent in the target skills when comparing their pre-and post-test performance.

E. Essential skills and course performance
An important goal of ES practice is to improve course performance.In this study, it is not possible to conclusively determine the extent to which ES practice improves course performance because the design does not include a control (no treatment) group.Nonetheless, the data do support important inferences about the relationships among ES knowledge, ES practice, and course performance.
First, a linear regression analysis was performed with course grade as the dependent variable and pretest score, pre to post score gain (difference of post-test and pretest scores), and number of ES practice units completed as the independent variables.The results are shown in Table IV, and the model itself is significant [Fð3Þ ¼ 140, p < 0.001, r 2 ¼ 0.29].More specifically, not only is the pretest score strongly correlated with the course grade (perhaps unsurprisingly) accounting for 20% of the variance in course grade, but also the gains in pretest to post-test scores and the number of completed ES units are significantly correlated to course grade.In fact, the analysis reveals that on average, for every ES unit completed, the course grade increased by 0.1 standard deviations.
Clearly there are significant correlations between pretest scores and grade and pre-post gain and grades, but can we provide evidence that these relations are causal?Specifically, does learning essential skills (here, vector skills) result in good grades, or are essential skills (vector skills) in fact necessary to do well in the course?While many instructors may assume that it is obvious that certain skills are necessary for obtaining a good grade, one must nonetheless empirically establish such a necessity.
To go beyond an analysis of correlations and to determine if there is some directionality in the relations consistent with causality (i.e., necessity or sufficiency), we use a simple method developed by Rosenblatt and Heckler [51] using cross-tabulation tables to analyze within student performance in the course and the ES test.Therefore, to gain insight into possible causal relations, we split students who completed both the pretest and the post test into either "high" course grade or "low" course grade categories (above or below the mean for the whole class), and high or low test scores, where high is defined as >60% correct (the mean for all pretest and post-test scores).We then cross tabulated high-low test score with high-low grade for both the pretest and post test (Tables V and VI, respectively).
The relatively low count in one of the diagonal cells of Table V indicates that scoring high on the vector skills pretest is not necessary for obtaining a high grade, but tends In contrast, the post-test results have the opposite implications compared to the pretest results.In Table VI we present post-test and grade results for only those students who score "low" on the pretest, which is 80% of the students.Thus we focus only on students who could improve from low to high on the vector skills test.For students scoring high on the pretest, almost all 200=210 ¼ 95% also scored high on the post test.In Table VI, therefore, students in the high post-test score row have improved their score from low to high.Table VI suggests that improving from low to high on the ES test does not tend to be a very good (or sufficient) indicator of obtaining a good grade, but improvement on the ES test does tend to be important (necessary is too strong) for obtaining a high grade: 57% (352=617) of students improving from low to high on the ES test also obtained a high grade, but significantly fewer students 35% (79=229) who did not improve from low to high on the vector skills test obtained a high grade in the course [χ 2 ð1Þ ¼ 34.0, p < 0.001].
Therefore, while we cannot conclude that knowing vector skills is necessary for obtaining a good grade, the data do suggest that students who enter the course scoring high on the vector skills test tend to do well in the course, and for students who score low initially, if they do not improve from low to high on the vector skills test, they are unlikely to obtain an above average grade.We note also that the relation between improvement on the vector skills test and obtaining a good grade does not necessarily mean that the content of the vector skills test alone is important but rather it could be that a student's demonstrated ability to learn the vector skills test material is an indicator of the ability to do well in the physics course.

F. Awarding regular course credit vs extra credit
In this section we examine whether the way in which course credit is awarded for ES assignments affects ES performance.Experiment 1 offered the assignments as a required part (1%) of the regular total course credit, while experiment 2 was offered for 1% extra credit.That is, grades for students in experiment 1 were made up of 99% other course assignments and 1% for ES assignments, while experiment 2 consisted of 100% for other assignments with up to 1% extra credit for the ES assignments.This ostensibly small difference in the course credit structure accounted for large differences in terms of participation and ultimately gains in vector skills scores.
Specifically, counting the ES assignments as part of the course grade (compared to extra credit), significantly improves participation.In experiment 1, 82% of all students in the course completed at least 9 out of 12 total ES assignments, compared to just 54% in experiment 2 [χ 2 ð1Þ ¼ 132, p < 0.001].Consistent with the results of Sec.V E, there is a significant difference in participation between low-grade students and high-grade students.In experiment 1, 72% of low-grade and 91% of high-grade students completed 9 out of 12 ES assignments, and 40% and 67% of low-and high-grade students, respectively, completed to 9 out of 12 ES assignments in experiment 2 (p 0 s <0.001).
Furthermore, on average students in experiment 1 completed 10.28=12 (86%) training assignments compared to the students in experiments 2 who completed only 7.43=12 (62%).It is worth noting that both populations show a similar rate of assignment mastery-the proportion of units completed per units attempted is above 95% for each experiment.The difference in participation simply comes from students not attempting as many of the training assignments.
The results also suggest that there are differences in pre-and post-test gains depending on how the credit was awarded.Figure 10 shows mean pre-and post-test scores for experiments 1 and 2. Both cases showed significant pretest and post-test gains with large effect sizes [experiment 1: paired tð1055Þ ¼ 46.01, p < 0.001, d ¼ 1.5; experiment 2: paired tð264Þ ¼ 15.01, p < 0.001, d ¼ 0.97.]However, a comparison of gains (post test minus pretest scores) between the two experiments reveals that assigning regular course credit for ES assignments results in higher average gains (28%) compared to average gains from awarding the equivalent value in extra credit points (21%) [tð1319Þ ¼ 5.02, p < 0.001, d ¼ 0.33].Further analysis suggests that this difference is due to the fact that assigning course credit results in more (here, about 3) assignments completed on average, which in turn results in higher gains in scores.To further quantify these effects, a linear regression was performed with a post-test score as the dependent variable and pretest score, number of ES assignments completed, and award structure (regular credit vs extra credit) as the input variables.Such a regression model resulted in r 2 ¼ 0.26, and only pretest (β ¼ 0.40, p < 0.001) and number ES assignments completed (β ¼ 0.27, p < 0.001) were significant; the award structure was not significant (p ¼ 0.37).Rather the award structure is causing a difference in how many ES assignments are completed, thus indirectly causing the larger score gain.This result is also consistent with our finding in Sec.V B that gain increases about 0.1 standard deviations for every assignment completed.
Note that there is a potential confound in this comparison: The students in experiment 1 were enrolled in Autumn and the students in Experiment 2 were enrolled in Spring.One might argue that any differences could be due to differences in population.However, we would argue that this is likely only a small effect.We have found that the main effect of assigning credit as part of the course grade is in a very large difference in participation in ES practice (as mentioned, 82% in Experiment 1 vs 54% in Experiment 2), for both high and low grade students.First, in our experience with these courses (doing education research and teaching these courses), there is no evidence of a difference in physics course performance between these two semesters (if there is any at all).Second, since we see this effect for both low and high performing students, this argues against a large confounding effect.
In sum, we have replicated the gains in vectors skills in a slightly different implementation context, and these findings suggest that in order to maximize student participation in ES practice and overall gains in ES skills, regular course credit should be assigned rather than extra credit.

G. Student feedback on ES practice
Immediately after both the pretest and the post test on vector skills, students answered a number of 5-point Likert scale questions designed to briefly probe their attitudes about the training (post test only; see  We also provide a place to type open-ended comments, and while many comments were positive ("I learned a lot"; "it helped me in the course"), one somewhat common complaint is that students must get several correct in a row in order to master a topic, and if they happen to get one wrong, they do not like to "start all over again at zero."

VI. SUMMARY AND DISCUSSION
The proposed essential skills design framework and successful implementation in this study provide support for the idea that many students have difficulties with basic, essential skills in STEM courses-even postinstruction-and distributed, interleaved practice using computer-based, mastery graded instruction with immediate feedback can significantly-perhaps even dramatically-improve and maintain accuracy and fluency with these skills with only a few hours of practice during a semester course.While this study has only implemented the framework for a relatively small set of essential skills (vector math), it is not unreasonable to expect that this framework will work for a wide range of other STEM skills as well, and some of our studies on other skills support this [4,5].
The conceptualization, justification, and design of ES practice could be seen as a fusion and application of wellknown and empirically tested ideas and principles, including deliberate practice [15,52] and, as discussed earlier, best cognitive learning and feedback practices, and discipline-based education research on specific difficulties with target skills.The computer application itself does not use adaptive techniques or artificial intelligence.Rather the application has a straightforward structure, and as such the methods used here could be adapted to other existing learning software.Nonetheless, it should be noted that considerable effort was invested in designing the content, which was based on a significant amount of education research and iteration.
While this study has been successful in showing gains in a set of essential vector math skills, we note that this is only one potential tool for achieving some of the many and varied instructional goals for introductory physics.ES practice is certainly not a silver bullet, nor is it likely appropriate for other goals, such as deep conceptual understanding and complex problem solving.In fact, from a previous study [4], we found evidence suggesting that it is likely that even some basic skills cannot be trained using the methods in this study.There are a variety of factors, including student attitudes and perceptions of the target skill, which may strongly affect student learning of that skill.For example, they may see the skill as irrelevant or that it is easy to look up online, so there is no need to commit it to memory.
Of course this study is just a small step in the process of determining how to improve student understanding and problem solving skills in STEM.For example, it still needs to be established that improving essential skills leads to improvement in problem solving that employs essential skills and how one might further facilitate the transfer of such skills to more complex problem solving or similar skills.
Further, it is would also be interesting to investigate how the improvement in essential skills interacts with improvements in conceptual understanding of topics related to such skills.Research in developmental psychology suggests that children discovering new rules for automating specific procedural mathematical skills frequently exhibit the new, more efficient rule before they are able to explicitly state the rule themselves [53].That is, procedural knowledge and fluency often precede formal declarative knowledge for procedural skills.This implies that training for accuracy and fluency before requiring the ability to state formal declarative rules is, in fact, a natural way for students to learn the simple skills, and further, this may play a role in the development of more complex problem solving skills and perhaps even robust conceptual understanding.
Another potential significant strength of ES practice is in helping students who are less prepared, and this may help reduce the number failing grades and course withdrawals.Clearly, we found that students with high and low grades obtained large gains in the skills.While we found strong relationships between the number ES assignments completed and higher grades, a more controlled study is needed to firmly establish that ES practice leads to higher grades.It is reasonable to expect such improvements, since this effect has been seen in similar mastery based practice systems such as ALEKS [54].
Finally, we note that there is more work to be done to develop ES practice for other skills beyond vector math.As noted earlier, we have made some progress on skills related to introductory engineering courses, such as reading logarithmic plots and unit analysis [4,5], but there are many skills yet to be identified, studied, and employed in an ES practice system.

FIG. 1 .
FIG. 1. Experimental design used in this study.Experiment 3 was conducted 11 weeks after the post test in experiment 1.
course-other students participated in other PER experiments for course credit.Participants in experiments 1 and 2 completed a pretest in the first week of the course and a post test approximately one week before the final exam, both administered online and completed outside of class.Between the pretest and post test, the students were assigned 12 weekly training assignments via an online application that we constructed for this purpose (see Sec. IV B 1). Participants in experiment 3 completed just the post test 11 weeks after the post test in experiment 1 for partial course credit (∼1% of grade).

FIG. 8 .
FIG. 8. Comparison of scores preinstruction and postinstruction, with and without essential skills practice.Skills to the left of the dotted line were explicitly taught in the first semester course, while skills to the right received no explicit instruction.Error bars are 1 S.E.

FIG. 9 .
FIG. 9. Longitudinal student assessment performance.A subset of experiment 1 students (pretest and post-test scores in dashed bars) completed a delayed post test (first solid bar) during the second semester physics course.A sample of students from the same second semester course who had not previously received training also completed the post test (second solid bar) at the same time.

4 .
Force diagrams for two different objects, C and D, are shown.How does the magnitude of the net force on object C (i.e., the vector sum of all forces on the object) compare to the magnitude of the net force on object D? a) The magnitude of the net force on object C is greater than the magnitude of the net force on object D. b) The magnitude of the net force on object C is less than the magnitude of the net force on object D. c) The magnitude of the net force on object C is equal to the magnitude of the net force on object D. 5. A block with weight W sits at rest on an inclined plane, as shown.What is the component of the weight vector W in the y direction?Note: The coordinate system provided specifies the positive directions for x and y.6.As shown in the picture to the right, vector A has length 3, and vector B has length 4. The angle between A and B is θ ¼ 40°.What is the magnitude of the cross product, jA × Bj?You may use a calculator.7. Multiple force vectors act on an object, as shown.Which of the following options represents the net force (i.e., the vector sum of all forces) on the object?Please circle your answer.8. Vector A Is shown.Which of the following options represents A y , the y component of vector A? Please circle your choice.

9 .
For two arbitrary vectors A and B, which of the following statements about dot and cross products is true?a) Both A × B and A • B are scalars.b) Both A × B and A • B are vectors.c) A × B is a scalar and A • B is a vector.d) A × B is a vector and A • B is a scalar.e) Both A × B and A • B can be either vectors or scalars.

13 .
Consider the two vectors below.A ¼ 1i þ 3j B ¼ −4i þ 2jWhich of the following options represents the vector difference, A − B? Please circle your answer.Note: i and j represent the unit vectors in the x and y directions, respectively.14.For each of the four pairs of vectors shown, determine the direction of the cross product A × B. (4 points) Answer using the key shown to the right.15.A vector with magnitude F is shown.Find the component of vector F in the y direction.Note: The coordinate system provided specifies the positive directions for x and y.

)
Minimize additions to student workload.Repeated practice is not viable in class, and given that problem-solving homework assignments are typically assigned to students, extra time assigned outside of class must be minimal.ES practice is designed to take approximately 10-20 min =week.Increase student self-efficacy in essential skills.

TABLE I .
Vector skill categories practiced.Examples shown in Fig.2.
a Both given vectors and answer format were varied between the ijk and arrow format.

TABLE II .
t-test comparison of pretest vs post test assessment scores with and without training.Note that asterisks indicate significance at the p < 0.05 level.

TABLE III .
Linear regression results for experiment 1. Post-test score is the dependent variable.Note that β represents the standardized regression coefficient.All values are significant at the p < 0.001 level.

TABLE IV .
Multiple linear regression analysis results with course grade as the dependent variable.Note that β represents the standardized regression coefficient.All values are significant at the p < 0.001 level.In all, this indicates that scoring high on the pretest-i.e., being accurate in essential skills prior to the course-tended to be sufficient but not necessary for obtaining a high grade in the course.In other words, being accurate in ES before the course was not essential to scoring high in the course.Rather the pretest scores suggest a causality that is more of a selection effect.Namely, awarding a low course grade tends to disproportionally select for students with low pretest scores.

TABLE VI .
Cross tabulation of post-test scores and course grades for students scoring low on the pretest.
FIG. 10.Pretest and post-test scores for experiments 1 and 2, which varied whether the essential skills were assigned for course credit or extra credit.
Table VII).Student attitudes about the ES practice were generally positive, and responses to post-test questions about training are shown in Table VII.In general, most students felt that

TABLE VII .
Student responses to attitudes and beliefs of post-test survey items.Note That percentages may not add to 100% due to rounding.I highly recommend the essential skills units ARE NOT used again 5 ¼ I highly recommend the essential skills units ARE used again the workload was fair, the assignments were not difficult, they learned new things during training, and-notablyrecommended that the ES assignments are used in the same course in the future.Still, 30% of students disliked or strongly disliked the ES practice, and we are investigating ways to reduce this number.