Implementing and assessing computational modeling in introductory mechanics

Students taking introductory physics are rarely exposed to computational modeling. In a one-semester large lecture introductory calculus-based mechanics course at Georgia Tech, students learned to solve physics problems using the VPython programming environment. During the term 1357 students in this course solved a suite of fourteen computational modeling homework questions delivered using an online commercial course management system. Their proficiency with computational modeling was evaluated in a proctored environment using a novel central force problem. The majority of students (60.4%) successfully completed the evaluation. Analysis of erroneous student-submitted programs indicated that a small set of student errors explained why most programs failed. We discuss the design and implementation of the computational modeling homework and evaluation, the results from the evaluation and the implications for instruction in computational modeling in introductory STEM courses.


I. INTRODUCTION
Computation (the use of a computer to numerically solve, simulate, or visualize a physical problem) has revolutionized scientific research and engineering practice. In science and engineering, computation is considered to be as important as theory and experiment. 1 Systems that cannot be solved in in closed form are probed using computation; experiments that are impossible to perform in a lab are studied numerically. 2,3 Yet, as computer usage has grown (e.g., facilitation of homework), most introductory courses have failed to introduce students to computation's problem solving powers.
Computer usage (as opposed to computation) is ubiquitous in large introductory courses, and much work has been done to understood about the role of computers in these courses. 4 By contrast, little is known about the role that computers might play as problem-solving tools in introductory courses. For example, given the constraints of a traditional large lecture STEM course, is it possible to provide sufficient instruction in computational problem solving techniques such that students can apply these tools to new problems? Moreover, what challenges do students face when learning and applying computational problem solving techniques, and how might we mitigate those challenges through instruction? In this paper, we describe our efforts to address these questions.
Using computation in introductory physics courses has several potential benefits. Students can engage in the modeling process to make complex problems tractable. Students can also use computation to explore the applicability and utility of physical principles. In a way, students who compute are participating in work that is more representative of what they will do as professional scientists and engineers. [5][6][7][8] When constructing simulations, students are constrained by the programming language to certain syntactic structures. Hence, they must learn to contextualize problems in a way that produces a precise representation of the physical model. 9,10 One of computation's key strengths lies in its utility for visualizing and animating solutions to problems. These visualizations can improve students' conceptual understanding of physics. 11 It is important to distinguish between computational modeling and simply writing program statements. Computational modeling is the expressing, describing, and/or solving of a science problem by using a computer. It requires additional steps beyond simple programming, such as contextualizing the programming task and making sense of the physical model. While programming is taught in most introductory computer science courses, computational modeling is not.
We used computation in a large-enrollment introductory calculus-based mechanics course at the Georgia Institute of Technology to develop students' modeling and numerical analysis skills. We built upon previous attempts to introduce computation in introductory physics laboratories 12,13 by extending its usage to other aspects of students' coursework. In particular, we taught students to use the VPython programming environment to construct models that predict the motion of physical systems. 14 We describe the design and implementation of homework problems to develop students' computational modeling skills in a high-enrollment foundational physics course (Sec. III). We also provide the first evaluation and explication of students' skills when they attempt individually to solve a novel computational problem in a proctored environment (Secs. IV-VII). We discuss implications for instructional design, considerations regarding student epistemology and the assessment of knowledge transfer as well as the broader implications of teaching computation to introductory physics students (Sec. VIII).

II. APPROACHES TO IMPLEMENTING COMPUTATION
Since the development of inexpensive modern microcomputers with visual displays, there have been a number of attempts to introduce computation into physics courses. We review these attempts by decomposing them along two dimensions (openness of the environment and size of intended population ) to indicate how our approach fits in with previous work.   14 an open computational environment, has been used to teach introductory physics students to create computational models of physical phenomena. 12 Typically, students write all the program statements necessary to model the physical system (e.g., creating objects, assigning variables, and performing numerical calculations) using IDLE, VPython's default development environment. The additional details of model construction (e.g., drawing graphics, creating windows, mouse interactions) are handled by VPython and are invisible to the students. VPython supports full three-dimensional graphics and animation without the additional burden to students of learning a great deal of object-oriented 4 programming. 22 Given its roots in the Python programming language, VPython can be a powerful foundation for students to start learning the tools of their science or engineering trade. Moreover, VPython is an open-source, freely available environment accessible to users of all major computing platforms.
The Matter & Interactions (M&I) curriculum 23 introduces computational modeling as an integral part of the introductory physics course. Many of the accompanying laboratory activities are written with VPython in mind, and a number of lecture demonstrations are VPython programs. In the traditional implementation of M&I, the practice of constructing computational models is limited to the laboratory. In a typical lab, students work in small groups to complete a computational activity by following a guided handout. They pause periodically to check their work with other groups or their teaching assistant (TA). Students' computational modeling skills are evaluated by solving fill-in-the-blank test questions in which they must write the VPython program statements missing from a computational model.

Our approach to teaching computation uses an open environment (in VPython) and
builds on our experience with M&I to extend the computational experience beyond the laboratory. We chose to use an open environment to teach computation in order to provide students with the opportunity to look inside the computational "black-box" and construct the model themselves. Furthermore, we aimed to teach students how to develop solutions to analytically intractable problems. We chose VPython (instead of e.g. Java, C, or Matlab) because it has a number of helpful features for novice programmers, can be used to construct high-quality three-dimensional simulations easily, and is freely available to our students.
VPython is also conveniently coupled to M&I, allowing us to leverage our years of experience with teaching M&I. While our implementation builds on our M&I experience, it is not limited to that experience. We describe our implementation philosophy in the next section.

III. DESIGN AND IMPLEMENTATION OF COMPUTATIONAL HOMEWORK
We aimed to develop an instructional strategy that helps computation permeate the course, but which does not require that students have previous programming experience. At Georgia Tech, half of our introductory mechanics students have no programming experience; the other half have had some introduction to programming in high school or in their previous two semesters at Georgia Tech. 24 The Georgia Tech "introduction to programming" course is typically taught using MATLAB, not VPython. Furthermore, our implementation had to be easily deployable across large lecture sections, the setting in which most introductory calculus-based courses are taught. Our philosophy held that students should learn computation by altering their own lab-developed programs to solve slightly modified problems outside of lab. This design philosophy was informed by what research scientists do quite often; they write a program to solve a problem and then alter that program to solve a different problem that is of interest to them. We envisioned developing computational activities that would start with guided inquiry and exploration in the laboratory, and finish with independent practice on homework. In this vision, students would work with TAs in the laboratory to develop a program that solves a problem. Students would then modify that program on their own to solve a different problem on their homework. In our implementation, eleven out of a total of thirteen mechanics laboratories contained a computational modeling activity.
Students solved thirteen computational modeling homework problems (one per week) which were embedded in their regular homework (∼15-21 problems per week).
The class of problems that becomes available to students who have learned computation is large and diverse; we chose to focus our efforts on teaching students to apply Newton's second law iteratively to predict motion. Students taking a typical introductory mechanics course would learn several motion prediction equations emphasizing kinematics, a way of describing the motion without explicitly connecting changes in the motion to forces (i.e., dynamics). These kinematic formulas are quite limited in their applicability; students can only apply them to problems in which the forces are constant. This can confuse students when they are presented with a situation where such formulas do not apply. 25 Students taking most introductory mechanics courses are limited to solving problems that are analytically tractable. Furthermore, the special case of constant force motion is usually the capstone of motion prediction in an introductory mechanics course. Some courses might teach students to analytically determine the motion of an object subjected to some integrable a(t) or subjected to linear (F ∼ v) air drag in one dimension, but neither case demonstrates the full predictive power of Newton's second law. By contrast, computation allows instructors to start first and foremost with Newton's second law and emphasize its full predictive power. Students can numerically model the motion of a system as long as they are able to develop a physical model of the interactions and express it in the computational environment. The numerical integration technique used to predict motion is a simple algorithm.
As a concrete example of our design, we show a mid-semester laboratory activity and homework problem in which students modeled the gravitational interaction between two bodies. In this example, students develop a VPython program that models the motion of a craft as it orbits the Earth (Fig. 1). Students later make a number of modifications to this program to solve a new problem on their homework. This example is useful because it illustrates both the level of sophistication we expect of students and the types of alterations we ask students to make on their homework.
In groups of three, students wrote a program in the laboratory to model the motion of low-mass craft as it orbits the Earth (Fig 1). In VPython, they created the objects (lines 4- In the week following the lab, students solved a computational homework problem in which they used the computational model that they had written in lab to solve a modified problem. Students were asked to alter their initial conditions to predict the position and velocity of the craft after a specified time. To solve this problem successfully, students had to identify and make changes to their initial conditions (lines 4, 6, 9-10, and 12) and their integration time (line 17). In addition, students had to add an additional print statement (after line 31) to print the final velocity of the craft.
Computational homework problems were deployed using the WebAssign course management system, which facilitated the weekly grading of students' solutions. To create the homework problem, we numerically integrated several hundred initial conditions and stored the solutions, including final quantitative and qualitative results. Each student was assigned a random set of initial conditions corresponding to a particular set of results. Randomization ensured a high probability that each student received a unique problem. Students used their assigned initial conditions and wrote additional statements to answer the questions posed in the problem. Students entered numerical answers into answer fields, and selected check-boxes to answer to qualitative questions. On these weekly assignments, only students' final results were graded; their code was uploaded for verification purposes, but not graded.
Grading programs for structure and syntax at this large scale requires additional work by TAs who are already charged with a number of other teaching and grading tasks. Computational homework problems were generally completed in the week that followed the associated laboratory activity.
To facilitate student success and to help students learn to debug their programs, each assignment contained a test case -an initial problem for which the solution (i.e., the results from the numerical integration) was given. When writing or altering any computer program, even experts can create programming errors (bugs). Learning to debug programs is part of learning how to develop computational models. This test case ensured that a student's program worked properly and helped to instill confidence in students who might otherwise have been uncomfortable writing VPython programs without the help of their group members or TAs. After a student checked her program against the test case, she completed the grading case -a problem without a given solution.
In keeping with our overall design philosophy, most of the homework problems which students solved had similar designs as the aforementioned example. In particular, students built a computational model in the laboratory and independently used that fully-functioning model to solve a modified problem on their homework. On the first four homework assignments (of which the previous example is the fourth) students made only a few modifications to their programs, altering their initial conditions and adding a new print statement. In the next several labs, students learned to model more complicated systems (e.g., three body gravitational problem, spring dynamics with drag) while learning new algorithms such as decomposing the net force vectors into radial and tangential components. Students also learned to represent these force components as arrows in VPython. On the homework problems associated with these labs, students still used their lab-developed programs to solve new problems by changing initial conditions and representing new quantities with arrows.
The last two homework problems which students solved were not related to the labo-9 ratory exercises; we intended to emphasize the utility of learning to predict motion using Newton's second law. To solve these problems, students wrote all the statements missing from a partially-completed code to predict the motion of two interacting objects. These were interactions which students had not seen before (e.g., the anharmonic potential and the Lennard-Jones interaction). In these problems, we omitted the appropriate initial conditions and the statements that numerically integrated the equations of motion. Students had to contextualize the word problem into a programming task and produce a precise representation of the problem in the VPython programming environment. With regard to programming tasks, students had to do no more than identify and assign variables and implement the usual motion prediction algorithm for these two problems. A similarly-designed problem was used as an evaluative assignment and is discussed in detail in Sec. IV.

IV. EVALUATING COMPUTATIONAL MODELING SKILLS
Students performed as well on computational modeling homework problems as they did on their analytic homework. We found no statistical difference between students' performance on their computational homework and their performance on analytic homework using a rank-sum test (Analytic 84.6% vs. Computational 85.8%). 26 A side effect of the additional practice was an overall increase in student performance on hand-written fill-in-the-blank programming questions on exams. However, this result did not indicate what fraction of students were able to solve these computational homework problems without assistance.
While randomizing initial conditions between each student's realization ensured that students' solutions differed with high probability, working programs could be distributed easily from student to student by email. We note that the distribution of students' programs might not be deleterious; students who received these programs must still have read and interpreted the program statements to enter in their initial conditions, make changes to the force law, or print additional quantities. This is a more complex interaction than simply plugging numbers into a algebraic solution that they discovered online. In a sense, students who worked with shared code were using a "closed" computational environment.
Nevertheless, we wanted to measure how effective students were at individually solving computational problems. We delivered a proctored laboratory assignment during the last lab of three different semesters to evaluate students' computational skills on an individual basis. Students received a partially-completed program that created two objects (one lowmass and one high-mass), initialized some constants, and defined the numerical integration loop structure. We aimed to evaluate students' engagement of the modeling process by contextualizing a physics problem into a programming task. Furthermore, we intended to assess certain programming skills, namely, students' abilities to identify and assign variables and implement the numerical integration algorithm. The assignment was delivered using We-bAssign in a timed mode (30 minutes) in each of the sixteen lab sections over eight meeting periods. TAs were not permitted to help students debug their programs, nor were students allowed any additional aids (Internet, book, notes, lab partners, et cetera). Furthermore, the assignment was IP-restricted and password-protected; students could only access the assignment during their assigned lab time. Even tough the assignment was proctored over 4 days, we believe it is unlikely that large number of students shared useful information about the assignment; scores for each lab section were statistically indistinguishable. The format of the assignment was identical to students' final two homework problems; students were given a test case to check their solution before solving the grading case.
For this assignment, students modeled the motion of the low-mass object as it interacted with the high-mass object through a central force. The nature of the force (attractive or repulsive) and its distance dependence (r n ) were randomized on a per-student basis.
We also randomized some of the variable names in the partially-completed program to hinder copying. After adding and modifying the necessary program statements, students ran their program and reported the final location and velocity of the low-mass object. During the assignment, students did not receive feedback from the WebAssign system about the correctness of their answer, but they were given three attempts to enter it. Similar to students' online homework, only the final numerical answer was graded. These additional attempts afforded students who experienced browser issues or made typing mistakes the ability to resubmit their answers without penalty.
Performance varied from semester to semester (Table I) because the assignment was modified slightly between each semester in order to streamline delivery (Version 1 to Version 2), reduce transcription errors, and improve presentation (Version 2 to Version 3). In the first semester, students were permitted to attempt Version 1 of the assignment twice due to a logistical issue with the initial administration of the assignment. The majority of students (64.3%) were able to model the grading case successfully on the second administration of the TABLE I. As part of a final proctored lab assignment, students completed a partially constructed program which modeled the motion of an object under the influence of a central force. The partially written program defined the objects, some constants and the numerical integration loop structure.
The initial conditions, the sign (±) and distance dependence (r n ) of the force, and object names were randomized on a per-student basis. Slightly modified versions (Ver.) of this assignment were given at the end of three different semesters. Modifications were made to streamline delivery (Version 1 to Version 2), minimize transcription errors and improve presentation (Version 2 to Version 3). Students' performance on Version 1 was likely inflated because some students were allowed to work the problem on two separate occasions. Overall, roughly 40% of the students were unable to model the grading case. Students with some previous programming experience were no more successful than those without any experience as indicated by a contingency table analysis. 28 To determine exactly what challenges they faced while completing this assignment, we reviewed the program of each student who failed to model the grading case. Students who uploaded their programs to the

Ver. Correct Incorrect % Correct
WebAssign system received a small extra credit bonus on the proctored assignment. Only a very small number of students (<1%) did not upload any code. We limited our review to the programs submitted for Versions 2 and 3 of the assignment. Students must perform several tasks to successfully write and execute the program for the proctored assignment. Students must interpret the problem statement; that is, they must 13 contextualize a word problem into a programming task. They must review the partiallycompleted program and identify the variables to update. Students need to apply their knowledge of predicting motion to the problem using VPython. They must identify that the force is non-constant, and then write the appropriate programming statements to calculate the vector force. Students need to then complete the motion prediction routine by writing a statement to update the momentum of the low-mass object.
Using an iterative-design approach, we developed a set of binary (affirmative/negative) codes to check which tasks students performed correctly and which errors they made. An initial review of students' uploaded programs yielded the mistakes that were made most often. These common mistakes formed the basis for the codes. The codes were developed empirically, and several iterations were made before they were finalized. Two raters tested the codes by coding a single section of student submitted programs (N = 45). The raters resolved their differences (which further explicated the codes) and then recoded the section.
The final codes (Table II)  Determining where students encountered difficulties with these tasks might help explain how students learn this algorithmic approach to use Newton's second law to predict motion.
Because we reviewed students' programs after they were written, we are unable to comment directly on students' challenges with contextualizing the problem. Our work was limited to analyzing students' procedural efforts (i.e., identifying variables and implementing the numerical integration algorithm). However, each of the authors has several years of experience with teaching computational modeling to introductory students from a wide variety of backgrounds. Hence, some information about students' thoughts and actions could be inferred from this analysis.

VI. FREQUENCY OF ERRORS IN STUDENTS' PROGRAMS
We measured the frequency of students' errors within each category (IC, FC and SL) by mapping binary patterns extracted from our coding scheme to common student mistakes.
The number of possible binary patterns we observed in our data ranged from nine for SL to seventeen for FC with 13 possible for IC. Not all the codes within a given category were independent, hence, the number of possible binary patterns is much less than 2 n . Within a given category, we found that a large percentage of students could be characterized by just a few error patterns (between four and seven).
The errors we observed were not necessarily unique to computational problems. The most notable errors involved calculating forces or updating the momentum. Most of these errors appeared to be physics errors reminiscent of those made on pencil and paper problems. Many of them could have been mitigated by qualitative analysis. Some errors were unique to computational models and the iterative description of motion because they did not prevent the program from running, but nevertheless caused it to model the system inap- propriately. Still others (e.g., replacing initial conditions) might have appeared to be simple careless mistakes; when investigated closely, however, these errors highlighted the fragility of students' knowledge.

A. Initial Condition Errors
Students had to identify and update a total of eight given values: the interaction constant (k), the "interaction strength" (n), the mass of the less massive particle, the position and velocity of both particles, and the integration time. Most students with incorrect programs (88.8%) fell into one of seven IC patterns (Table III) (Table II) were populated by more than 3% of the students. The patterns (ICx) are given by affirmatives (Y) and negatives (blank) in the code columns (IC#). The percentage of students with each pattern is indicated by the last column (%). These 7 patterns accounted for 88.8% of students with erroneous programs. Students in ICa were most likely stuck on the test case because they had trouble with another aspect of the problem. These students were unable to obtain the solutions provided in the test case, and so kept working on it. It is possible a number of these students ran out of time while trying to debug their programs.

Initial Condition Codes
It is difficult to say definitively if students with mixed initial conditions (ICb and ICe) were unable to identify the appropriate values, as we reviewed students' programs only after they were submitted. It is possible that these students were just careless when making changes, but they might have been unable to identify and update these quantities. Some students could have been in the process of updating these quantities when they ran out of time and uploaded their programs.
Identifying and updating variables in a program is not a trivial task for students. In fact, students' difficulties with updating variables highlights the fragility of their computational knowledge. As an example, consider the students who confused the exponent on the length unit of the interaction constant (k) for the exponent in the scientific notation of the numerical value of k when they defined it in their programs (ICd). The distance dependence of the central force was randomized, and hence the units of the interaction constant (k) were dependent on a student's realization. In Version 2 of the assignment, the exponent on the length unit of k was colored red (WebAssign's default behavior for random values). A student in ICd would read k= 0.1 Nm 3 to mean k = 0.1E3 = 100 rather than k = 0.1 Newton times meters cubed. In Version 3 of the assignment, we changed the exponent's text color to black like the rest of the non-random text. The overall frequency of this mistake dropped from 30.5% to 9.1%.

B. Force Calculation Errors
Students were given the magnitude of the force as an equation (F = kr n ) and told that their (attractive or repulsive) force acted along the line that connected the two objects.
In solving this problem, students had to correctly calculate the magnitude of the central force and identify the unit vector (r) and sign (±) for their own realization. Almost all students (98.8%) appeared with one of five FC patterns (Table IV) (Table II) were populated by more than 3% of the students. The patterns (FCx) are given by affirmatives (Y) and negatives (blank) in the code columns (FC#). The percentage of students with each pattern is indicated by the last column (%). These 6 patterns accounted for 98.8% of students with erroneous programs.

Force Calculation Codes
completely incorrect (e.g., used the differential form of the Impulse-momentum theorem) or was calculated outside the numerical integration loop (i.e., a constant force). Those students in FCf (8.4%) had an appropriate force calculation procedure (FC2) but invented a unit vector for the net force (FC5).
The difficulties that students faced when numerically computing the net force likely stems from a weak grasp of the concept of vectors. Students in FCa made directional mistakes (e.g., changing the sign of one of lines 21-23 in Fig. 1) that could have been easily identified and rectified by drawing a sketch of the situation, a problem-solving strategy that is practiced in the laboratory. Those who raised the separation vector to a power (FCd) likely transcribed the central force equation (replacing r by r) without thinking that this operation was mathematically impossible ( F ∼ k( r) n vs. F ∼ k| r| nr ). We have found that students attempt a similar operation on pencil and paper problems; raising components of a vector to a power (e.g., ( r) n = r n x , r n y , r n z ). Such mistakes are quite common among introductory physics students. 29 However, in the pencil-and-paper case, students are not immediately directed to their mistake as they are in a programming environment. VPython raises an exception error when this operation is attempted. These students appeared to be unable to parse this error into any useful information. Students who make this type of error might be helped by additional exposure to translating force equations to precise programmatic representations. 10 Some students invented a unit vector (FCf) for the net force. This was most likely because they had computed a scalar force and tried to add a scalar impulse to the vector momentum. VPython raises a different exception error upon any attempt to add a scalar to a vector. These students were able to parse this error, but resolved it incorrectly.
Other students (FCc) might have incorrectly contextualized the problem by including an irrelevant force (i.e., gravitational or electrical interactions). The problem clearly stated that the two objects were far from all other objects. It did not explicitly state to neglect the gravitational interaction between the objects. However, the gravitational interaction could be safely neglected for the range of masses and distances we had chosen. Furthermore, nothing about the charge of the objects was mentioned in the problem statement. It is surprising that students included these interactions in their models. One possible explanation for the inclusion of these interactions is that students had memorized how to solve the gravitational and Coulomb problems because these problems had appeared on their homework several times and on an exam. They might have panicked and simply wrote all possible forces they could remember. We have observed similar contextualization issues in laboratory instruction and on hand-written exams.
A number of students (FCe) did not employ the force calculation algorithm at all. Some of these students computed the net force (e.g., lines 21-23 in Fig. 1) outside the numerical integration loop (e.g., before line 19 in Fig. 1). In this case, the net force was effectively constant and therefore only correct at t = 0 . A program with correct syntax will run regardless of its physical implications. This error is unique to computational problems in which motion is predicted iteratively. Students in introductory physics rarely use Newton's second law to predict the motion of objects that experience non-constant forces. Other students who fell into FCe wrote "creative" program statements. Students in this group manipulated some quantities in the loop but did not perform any physically relevant calculations. The number of students with "creative" program statements was relatively small.

C. Newton's Second Law Errors
Students had to write a program statement similar to line 25 in Fig. 1 to properly update the momentum using Newton's second law. Most students demonstrated no difficulty in  (Table II) were populated by more than 3% of the students. The patterns (SLx) are given by affirmatives (Y) and negatives (blank) in the code columns (SL#). The percentage of students with each pattern is indicated by the last column (%). These 4 patterns accounted for 95.7% of students with erroneous programs.

Second Law Codes
remembering the formula for the momentum update but some met challenges with making that description precise. 10 Nearly all students (95.7%) fell into one of four SL patterns (Table   V). Most students appeared in SLa (69.7%) because they wrote Newton's second law in the correct iterative format, p = p + F*dt (SL1). A much smaller number of students fell into SLb (13.2%). These students wrote Newton's second law in an iterative format (SL2), but they attempted to update the vector momentum with a scalar force (SL3). Students in SLc (7.9%) were unable to write Newton's second law in any form that updated (all codes negative). A small fraction (SLd, 4.9%) wrote Newton's second law in an iterative form (SL2), but did so incorrectly (e.g., by creating a new variable).
Students who attempted to update the momentum with a scalar force (SLb) might have been facing difficulties with understanding vectors. The momentum update was presented as a vector equation ( p f = p i + F ∆t). These students might have been unable to unpack that representation into a precise programmatic description, but it was more likely that they calculated a scalar force (FCc) and then simply wrote the correct (vector) second law syntax. VPython raises an exception error if an attempt to add a vector to a scalar is made.
The students appeared unable to parse this error into any useful information.
Students who were unable to write Newton's second law in any form that updated (SLc) might have experienced difficulties with converting the second law formula into a precise and 20 useful programmatic representation. Students in this category either wrote Newton's second law in a non-update form (e.g. writing deltap = Fnet*deltat or pf -pi = Fnet*deltat as line 25 in Fig. 1), or wrote a number of program statements which manipulated quantities but performed no useful calculations. In either case, these students could benefit from the precision required by a programming language. 10 By requiring them to accurately represent Newton's second law in their programs, they might begin to distinguish between the utility and applicability of its various algebraic forms.
Students who wrote Newton's second law in a form that updated incorrectly (SLd) either remembered the formula for the second law incorrectly or made a typo. These students would generally leave off the timestep in the momentum update (e.g., p = p + F) or divide by it (p = p + F/deltat). Dividing by the timestep is a particularly egregious error because the numerical value of the timestep was quite small. Hence, the impulse added in this case would have been huge. Students who made this error were unable to assess the state of the visualization (the particle flew off to "infinity") to debug this error.

VII. COMMON ERROR PATTERNS IN STUDENTS' PROGRAMS
The patterns within individual categories (IC, FC, and SL) indicated the frequency of common mistakes students made when solving the proctored assignment, but a single student could make one or more of these mistakes. Evaluating a student's complete solution requires an analysis using all the codes (Table II). In principle, the codes we developed could have had up to ∼ 4300 possible error patterns using all sixteen codes. In fact, the intersections of code categories indicated that the number of distinct errors made by students across all categories was relatively small; we found only 111 distinct binary patterns. It is possible to relate these unique patterns in a manner that suggests dominant common errors.
Cluster analysis, a technique borrowed from data mining, is particularly well suited for this application because it characterizes patterns in complex data sets. 30,31 This technique has been used previously to classify students' responses to questions about acceleration and velocity in two dimensions. 32 It was used here to determine the major features in students' incorrect programs which were responsible for their failure.
We applied the cluster analysis technique to the data generated from our set of binary codes. We used the Jaccard metric 33 to measure inter-cluster distances and linked clusters 21 using their average separation. 34 We tested several other metrics (e.g., Hamming, city block, etc.). The Jaccard metric was chosen because it neglects negative code pairs. Both the Hamming and city block metrics produced similar pairings at low levels, but higher order clusters were difficult to interpret. We used average linkage to avoid the effects of "chaining" that appeared when nearest neighbor linkage 35 was used and because useful clusters were more difficult to distinguish when farthest neighbor linkage 36 was used.
Thirty clusters with inter-cluster distances below 0.5 were reviewed in detail. This cutoff was selected to minimize the number of unique clusters while still rendering clusters with useful interpretations. Most students (86.5%) appeared in seven of the thirty clusters (Table   VI). These clusters had very few students (<1%) with affirmatives in the "Other" category.
Codes O1 and O2 were dropped from Table VI (Table VI).
Students in cluster A (23.8%) tended to remain stuck on the test case (ICa) due to an error in their force calculation. Reversing the direction of the force (FCa) was the most common mistake, followed by raising the separation vector to a power (FCd). Most students in this cluster had no trouble expressing Newton's second law (SLa). These students worked diligently to solve the test case but were unable to do so. As a result, they did not proceed to the grading case.
Cluster B (19.8%) contained students who made mistakes while replacing the given values and initial conditions (any IC code except ICa). Some of these students worked with the grading case (ICc and ICg). Others might have been working with either case and had mixed conditions (ICb and ICe) or simply incorrect ones (ICf). Still others might have incorrectly assigned the exponent on the length unit of k to the exponent of k in scientific notation (ICd). At any rate, most students in this cluster were able to construct a working albeit incorrect program. Given their unfamiliarity with general central force interactions, these students might have believed their solutions were correct. In fact, it is possible that students who were working with the grading case (ICc and ICg) had solved the test case correctly and simply made a typo.
Students in cluster C worked with either the grading or test case and might have made a number of mistakes with their initial conditions (any IC code except ICa). The dominant TABLE VI. Only seven of the thirty clusters with an inter-cluster distance of less than 0.5 were populated by more than 3% of the students. The bottom 18 clusters were populated by less than 1% of students each. These seven clusters accounted for 86.5% of students. The percentage of affirmatives for each code (Table II)  Students in cluster E (7.6%) tended to raise the separation vector to a power (FCd) and have mixed initial conditions (ICb, ICd, ICe and ICf). These students generally had no difficulty with writing Newton's second law correctly (SLa). The dominant error for students in cluster E was raising the separation vector to a power (FCd). This mathematically impossible operation raises a VPython error. Students in this cluster were unable to parse this error into any useful information.
Cluster F (7.1%) contained students who worked solely with the test case (ICa) and either had no issue with their force calculation (FCb) or had no evident force calculation procedure Students in cluster G (4.1%) all invented an incorrect unit vector for the force rather than usingr (FCf) regardless of the case with which they worked (ICc, ICb, and ICf). These students generally had no difficulty updating the momentum using Newton's second law (SLa). Most likely, these students computed the magnitude of the force (similar to students in cluster C) but were able to parse the resulting VPython error. Students in cluster G corrected their mistake by assigning some unit vector to the force before the momentum was updated.

VIII. CLOSING REMARKS
Students with little to no programming experience can, in large introductory physics courses, develop the computational skills necessary to predict the motion of sundry physical systems. After solving a suite of computational homework problems, most students (∼ 60%) were able to model the motion of a novel problem successfully. In our work, we discovered that most students who were unsuccessful encountered challenges when calculating the net 24 force acting on the object in the motion prediction algorithm (Clusters A and C through G in Table VI). By contrast, there were fewer students whose primary challenge was identifying and assigning variables (Cluster B in Table VI). We acknowledge that we have limited the development of our students' computational skill set to contextualizing a word problem into a programming task, identifying and updating input variables, and applying a motion prediction algorithm. We believe that further development of our homework problems and other novel deployments could broaden the scope of the skills students develop.
Our work provides a touchstone example of computational instruction for the STEM education community. This work provides clear evidence that newcomers to both computer programming and computational modeling can successfully acquire, in a large lecture introductory STEM course, basic skills needed to do computational modeling. However, the success of our students on the proctored assignment should not be overemphasized. Ours is but one of many possible implementations that might be successful in a large lecture setting.
Moreover, the level of instruction in this course was limited to the iterative prediction of motion. What is more fascinating are the types of errors which students made when solving the proctored assignment and the commonalities among those errors.
Procedural errors such as those we have documented (Secs. VI & VII) could be corrected by developing additional materials aimed at addressing each error individually. However, the results from this work indicate that instructional efforts should be focused not only on correcting procedural mistakes but also on developing students' qualitative habits of mind.
Training students to write programs to predict motion might help them to be successful in a highly structured environment, but they would be better served by learning the practice of debugging. Here, debugging includes identifying syntax errors (of which we found few) and, more importantly, performing the type of qualitative analysis that is typically taught for solving analytic problems. Students who could synthesize their analytic and computational skills would be better prepared to solve the open-ended problems they will face in their future work. Students' use of debugging is the subject of active investigation by the computer science education community. [37][38][39] Developing the materials to teach these skills requires an evaluation of how students contextualize computational problems. We do not claim to understand this contextualization presently, although we have been able to glean some suggestive information based on students' errors. Investigating what students think about when solving computational problems requires structured student interviews (i.e., a think-aloud study). In the future, we plan to perform such a study to not only characterize students' abilities to contextualize but also to elucidate the mechanism for some of the errors we reported in Secs. VI & VII. Moreover, this work provides the foundation for others in the STEM community to perform similar investigations.
Research into skill development in math and science has shown a strong correlations with student epistemology. 40,41 Epistemology is important because the views that students hold affect how they learn 42 and, ultimately, how successful they are in their science courses. 43,44 It is therefore crucial that we understand students' sentiments about learning a new tool such as computation. Our students expressed anxiety and demonstrated a lack of self confidence in laboratory instruction and homework help sessions, even with their additional exposure to computational problems and improved scores on hand-written programming questions.
We are developing an attitudinal survey aimed at exploring these and other beliefs in detail.
Students who learn to use computational modeling and are confident in their abilities will be better prepared to solve challenging problems.
We do not claim to have assessed a transfer of computational knowledge. We designed a set of problems (Sec. III) that students solved over the course of the semester with an eye toward a final assessment of their skills using a novel problem. This problem (Sec. IV) was similar to some of the homework problems students had solved previously. It focused on key skills that we desired students to acquire: contextualizing a problem, identifying and assigning variables in a program, and carrying out the motion prediction algorithm. An evaluation of transfer would require that students apply these computational skills to a different domain (e.g., electromagnetism) or a different task (e.g., open-ended inquiry). Demonstrating transfer of computational knowledge is a necessary step in developing students into flexible problem solvers for the 21st century.
It is the goal of many reforms in physics education to develop students into flexible problem solvers while exploring the practice of science. Teaching computational modeling alongside physics provides support for that effort. Students learn the tools for doing science while developing a qualitative understanding of physical systems, exploring the generality of physics principles, and learning broadly applicable problem solving methods in computation.
apply. This code excluded the integration time. It was intended that the larger mass object was to remain at its location (See IC1).
IC3 -Student used the correct integration time from either the grading case or test case.
A student must replace the default integration time (1 s) with the values given in the case with which they intended to work (grading or test). A student who mixed initial conditions was given an affirmative on this code if the majority of their initial conditions were from the same case as the integration time.
IC4 -Student used mixed initial conditions. A student who used some but not all of the initial conditions from any of the cases (default, test, or grading) was given an affirmative on this code. This code excluded the integration time.
IC5 -Students confused the exponents on the units the exponent of k (interaction constant). Many students incorrectly thought the exponent on the length unit of the interaction constant was the scientific notation exponent for the interaction constant itself. For example, a student thought k = 0.1 Nm 3 meant k = 100 rather than k = 0.1 Newton times meters cubed.

Implementing the force calculation (FC) Codes
We reviewed how the students employed the force calculation algorithm in each of the programs written for the proctored assignment. The partially-written program given to the students left out all statements related to the force calculation. Students were required to fill in this procedure using the appropriate VPython syntax. We present the codes used to categorize each student's program with respect to computing the vector force acting on the low-mass object.
FC1 -The force calculation was correct. For this code to apply, a student must compute the separation vector, its magnitude, its unit vector, the magnitude of the force, and the vector force correctly. Each of these steps may be combined as long as the final result computes the vector force acting on the less massive particle at each instant. These steps must all appear in the numerical integration loop.
FC2 -The force calculation was incorrect, but the calculation procedure was evident. In the numerical integration loop, the student must perform a position vector subtraction, a calculation of the force magnitude and some attempt at combining magnitude with unit 28 vector (any unit vector was acceptable) for this code to apply. If a student treated the problem using components and had some force which is a vector, it was coded as evident.
If any part of the calculation was performed outside the loop, it was coded as not evident.
FC3 -The student attempted to raise the separation vector ( r) to a power. Students who raised the separation vector to a power generated a VPython exception error: unsupported operand type(s) for ** or pow(): 'vector' and 'int'.
This error told them that VPython cannot raise a vector to a power, as it is a mathematically impossible operation.
FC4 -The direction of the force was reversed. Students had to assign the correct unit vector and sign to the force depending on whether their force was attractive or repulsive for this code to be used. This code was not used if the student calculated the force as a magnitude only, raised r to a power, or invented a unit vector (e.g., 1, 0, 0 ). Visual feedback (i.e., the lower mass particle flying off to infinity) indicated a simple sign mistake.
FC5 -Student had some other force direction confusion. Some students used vectors other than r or − r to compute F . Other students computed the force as a magnitude and then multiplied it by an "invented" unit vector (e.g., 1, 0, 0 ,p). Both of these errors were given an affirmative for this code.

Updating with the Newton's second law (SL) Codes
We reviewed how the students employed the momentum update in each of the programs written for the proctored assignment. The partially written program given to the students left out the one line of code necessary to update the momentum. Students were required to fill in this line using the appropriate VPython syntax. We present the codes used to categorize each student's program with respect to updating the momentum of the low-mass particle.
SL1 -Newton's second law was correct. Correct Newton's second law meant that it was "correct as a physics principle" and also that it appeared "in the update form". This meant that pfinal = pinitial + Fnet*deltat alone in a loop did not fall under "correct Newton's second law". It is an incorrect update form.
SL2 -Newton's second law was incorrect but in a form that updates. Newton's second law updates the momentum, but not necessarily correctly. (e.g., p = p + Fnet, p = p +