Assessing learning outcomes in middle-division classical mechanics: The Colorado Classical Mechanics and Math Methods Instrument

Reliable and validated assessments of introductory physics have been instrumental in driving curricular and pedagogical reforms that lead to improved student learning. As part of an effort to systematically improve our sophomore-level classical mechanics and math methods course (CM 1) at CU Boulder, we have developed a tool to assess student learning of CM 1 concepts in the upper division. The Colorado Classical Mechanics and Math Methods Instrument (CCMI) builds on faculty consensus learning goals and systematic observations of student difficulties. The result is a 9-question open-ended post test that probes student learning in the first half of a two-semester classical mechanics and math methods sequence. In this paper, we describe the design and development of this instrument, its validation, and measurements made in classes at CU Boulder and elsewhere.

Figure 1

Certain topic-scale learning goals are evaluated by the CCMI questions. The sample question appears on the CCMI pre- and post tests; vector calculus is a prerequisite for CM 1. This question constitutes 9% of the total post-test score.

Figure 2

Grading rubric for the question appearing in Fig. 1. The format focuses the grader’s attention on the final response provided by the student. The grading rubric was not designed to elucidate details of student difficulties, but rather to capture the common final responses provided by students and score them accordingly.

Figure 3

The absolute difference in CCMI scores assigned by a trained and untrained grader is presented. The average difference on total CCMI score between the trained and untrained grader is 3.5% $\pm 2.7\%$. The graders agreed to 5% on overall score for 79% of the exams.

Figure 4

For each question, the score differences between a trained and untrained grader are shown. These percentage agreement between graders was binned as exact (same score, blue), close (within $\pm 20\%$, orange), moderate (within $\pm 20\%–50\%$, purple), and poor (more than $\pm 50\%$, gray) agreement. For all questions, exact agreement was the most prevalent form of agreement ($\ge 50\%$ of exams for all questions).

Figure 5

A histogram of student post-test scores on the CCMI is presented ($N=462$; CU and other institutions). The average score for students taking the CCMI is indicated (orange arrow): $49.0\%\pm 1.0\%$ as well as the performance by first-year physics graduate students at CU Boulder (green arrow): $74.5\%\pm 3.4\%$ ($N=5$).

Figure 6

The mean—blue bar—and median—orange bar—of student performance on each question is provided. The median score for question 2 is zero. In addition, we show the score—gray square—the score that separates the top three deciles from the bottom seven deciles, that is, the score which the top 30 percent of scores lies above.