Assessment of representational competence in kinematics

A two-tier instrument for representational competence in the field of kinematics (KiRC) is presented, designed for a standard (1st year) calculus-based introductory mechanics course. It comprises 11 multiple choice (MC) and 7 multiple true-false (MTF) questions involving multiple representational formats, such as graphs, pictures, and formal (mathematical) expressions (1st tier). Furthermore, students express their answer confidence for selected items, providing additional information (2nd tier). Measurement characteristics of KiRC were assessed in a validation sample (pre- and post-test, $N=83$ and $N=46$, respectively), including usefulness for measuring learning gain. Validity is checked by interviews and by benchmarking KiRC against related measures. Values for item difficulty, discrimination, and consistency are in the desired ranges; in particular, a good reliability was obtained ($\mathrm{KR}20=0.86$). Confidence intervals were computed and a replication study yielded values within the latter. For practical and research purposes, KiRC as a diagnostic tool goes beyond related extant instruments both for the representational formats (e.g., mathematical expressions) and for the scope of content covered (e.g., choice of coordinate systems). Together with the satisfactory psychometric properties it appears a versatile and reliable tool for assessing students’ representational competency in kinematics (and of its potential change). Confidence judgments add further information to the diagnostic potential of the test, in particular for representational misconceptions. Moreover, we present an analytic result for the question—arising from guessing correction or educational considerations—of how the total effect size (Cohen’s $d$) varies upon combination of two test components with known individual effect sizes, and then discuss the results in the case of KiRC (MC and MTF combination). The introduced method of test combination analysis can be applied to any test comprising two components for the purpose of finding effect size ranges.

Figure 1

Coherent representations of an isomorphic kinematics problem: velocity-time graph; strobe picture (the dots record the position at equal time intervals), and vector picture (acceleration or net force vectors); differential equation ($g=$ Earth’s acceleration, $k=\text{constant}$); verbal problem statement and data table.

Figure 2

Two example KiRC items: Multiple choice item in graphical representation format about free fall with air resistance presenting three alternatives and a confidence rating scale (left), and multiple true-false items dealing with the concept of frame of reference (right).

Figure 3

KiRC test correlations. Note that MC3, MTF, and FCI measures were taken at the same time (pre-instruction). Math and physics scores reflect achievements in school, the exam score refers to performance on the total exam taken at the end of the semester (performance on one exam problem dealing with kinematics exclusively was investigated separately). Pearson’s $r$ is reported ($df=83$, $*p<0.05$, $**p<0.01$, $***p<0.01$)—the $p$ value refers to correlations significantly greater than zero. Irrelevant correlations have not been calculated and are indicated by the gray space.

Figure 4

Item 6 related sketch drawn by student L1 during the interview. Velocity-time graph was drawn first (ballpoint pen, blue), acceleration-time graph subsequently (pencil).

Figure 5

Test combination analysis: Pre-post effect size (Cohen’s $d$) of composite test score $T=T_1+\kappa T_2$ as a function of $\kappa$. According to Eq. (11), data points were obtained from raw pre- and post-test scores by calculating the composite test score for each student using 50 different values of $\kappa$. The fit line was determined by Eq. (12) using ${\theta }_1$, ${\theta }_2$, and $\varphi$ calculated from the data. Note that the $\kappa$ axis is logarithmic in order to illustrate the relevant values of $\kappa$ according to Table 5 with sufficient resolution.

Figure 6

KiRC results by question for the introductory physics course ordered by pre-instruction item difficulty. The left part of the diagram shows the MC3 items; the right part shows the MTF items.

Figure 7

KiRC item difficulties obtained in winter term WT14/15 and WT16/17. Each point represents one item (item 1 being highlighted). The dashed line represents equal item difficulties. $R^2=0.70$.