Comparison of integrated testlet and constructed-response question formats

Constructed-response (CR) questions are a mainstay of introductory physics textbooks and exams. However, because of the time, cost, and scoring reliability constraints associated with this format, CR questions are being increasingly replaced by multiple-choice (MC) questions in formal exams. The integrated testlet (IT) is a recently developed question structure designed to provide a proxy of the pedagogical advantages of CR questions while procedurally functioning as set of MC questions. ITs utilize an answer-until-correct response format that provides immediate confirmatory or corrective feedback, and they thus allow not only for the granting of partial credit in cases of initially incorrect reasoning, but, furthermore, the ability to build cumulative question structures. Here, we report on a study that directly compares the functionality of ITs and CR questions in introductory physics exams. To do this, CR questions were converted to concept-equivalent ITs, and both sets of questions were deployed in midterm and final exams. We find that both question types provide adequate discrimination between stronger and weaker students, with CR questions discriminating slightly better than the ITs. There is some indication that any difference in discriminatory power may result from the baseline score for guessing that is inherent in MC testing. Meanwhile, an analysis of interrater scoring of the CR questions raises serious concerns about the reliability of the granting of partial credit when this traditional assessment technique is used in a realistic (but nonoptimized) setting. Furthermore, we show evidence that partial credit is granted in a valid manner in the ITs. Thus, together with consideration of the vastly reduced costs of administering IT-based examinations compared to CR-based examinations, our findings indicate that ITs are viable replacements for CR questions in formal examinations where it is desirable both to assess concept integration and to reward partial knowledge, while efficiently scoring examinations.

Figure 1

Examples of concept-equivalent constructed-response and integrated testlet questions. Integrated testlets comprise a set of MC items with varying levels of integration. The CR and IT questions share a common stimulus, and the final multiple-choice item in the IT is the same as the final CR subquestion. (a) CR3 and IT3 are examples from the midterm exam and cover concepts such as 2D projectile motion and kinetic friction. (b) CR8 and IT8 are examples from the final exam and cover concepts such as rotational motion, torque, and work. Note that IT8-iii exists to cue students to the most efficient means of solving IT8-iv. By contrast, such cuing is absent in CR8.

Figure 2

A comparison of item difficulty parameter $p^{\prime }$ for each matching pair of constructed response and integrated testlet questions. The solid line labeled equivalency represents the expected relationship between questions of equal difficulty. Similarly, the solid line labeled $\mathrm{equivalency}+\mathrm{guessing}$ represents the case of equivalent difficulty where those students who do not know the answer choose to guess. Note that the majority of data points lie between these two lines.

Figure 3

A conceptual and procedural map of the two concept-equivalent exam structures shown in Fig. 1. The various subitems in the constructed response are labeled and highlighted by dotted stipling. Individual items within a testlet are labeled and highlighted with gray shading. (a) Constructed response question 3(a)–3(c) (CR3) and a 4-item testlet (IT3). $\mathrm{FBD}=\mathrm{free}\text{−}\mathrm{body}$ diagram; $d_{\text{free-fall}}=\mathrm{horizontal}$ ice landing distance from roof edge. (b) Constructed response questions 8(a),8(b) and 4-item testlet 8 (IT8). $\mathrm{cyl}=\mathrm{cylinder}$; $v_{\text{bucket}}^{\prime }=\mathrm{bucket}$’s speed at ground height. Arrows indicate which concepts and parameters are needed for developing other concepts and parameters. Two alternate conceptual approaches to the final step are indicated. Unlike in the CR question, the integrated testlet cues and builds scaffolding for an easier approach to the final question.