Test equity in developing short version conceptual inventories: A case study on the Conceptual Survey of Electricity and Magnetism

Standardized concept inventories (CIs) have been widely used in science, technology, engineering, and mathematics education for assessment of student learning. In practice, there have been concerns regarding the length of the test and possible test-retest memory effect. To address these issues, a recent study developed a method to split a CI into two equivalent short CIs, which have been shown to provide equivalent mean score measures. However, the previous approach does not fully examine common requirements of test equating. This study extends the existing method with test equating analysis to form a revised algorithm for developing and validating equivalent short CIs. The method is applied to split the Conceptual Survey of Electricity and Magnetism (CSEM) into two half-length CSEMs (HCSEMs). Through a series of test-equating and validation analysis, the HCSEMs are confirmed to measure the same content clusters and the same construct of students’ understanding as that of the original CSEM at a similar level of reliability regarding their question design features. Furthermore, a best performing equating function to convert scores among the two HCSEMs and the CSEM is identified through comparisons of four commonly used equating functions. The conversions are further validated using the existing data sets as well as a data set collected from an empirical study using randomized testing. The results confirm that the three CIs can provide equivalent measures in different subpopulations. Overall, the results from this study have shown that the revised algorithm can provide a more complete framework in developing and validating equivalent short CIs. In addition, this method can also be applied to develop and validate parallel CIs in general.

Figure 1

The revised algorithm for developing and validating short CIs.

Figure 2

Wright maps for CSEM, HCSEM1, and HCSEM2 from left to right. In each Wright map, the left-hand side is the distribution of student ability on a logit scale; while the right-hand side is the distribution of item difficulty on the same logit scale. Items are labeled with Q1–Q32 according to their original order in CSEM. In each plot of the distribution of student ability, $M$, $S$, and $T$ are used to show the mean value, one standard deviation, and two standard deviations of the distribution.

Figure 3

Results for equating HCSEM1 to CSEM. (a) Equating functions. (b) Parametric bootstrapped SE. (c) Parametric bootstrapped bias. (d) Parametric bootstrapped RMSE.

Figure 4

Test characteristic curves (TCCs) of the two HCSEMs and the original CSEM. The three TCCs are nearly identical, overlapping on top of each other throughout the range of the abilities.

Figure 5

Standardized mean differences between DS1 and DS2 for the three CIs.

Figure 6

Mean differences of measured scores (shown as squares) and 90% confidence intervals (CIs; thick horizontal lines) and 95% CIs (thin horizontal lines) for the three CIs. Cohen’s $d$ equivalence boundaries are ${\mathrm{\Delta }}_L=-0.49$ and ${\mathrm{\Delta }}_U=0.49$. Corresponding measured score equivalence boundaries are ${\mathrm{\Delta }}_L=-8.72\%$ and ${\mathrm{\Delta }}_U=8.72\%$, ${\mathrm{\Delta }}_L=-7.78\%$ and ${\mathrm{\Delta }}_U=7.78\%$, ${\mathrm{\Delta }}_L=-8.68\%$ and ${\mathrm{\Delta }}_U=8.68\%$ for CSEM vs HCSEM1(red dashed lines), CSEM vs HCSEM2 (green dashed lines), and HCSEM1 vs HCSEM2 (blue dashed lines), respectively.

Figure 7

Mean differences of equated scores (shown as squares) and 90% confidence intervals (CIs; thick horizontal lines) and 95% CIs (thin horizontal lines) for the three CIs. Cohen’s $d$ equivalence boundaries are ${\mathrm{\Delta }}_L=-0.49$ and ${\mathrm{\Delta }}_U=0.49$. Corresponding measured score equivalence boundaries are ${\mathrm{\Delta }}_L=-8.26\%$ and ${\mathrm{\Delta }}_U=8.26\%$, ${\mathrm{\Delta }}_L=-7.67\%$ and ${\mathrm{\Delta }}_U=7.67\%$, ${\mathrm{\Delta }}_L=-8.32\%$ and ${\mathrm{\Delta }}_U=8.32\%$ for CSEM vs HCSEM1 (red dashed lines), CSEM vs HCSEM2 (green dashed lines), and HCSEM1 vs HCSEM2 (blue dashed lines), respectively.

Figure 8

Results for equating HCSEM2 to CSEM. (a) Equating functions. (b) Parametric bootstrapped SE. (c) Parametric bootstrapped bias. (d) Parametric bootstrapped RMSE.

Figure 9

Results for equating HCSEM1 to HCSEM2. (a) Equating functions(b) Parametric bootstrapped SE. (c) Parametric bootstrapped bias. (d) Parametric bootstrapped RMSE.