Improving test security and efficiency of computerized adaptive testing for the Force Concept Inventory

This paper presents improvements made to a computerized adaptive testing (CAT)-based version of the FCI (FCI-CAT) in regards to test security and test efficiency. First, we will discuss measures to enhance test security by controlling for item overexposure, decreasing the risk that respondents may (i) memorize the content of a pretest for use on the post-test or (ii) share information about the items with their classmates who take the assessment later. Second, we will discuss measures to enhance test efficiency, so that a shorter test length can yield a desired accuracy and precision of the measurement. Specifically, we utilized collateral information in the form of a pretest proficiency estimate of each respondent for selecting items and estimating respondent proficiency level in the post-test. To shorten the total testing time further, we also allowed the test lengths to be different for the pre- and post-test. To analyze how these improvements affect the accuracy and precision (which we measure in terms of root-mean-square error) of Cohen’s $d$, we conducted a Monte Carlo simulation and a post hoc simulation. Then, we calculated the minimal test length of the FCI-CAT whose accuracy and precision are equivalent to that of the paper-and-pencil version of the FCI. Consequently, we obtained the following three findings: (i) By using collateral information, we can achieve the accuracy and precision of the full-length FCI with fewer items via the FCI-CAT. (ii) For a class size of 40, we can control for test security while still reducing the sum of the pre- and post-test lengths of the FCI-CAT to a total of 33 items (17 items on the pretest and 16 items on the post-test), thereby reducing the testing time to 55%. (iii) If one’s goal is to maximize test efficiency, the pretest length should be slightly larger than the post-test length. On the other hand, if the goal is to maximize test security, the pretest length should be smaller and the post-test length should be larger. If one desires a balance of these two goals, it would be reasonable to choose equal pre- and post-test lengths.

Figure 1

Illustration of the use of collateral information for proficiency estimation on the post-test.

Figure 2

Comparison of the effects of test security algorithms on RMSE (maximum Fisher information, red; progressive, green; proportional, blue). The class size is 40 and the test length of the pre- and post-test is fixed to be the same. The plus symbol shows the result with content balancing and the circle symbol shows the result without content balancing.

Figure 3

Comparison of the effects of collateral information on RMSE for each test security algorithm (maximum Fisher information, progressive, and proportional). The $x$ ($y$) axis is the pre- (post-) test length. The class size is 40. The results show the case with content balancing. The light (dark) color indicates a low (high) value of RMSE, the bin width of the RMSE is set to 0.02 (except for the highest bin, where $\mathrm{RMSE}>0.26$).

Figure 4

The line $l_{\mathrm{post}}=29-l_{\mathrm{pre}}$ passing through the area $\mathrm{RMSE}<0.1$ in MFI with CI (Fig. 3, upper right). This derives the minimal test length as $l_{\mathrm{pre}}=16$ and $l_{\mathrm{post}}=13$ ($l_{\text{total}}=29$).