Analysis test of understanding of vectors with the three-parameter logistic model of item response theory and item response curves technique

This study investigated the multiple-choice test of understanding of vectors (TUV), by applying item response theory (IRT). The difficulty, discriminatory, and guessing parameters of the TUV items were fit with the three-parameter logistic model of IRT, using the parscale program. The TUV ability is an ability parameter, here estimated assuming unidimensionality and local independence. Moreover, all distractors of the TUV were analyzed from item response curves (IRC) that represent simplified IRT. Data were gathered on 2392 science and engineering freshmen, from three universities in Thailand. The results revealed IRT analysis to be useful in assessing the test since its item parameters are independent of the ability parameters. The IRT framework reveals item-level information, and indicates appropriate ability ranges for the test. Moreover, the IRC analysis can be used to assess the effectiveness of the test’s distractors. Both IRT and IRC approaches reveal test characteristics beyond those revealed by the classical analysis methods of tests. Test developers can apply these methods to diagnose and evaluate the features of items at various ability levels of test takers.

Figure 1

Item characteristic curve (ICC) of item 1 in the TUV, modeled from data on Thai students ($N=2392$).

Figure 2

Item 2 in the TUV, relating to the graphical representation of a unit vector in the direction of $\overrightarrow{A}=2\widehat{i}+2\widehat{j}$.

Figure 3

The ICCs for items 2, 5, 8, 17, and 19 in the TUV, according to data on Thai students.

Figure 4

Test information curve of the 20-item TUV across ability levels of the Thai students tested.

Figure 5

The IRC of item 1, showing the percentage of respondents by choice, with symbols for the choices being $\diamond =\mathrm{A}$, $\square =\mathrm{B}$, $\mathrm{\Delta }=\mathrm{C}$, $\mathrm{o}=\mathrm{D}$, and $\times =\mathrm{E}$. The correct choice was E. The horizontal scale is the ability in plot 1(a), and the total test score in plot 1(b).

Figure 6

IRCs for items 2, 3, 5, and 10, representing the fractions of respondents at any given ability that chose option $\diamond =\mathrm{A}$, $\square =\mathrm{B}$, $\mathrm{\Delta }=\mathrm{C}$, $\mathrm{o}=\mathrm{D}$, or $\times =\mathrm{E}$.