Motivations for using the item response theory nominal response model to rank responses to multiple-choice items

Several recent studies have employed item response theory (IRT) to rank incorrect responses to commonly used research-based multiple-choice assessments. These studies use Bock’s nominal response model (NRM) for applying IRT to categorical (nondichotomous) data, but the response rankings only utilize half of the parameters estimated by the model. We present a mathematical argument for why this practice of using half of the NRM parameters when ranking responses is appropriate based on the primary question of multiple-choice tests: How can we use students’ responses to test items to estimate their overall knowledge levels? We provide additional motivation for this practice by recognizing the similarities between Bock’s NRM and the probability function of the canonical ensemble with degenerate energy states. As physicists often do, we exploit these mathematical similarities to gain new insights into the meaning of the IRT parameters and a richer understanding of the relationship between these parameters and student knowledge.

Figure 1

IRT category characteristic curves showing the probability of selecting a particular response for item 14 of the FMCE, given a students’ latent trait $\theta$. Curves were generated from Eq. (3) using the $\{a_{k,14},d_{k,14}\}$ parameters from Table 1.

Figure 2

Probability distributions for a student having a particular value of $\theta$, given that they chose one of the response options to item 14 on the FMCE. Curves were generated from Eq. (5) using the parameters from Table 1, and assuming a standard normal distribution for $P(\theta )$ and values of $P_{14}(k)$ that normalize the probability distribution.

Figure 3

Probability distributions for the 3-particle spin-$1/2$ paramagnet as a function of the temperature parameter $-\beta$. The horizontal axis has been scaled by $\mu H$ to obtain a dimensionless quantity. Because of the negative sign, colder temperatures are on the left and hotter temperatures are on the right. Macrostates are labeled by the net magnetization.

Figure 4

(a) Average energy $U$ for the three-particle spin-$1/2$ paramagnet, plotted as a function of the temperature parameter $-\beta$. All axes have been scaled to obtain dimensionless quantities. The dotted lines show the energy value for each macrostate, labeled by magnetization. (b) The IRT equivalent of average energy $U_{\mathrm{nrm}}$ plotted as a function of the latent trait $\theta$ for item 14 of the FMCE. The dotted lines show the $a_{k,14}$ energy value for each response option.