Students’ proficiency scores within multitrait item response theory

In this paper we present a series of item response models of data collected using the Force Concept Inventory. The Force Concept Inventory (FCI) was designed to poll the Newtonian conception of force viewed as a multidimensional concept, that is, as a complex of distinguishable conceptual dimensions. Several previous studies have developed single-trait item response models of FCI data; however, we feel that multidimensional models are also appropriate given the explicitly multidimensional design of the inventory. The models employed in the research reported here vary in both the number of fitting parameters and the number of underlying latent traits assumed. We calculate several model information statistics to ensure adequate model fit and to determine which of the models provides the optimal balance of information and parsimony. Our analysis indicates that all item response models tested, from the single-trait Rasch model through to a model with ten latent traits, satisfy the standard requirements of fit. However, analysis of model information criteria indicates that the five-trait model is optimal. We note that an earlier factor analysis of the same FCI data also led to a five-factor model. Furthermore the factors in our previous study and the traits identified in the current work match each other well. The optimal five-trait model assigns proficiency scores to all respondents for each of the five traits. We construct a correlation matrix between the proficiencies in each of these traits. This correlation matrix shows strong correlations between some proficiencies, and strong anticorrelations between others. We present an interpretation of this correlation matrix.

Figure 1

The standard item response curve of a three-parameter model depicted together with the graphical interpretation of each corresponding parameter, $a$, $d$, and $g$.

Figure 2

Different information criteria are plotted for item response models with varying number of traits. Although Akaike’s Information Criterion and its corrected version (AICc) show a broad minimum plateau for $5\le n\le 7$ the other two information criteria have a definite minimum at $n=5$.

Figure 3

The item characteristic curves shown for a single-trait two-parameter model for all 30 items. It is apparent that the $-3\le \theta \le 3$ is well covered by items, and also there are at least two items (question 15 and 26) which poll the students with higher skills. These two items correspond to the two rightmost curves in this graph. The overall test information curve has also been depicted with a heavier line and with its scale denoted on the right vertical axis.

Figure 4

The histogram of the students’ proficiency scores (right) with the individual item difficulties (left) for the single-trait model. On the right-hand side the smoothed distribution of the skill (bold dashed line) and the test information curve (solid line) have also been depicted.