Multidimensional item response theory and the Brief Electricity and Magnetism Assessment

This work is the fourth of a series of papers applying multidimensional item response theory (MIRT) to widely used physics conceptual assessments. This study applies MIRT analysis using both exploratory and confirmatory methods to the Brief Electricity and Magnetism Assessment (BEMA) to explore the assessment’s structure and to determine a well-fitting model of student knowledge measured by the assessment. These methods were used to investigate a large dataset ($N=9666$) from a research university in the United States. Exploratory analysis showed that a five-factor model had the best fit statistics; the items with the highest loadings in four of the five factors were items in the same item block. Confirmatory MIRT analysis fit a theoretical model developed from expert solutions to the instrument and identified two models with superior model fit: a principle model and a topical model. The principle model consisted of 28 principles, fundamental reasoning steps needed to solve items in the instrument; this was more principles than any of the models in the previous confirmatory MIRT studies of the Force Concept Inventory, the Force and Motion Conceptual Evaluation, and the Conceptual Survey of Electricity and Magnetism. A second model, the topical model, consisted of five general subtopics of electromagnetism. Both the principle and the topical model had excellent fit statistics; however, unlike the other conceptual instruments studied, the topical model had better fit statistics. The five topical divisions were explored as possible subscales; however, none of these subscales had a Cronbach’s $\alpha$ of 0.7, the minimum value for required low-stakes testing.

Figure 1

Probability of selecting the correct response, $\pi (\theta )$, versus ability $\theta$ using $d=0$ and $a=1$. The dashed line represents the slope at $\theta =0$ and has slope $a/4=0.25$.

Figure 2

Correlation matrix. Solid (green) lines represent positive correlations; dashed (red) lines negative correlations. Thicker lines represent larger correlations.

Figure 3

Partial correlation matrix. Solid (green) lines represent positive correlations; dashed (red) lines negative correlations. Thicker lines represent larger correlations.