Classical test theory and item response theory comparison of the brief electricity and magnetism assessment and the conceptual survey of electricity and magnetism

For proper assessment selection understanding the statistical similarities amongst assessments that measure the same, or very similar, topics is imperative. This study seeks to extend the comparative analysis between the brief electricity and magnetism assessment (BEMA) and the conceptual survey of electricity and magnetism (CSEM) presented by Pollock. This is accomplished by using large samples ($N_{\mathrm{BEMA}}=5368$ and $N_{\mathrm{CSEM}}=9905$) within classical test theory (CTT) and item response theory (IRT) frameworks. For the IRT comparison, after consideration of the conceptual content addressed in each assessment, it was assumed that each of these assessments are measuring the same student latent ability ($\theta$), specifically a student’s ability to do introductory electricity and magnetism. Via a CTT and IRT analysis it was found that both assessments are essentially equal in overall difficulty. Classical item analysis applied to 7 questions used by both assessments revealed that each assessment functions slightly differently internally. The test information curves found from IRT show that the CSEM has superior information compared to the BEMA in estimating student latent abilities for the entire range of typical latent abilities achieved by students on each assessment, $\theta \approx -2$ to $\theta \approx 3$. Information in this case is interpreted as how well a student’s latent ability was estimated by an assessment as a function of latent ability. When the circuits questions are removed from the BEMA the majority of the information is lost in the $\theta \approx 0$ to $\theta \approx 2$ range. This means the circuits questions on the BEMA are information heavy for higher ability scores. So, special considerations should be made as to which assessment a study uses depending on the specific questions a researcher is attempting to answer.

Figure 1

Histograms of the fraction of the total number of students for all possible total scores. For the BEMA there were no scores below 3 and for the CSEM there were no scores below 2. The vertical red line indicates the sample mean and blue is for the median.

Figure 2

The error bars are $\pm 2$ standard deviations from 10 000 randomly sampled classes of 250 students from the full sample for the BEMA. The horizontal red lines indicate an ideal range of difficulties for well-functioning questions. The colored bars are the 7 identical items between the BEMA and CSEM.

Figure 3

The error bars are $\pm 2$ standard deviations from 10 000 randomly sampled classes of 250 students from the full sample for the CSEM. The horizontal red lines indicate the range of ideal difficulties for well-functioning questions. The colored bars are the 7 identical items between the BEMA and CSEM.

Figure 4

The error bars are $\pm 2$ standard deviations from 10 000 randomly sampled classes of 250 students from the full sample for both the BEMA and the CSEM. The horizontal red line indicate the minimum appropriate discrimination for well-functioning questions. The upper and lower 27% was used to calculate the classical discrimination. The colored bars are the 7 identical items between the BEMA and CSEM.

Figure 5

The point biserial for each of the items in the BEMA and CSEM. The red line is at 0.2, which is an accepted minimum for item point-biserial values. There error bars represents the 95% confidence interval for each of the bins, respectively. The colored bars are the 7 identical items between the BEMA and CSEM.

Figure 6

Scree plots for each of the assessments. The hollow boxes represent the BEMA’s eigenvalues and the solid dots represent the CSEM. These plots support the assumption that these assessments can be treated as unidimensional for an IRT model.

Figure 7

An example of an item characteristic curve (solid, black) and its associated item information curve (dashed, black) using the 2PL model. When the student ability is equal to the item difficulty ($\theta =\delta$) the probability of the student answering the questions correctly is 50%; this is also the location of the maximum for the item information curve. The slope ($m$) of the tangent line (dot-dashed, blue) at the 50% mark is proportional to the item discrimination ($\alpha$). $D$ is a constant generally set so 1.702 so that the ability axes better lines up with spacings that are equal to a standard deviation of student ability.

Figure 8

Item response theory test information curves for the CSEM (blue, solid), BEMA (purple, dashed), and BEMA(R) (purple, dot-dashed).

Figure 9

Plotted in the CSEM scale, the histogram of estimated student abilities for the BEMA is overlaid with the BEMA’s test information curve. This shows that the bulk of the students are located in a region of ability space where their latent abilities will be well estimated.

Figure 10

Plotted in the CSEM scale, the histogram of estimated student abilities for the CSEM is overlaid with the CSEM’s test information curve. This shows that the bulk of the students are located in a region of ability space where their latent abilities will be well estimated.

Figure 11

Item characteristic curves for the item 1–15 on the BEMA, in the CSEM scale. The gray error bars represent 1 standard deviation, from top to bottom, in the student responses. The horizontal axis is student latent ability.

Figure 12

Item characteristic curves for the item 16–31 on the BEMA, in the CSEM scale. The gray error bars represent 1 standard deviation, from top to bottom, in the student responses. The horizontal axis is student latent ability.

Figure 13

Item characteristic curves for the item 1–15 on the CSEM. The gray error bars represent one standard deviation, from top to bottom, in the student responses. The horizontal axis is student latent ability.

Figure 14

Item characteristic curves for the item 16–30 on the CSEM. The gray error bars represent one standard deviation, from top to bottom, in the student responses. The horizontal axis is student latent ability.

Figure 15

Item characteristic curves for the item 31–32 on the CSEM. The gray error bars represent one standard deviation, from top to bottom, in the student responses. The horizontal axis is student latent ability.