Extending item response theory to online homework

Item response theory (IRT) becomes an increasingly important tool when analyzing “big data” gathered from online educational venues. However, the mechanism was originally developed in traditional exam settings, and several of its assumptions are infringed upon when deployed in the online realm. For a large-enrollment physics course for scientists and engineers, the study compares outcomes from IRT analyses of exam and homework data, and then proceeds to investigate the effects of each confounding factor introduced in the online realm. It is found that IRT yields the correct trends for learner ability and meaningful item parameters, yet overall agreement with exam data is moderate. It is also found that learner ability and item discrimination is robust over a wide range with respect to model assumptions and introduced noise. Item difficulty is also robust, but over a narrower range.

Figure 1

An example of a homework problem from the course under investigation. Different students have different masses and angles, as well as different multiple-choice options in different order. In this course, students have 99 tries on the free-form part of the problem. Students can use any physical unit with the right dimension.

Figure 2

An example of an exam problem from the course under investigation. Different students have different relative masses ($m_1<m_2$ versus $m_1>m_2$), as well as different speeds and angles $\theta$. The numerical answer options and predetermined physical unit are the automated exam rendering of a free-form problem.

Figure 3

Item characteristic curves for exam items (left) and homework items (right). Homework items were considered correct if the learner solved them eventually, regardless of number of attempts.

Figure 4

Item characteristic curves for homework. Homework items were considered correct if they were solved on the first attempt.

Figure 5

Log-likelihood of the 2PL IRT analysis of the homework data as a function of the maximum number $N$ of considered attempts. Missing data points indicate that the fit did not converge. For comparison, without imposing a threshold on the number of attempts, the log-likelihood is $-30070$.

Figure 6

Average difficulty of the homework items as a function of the maximum number $N$ of considered attempts. Missing data points indicate that the fit did not converge. The error bars indicate the average of the standard errors on the difficulties.

Figure 7

Average discrimination of the homework items as a function of the maximum number $N$ of considered attempts. Missing data points indicate that the fit did not converge. The error bars indicate the average of the standard errors on the discriminations.

Figure 8

Cosine similarity of the item difficulties from the “clean” exam values as a function of simulated copy or guess occurrence. Values are shown for a success rate of 10% (“educated guessing”), 50%, and 90% (“copying”).

Figure 9

Cosine similarity of the item discriminations from the “clean” exam values as a function of simulated copy or guess occurrence. Values are shown for a success rate of 10% (“educated guessing”), 50%, and 90% (“copying”).

Figure 10

Cosine similarity of the learner abilities from the “clean” exam values as a function of simulated copy or guess occurrence. Values are shown for a success rate of 10% (“educated guessing”), 50%, and 90% (“copying”).