How to quantify the efficiency of a pedagogical intervention with a single question

In many situations, the change in the conceptual understanding of students is measured using a single question. This is, for instance, the case in peer instruction where students answer twice to the same questions, before and after the discussion phase. Using item response theory and assuming that students proficiencies are normally distributed, it is shown that the Cohen’s $d$ effect size characterizing the change of mean proficiencies can be estimated by taking 0.6 times the log of the odds ratio of class scores. Moreover the polychoric correlation coefficient between students’ answers is suggested as an additional indicator to detect abnormal changes in scores when its value is below 0.3. Taken together, these two indicators give both a precise measurement of a pedagogical intervention—a peer discussion or something else—and a coefficient of security to detect random answers or poor writing of questions. The application is made to the evaluation of peer discussions that took place in an introductory mechanics course taught using peer instruction.

Figure 1

The logit function. Vertical abscissa is $0.6\ln [p/(1-p)]$, where $p$ is given by the horizontal abscissa.

Figure 2

Distributions of students’ proficiencies. Top, before the intervention; bottom, after the intervention. The vertical dashed line is the question difficulty. Proportions of correct answers correspond to gray areas.

Figure 3

Isovalues of the Cohen’s $d$ effect size (Eq. (6): no effect ($d=0$), small ($d=0.2$), medium ($d=0.5$), large ($d=0.8$), and very large ($d=1.2$).

Figure 4

Results of the peer discussion for the 37 ConcepTests. Each point is the proportion of correct students’ answers to a given ConcepTest. Lines are isovalues of the Cohen’s $d$ effect size (no effect, small, medium, large, and very large effect).

Figure 5

Effect size as a function of the correlation coefficient estimated using ML for the 37 ConcepTests. Horizontal lines correspond to values for small (0.2), medium (0.5), large (0.8), and very large (1.2) effect sizes. The vertical line corresponds to $\rho =0.3$.

Figure 6

Values of the correlation coefficient using Eq. (7) as a function of its value using the maximum likelihood method.

Figure 7

Effect size as a function of the correlation coefficient estimated using Eq. (7) for the 37 ConcepTests. Horizontal lines correspond to values for small (0.2), medium (0.5), large (0.8), and very large (1.2) effect sizes. The vertical line corresponds to $\rho =0.3$.