Representing uncertainty on model analysis plots

Model analysis provides a mechanism for representing student learning as measured by standard multiple-choice surveys. The model plot contains information regarding both how likely students in a particular class are to choose the correct answer and how likely they are to choose an answer consistent with a well-documented conceptual model. Unfortunately, Bao’s original presentation of the model plot did not include a way to represent uncertainty in these measurements. I present details of a method to add error bars to model plots by expanding the work of Sommer and Lindell. I also provide a template for generating model plots with error bars.

Figure 1

Various regions of a model plot with two well-documented mental models. Figure recreated from Ref. [1] and originally published in Ref. [4], where ${\sigma }_{\mu }^2$ is the $\mu$th eigenvalue of the class density matrix, and $v_{k,\mu }$ is the $k$th component of the $\mu$th eigenvector.

Figure 2

Model plots for the example data: (a) no error bars, (b) assuming equal fractional uncertainties for both coordinates and equal initial error estimates for all models (pooled standard error), and (c) assuming coordinate-specific uncertainties and model-specific initial error estimates (standard error of the distribution for each model).