Rigging the deck: Selecting good problems for expert-novice card-sorting experiments

A seminal study by Chi et al. firmly established the paradigm that novices categorize physics problems by “surface features” (e.g., “incline,” “pendulum,” “projectile motion,” etc.), while experts use “deep structure” (e.g., “energy conservation,” “Newton 2,” etc.). Yet, efforts to replicate the study frequently fail, since the ability to distinguish experts from novices turns out to be highly sensitive to the problem set being used. Exactly what properties of problems are most important in problem sets that discriminate experts from novices in a measurable way? To answer this question, we studied the categorizations by known physics experts and novices using a large, diverse set of problems. This set needed to be large so that we could determine how well experts and novices could be discriminated by considering both small subsets using an exhaustive Monte Carlo approach and larger subsets using simulated annealing. We found that the number of questions required to accurately classify experts and novices can be surprisingly small so long as the problem set is carefully crafted to be composed of problems with particular pedagogical and contextual features. Finally, we found that not only was what you ask (deep structure) important, but also how you ask it (problem context).

Figure 1

(a) The PCA plot of the sorters for the entire set of problems from our previous study [2]. Both a Cramer’s test ($p=0.048$) and a Hotelling’s test ($p<{10}^{-5}$) find the expert and novice groups to be distinct at a 95% confidence level. (b) The PCA plot of the sorters considering only the problems from Singh’s study. Both a Cramer’s test ($p=0.041$) and a Hotelling’s test ($p<{10}^{-5}$) find the expert and novice groups to be distinct at a 95% confidence level. For both plots, PC1 is the coordinate along the first principal axis, and PC2 is the coordinate along the second principal axis. Sorters known by us to be experts are marked by circles while sorters known by us to be novices are marked by filled triangles. The second principal component discriminates experts from novices, as experts congregate higher on this axis than novices.

Figure 2

Histogram of the rigging fraction for the subsets with a random starting point. The shaded gray box indicates the rigging fraction of the Singh subset. From this, we can see that the Singh subset has more problems in common with its nearest local optimum than do any of the random subsets studied.

Figure 3

The pillars of expert-novice differentiation.