Empirical approach to interpreting card-sorting data

Since it was first published 30 years ago, the seminal paper of Chi et al. on expert and novice categorization of introductory problems led to a plethora of follow-up studies within and outside of the area of physics [Cogn. Sci. 5, 121 (1981)]. These studies frequently encompass “card-sorting” exercises whereby the participants group problems. While this technique certainly allows insights into problem solving approaches, simple descriptive statistics more often than not fail to find significant differences between experts and novices. In moving beyond descriptive statistics, we describe a novel microscopic approach that takes into account the individual identity of the cards and uses graph theory and models to visualize, analyze, and interpret problem categorization experiments. We apply these methods to an introductory physics (mechanics) problem categorization experiment, and find that most of the variation in sorting outcome is not due to the sorter being an expert versus a novice, but rather due to an independent characteristic that we named “stacker” versus “spreader.” The fact that the expert-novice distinction only accounts for a smaller amount of the variation may explain the frequent null results when conducting these experiments.

Figure 1

Simple example graph. When two problems are in the same category more than once (problems 1 and 9 as well as problems 3, 5, and 7 in this example), the edges drawn between those two corresponding vertices are thicker. The line width of each edge was taken proportional to the square of the number of connections between two vertices.

Figure 2

MSU physics study sorter graphs. Displayed from left to right are the categorization graphs for representative sorters. Sorters 2 and 16 were experts and sorters 20 and 30 were novices. Sorters 2 and 30 did very little multiple categorization, sorter 20 did a good bit of multiple categorization. Sorter 16 was unique in choosing to categorize each problem between 2 and 3 times.

Figure 3

Distribution of number of categories. Here we see the ECDFs of the number of category distributions for experts and novices separately. The faculty set is displayed using the dashed curve and the novice set is displayed using the dotted curve. We also compare these distributions to a sample ($N=1000$) shifted binomial distribution with probability $\rho =0.204$. A Kolmogorov-Smirnov test comparing these two distributions suggests that expert and novice categorizations are not distinguishable based on category number ($p=0.4793$), and both are well approximated by the same shifted binomial distribution.

Figure 4

Number of 3-cycles. This is the distribution of the number of 3-cycles for experts and novices. A Kolmogorov-Smirnov test suggests that experts and novices are not distinguishable based on their 3-cycle distributions ($p=0.1584$).

Figure 5

Maximum clique size. This is the distribution of the maximum clique size for experts and novices. A Kolmogorov-Smirnov test suggests that experts and novices are not distinguishable based on their maximum clique size distributions $p=0.0587$).

Figure 6

Diameter. This is the distribution of the diameter of the experts and novices. A Kolmogorov-Smirnov test suggests that experts and novices are not distinguishable based on their diameter distributions ($p=0.6432$).

Figure 7

Average path length. This is the distribution of the average path length for experts and novices. A Kolmogorov-Smirnov test suggests that experts and novices are not distinguishable based on their average path length distributions ($p=0.3906$).

Figure 8

Erdös-Renyi and Barabasi graphs. The two graphs on the left are Erdös-Renyi graphs created using optimized parameters that best fit the 3-cycle distributions of experts and novices. On the right, we see two Barabasi graphs.

Figure 9

3-cycle distributions. On the left is an ECDF of the 3-cycle distribution for sorters and the Erdös-Renyi model. On the right is the corresponding ECDF for the sorters and the Barabasi model. In both graphs the dashed line corresponds to the appropriate random model while the solid line corresponds to the sorter data.

Figure 10

3-cycle distributions. Here we see the 3-cycle distributions for the different CCMs. The model that fits the best is v3 [Eq. (6)], where the multiple categorization probability is ${\beta }^C$.

Figure 11

Representative CCM graphs. These are some representative graphs for the CCM using optimized input parameters. Qualitatively they match up much better to the sorter graphs seen in Fig. 2.

Figure 12

PCA of the sorter data. On the left we see the PCA plot of the sorters. PC1 is the coordinate along the first principal axis, and PC2 is the coordinate along the second principal axis. Sorters known by us to be experts are marked by circles while sorters known by us to be novices are marked by triangles. Each point is labeled on the left by the sorter number. The second principal component discriminates experts from novices. On the right is a plot of the cumulative relative importance of each subsequent principal component.