Using module analysis for multiple choice responses: A new method applied to Force Concept Inventory data

We describe Module Analysis for Multiple Choice Responses (MAMCR), a new methodology for carrying out network analysis on responses to multiple choice assessments. This method is used to identify modules of non-normative responses which can then be interpreted as an alternative to factor analysis. MAMCR allows us to identify conceptual modules that are present in student responses that are more specific than the broad categorization of questions that is possible with factor analysis and to incorporate non-normative responses. Thus, this method may prove to have greater utility in helping to modify instruction. In MAMCR the responses to a multiple choice assessment are first treated as a bipartite, student X response, network which is then projected into a response X response network. We then use data reduction and community detection techniques to identify modules of non-normative responses. To illustrate the utility of the method we have analyzed one cohort of postinstruction Force Concept Inventory (FCI) responses. From this analysis, we find nine modules which we then interpret. The first three modules include the following: Impetus Force, More Force Yields More Results, and Force as Competition or Undistinguished Velocity and Acceleration. This method has a variety of potential uses particularly to help classroom instructors in using multiple choice assessments as diagnostic instruments beyond the Force Concept Inventory.

Figure 1

The raw data are initially stored in a $143\times 150$ matrix because our data set included 143 students and 150 response items.

Figure 2

Cut out of bipartite network showing students (purple) and response items (green). The normative response items form a central cluster in the middle. Two non-normative response items, 15c and 26d, are on the outskirts of this cluster. The eastern parts of the network seems to show more complex patterns.

Figure 3

Two example schematics of how a bipartite network (top row) is projected into a response network (bottom row).

Figure 4

Normative responses (red) have been chosen much more frequently by students than non-normative (white and blue). In a network, they will become hubs or attractors that yield little information about student thinking. Some responses (blue) have not been chosen at all, and thus do not show up in the response network. Our analysis focus on non-normative responses that have been chosen at least once (white).

Figure 5

The response item backbone network also contains 96 nodes but only 401 weighted connections. Node and font size represent item frequency, node color number of connections (red means more). Very infrequent nodes (small font sizes) can be viewed by zooming in on the image online.

Figure 6

Result of InfoMap partitioning of the backbone response network. (Left) InfoMap finds 9 different modules, each of which are connected to each other. (Right) Each module also has internal structure, here shown for the module labeled “X17a.”

Figure 7

Three example icons.

Figure 8

Module co-occurrence matrix comparing module pairs found by 1000 iterations of InfoMap with the modules we use here. The color indicates the fraction of times two items have been grouped together, ranging from 0 (blue) to 1 (red). Note that modules 3–5 might be interpreted as one big module or as 3 separate modules.

Figure 9

Illustration of the impetus module using pictograms. Each pictrogram represents a response item.

Figure 10

Illustration of the more force yields more results module using pictograms. Each pictogram represents a response item.

Figure 11

Illustration of the third module, which has two main interpretations; force as competition or lack of distinction between velocity and acceleration using pictograms. Each pictogram represents a response item.

Figure 12

Module 4.

Figure 13

Module 5.

Figure 14

Module 6.

Figure 15

Module 7.

Figure 16

Module 8.

Figure 17

Module 9.