Visualizing depth of student conceptual understanding using subquestions and alluvial diagrams

We aim to graphically analyze the depth of conceptual understanding behind the Force Concept Inventory (FCI) responses of students, focusing on three questions (questions 1, 15, and 28). In our study, we created and implemented subquestions to clarify and quantify the students’ reasoning steps in reaching their responses to the original FCI questions. We used alluvial diagrams to visualize the responses of students on the subquestions and original FCI questions, representing the transition rates across the question options as flows. The flows represent common patterns of consistency and inconsistency in the students’ conceptual understandings across the original FCI question and the subquestions. By combining subquestions and alluvial diagrams as we did, we are able to efficiently analyze the depth of understanding for a given situation and to visualize the existence of a specific misconception. For example, we found that even among those who correctly answered FCI question 15 about Newton’s third law, many students incorrectly answered a corresponding subquestion asking about whether Newton’s third law holds at the end of the interaction between the two objects. This misconception turned out to be fairly robust, as these same students had a similar answering pattern for question 28. Alluvial diagrams can be used for a broad range of applications, including analysis of multiple-tier tests, testlets (e.g., FCI questions 8 through 11), tests for representational consistency (e.g., a representational variant of the FCI), and so on.

Figure 1

The figure of the FCI Q.15.

Figure 2

The figure of the FCI Q.28.

Figure 3

(Left) Alluvial diagram for the item unit of FCI Q.1. The selection rate for each option is shown as a bar plot, and the transition rate between responses is shown as the width of the flow. The percentages of selecting each option or flow, the values shown in the vertical axis, are calculated among the total valid responses ($N_v=359$). The path correctly answered in the leftmost question is colored in green, and the paths incorrectly answered in the leftmost question are in red, orange, blue, and purple in descending order of selection rate. (Right) Alluvial diagram for the item unit of FCI Q.1 with flows with ratios less than 3% hidden.

Figure 4

Alluvial diagrams for the item unit of FCI Q.1 for different universities. The percentages of selecting each option or flow, the values shown in the vertical axis, are calculated among the number of valid responses for each University (1) $N_v=46$, (2) $N_v=70$, (3) $N_v=33$, and (4) $N_v=43$, respectively. The color scheme is the same as in Fig. 3. The options in the bar chart are (automatically with R) ordered to make it easier to see the flow boundaries.

Figure 5

(Left) Alluvial diagram for the item unit of FCI Q.15. The percentages shown in the vertical axis are calculated among the total valid responses ($N_v=359$). The color scheme is the same as in Fig. 3. (Right) Alluvial diagram for the item unit of FCI Q.15 with flows with ratios less than 3% hidden.

Figure 6

(Left) Alluvial diagram for the item unit of FCI Q.28. The percentages shown in the vertical axis are calculated among the total valid responses ($N_v=359$). The color scheme is the same as in Fig. 3. (Right) Alluvial diagram for the item unit of FCI Q.28 with flows with ratios less than 3% hidden. The options in the bar chart are (automatically with R) ordered to make it easier to see the flow boundaries.

Figure 7

(Left) Alluvial diagram for the subquestions of Q.15 and Q.28. The percentages shown in the vertical axis are calculated among the total valid responses ($N_v=359$). The color scheme is the same as in Fig. 3. (Right) Alluvial diagram for the subquestions of Q.15 and Q.28 with flows with ratios less than 3% hidden.