How to evaluate students’ decisions in a data comparison problem: Correct decision for the wrong reasons?

Data comparison problems are used in teaching and science education research that focuses on students’ ability to compare datasets and their conceptual understanding of measurement uncertainties. However, the evaluation of students’ decisions in these problems can pose a problem: e.g., students making a correct decision for the wrong reasons. Three previous studies, that share the same context and data comparison problem but where participants had increasing conceptual knowledge of measurement uncertainties, are revisited. The comparison shows a troublesome result: increasing conceptual knowledge does not lead to better decision making in the data comparison problem. In this research, we have looked into this apparent discrepancy by comparing and reanalyzing the data from these three studies. We have analyzed students’ justifications by coding them based on the compared quantity and the deciding criterion, giving a highly detailed insight into what they do when comparing the datasets. The results show clear differences in the quality of the justifications across the studies and by combining the results with the decisions, we could successfully identify four cases of correct and incorrect decisions for right or wrong reasons. This analysis showed a high prevalence of correct decisions for wrong reasons in two of the studies, resolving the discrepancy in the initial comparison of these studies. The implication of our analysis is that simply asking students to make a decision in data comparison problems is not a suitable probe to gauge their ability to compare datasets or their conceptual understanding of measurement uncertainties and a probe like this should always be complemented by an analysis of the justification.

Figure 1

The setup depicting the falling and the horizontally launched balls in the context of the experiment as it was shown to students. Figure taken from Kok et al. [22].

Figure 2

The changes in students’ decisions before and after having seen the data for the three groups. No significant difference in switching categories can be found.

Figure 3

Distribution of the justification codes for the comparison code (left) and the criterion code (right) for participants’ justifications in the data comparison problem after having seen the data.

Figure 4

The combination of comparison and criterion codes for the justifications after having seen the data for group ${\mathrm{K}}_{\mathrm{C}}$ (top), ${\mathrm{W}}_{4\mathrm{SM}}$ (middle), and ${\mathrm{C}}_{\mathrm{C}}$ (bottom). The size of the bubbles indicates the percentage of occurrences of the code combinations and the color the association with the point (orange) and set (green) paradigms. Set paradigm associations occur only for combinations of set or interval comparison with overlap criterion.

Figure 5

The code combinations of the justifications for the three groups ${\mathrm{K}}_{\mathrm{C}}$ (top row), ${\mathrm{W}}_{4\mathrm{SM}}$ (middle row), and ${\mathrm{C}}_{\mathrm{C}}$ (bottom row), but now split for correct decisions (left) and incorrect decisions (right). The size of the bubbles indicates the percentage of occurrences of the code combinations and the color the association with the point (orange) and set (green) paradigms. Set paradigm associations occur only for combinations of set or interval comparison with overlap criterion.