Development of a multiple-choice problem-solving categorization test for assessment of student knowledge structure

Problem-solving categorization tasks have been well studied and used as an effective tool for assessment of student knowledge structure. In this study, a traditional free-response categorization test has been modified into a multiple-choice format, and the effectiveness of this new assessment is evaluated. Through randomized testing with Chinese college students, the multiple-choice categorization test has been shown to provide equivalent measurement compared to the open-ended test. In addition, the influence of including diagrams in problems on students’ categorization performances has also been studied, which suggests that the inclusion of diagrams in problems can improve students’ performances in identifying the relevant concepts. The results also show that the knowledge structures of the Chinese college students are in the early transitional stage between novices and experts.

Figure 1

Question 7 in tests $B$ and $C$: (a) without a diagram in test $B$, and (b) with a diagram in test $C$.

Figure 2

The answer sheet section: (a) In test $A$, students are asked to write down up to 7 categories. The section below shows the answer section for one category, which is repeated 7 times on the answer sheet; (b) tests $B$ and $C$.

Figure 3

Score distribution for question 1 of test $B$ (a) before the conversion, and (b) after the conversion. The horizontal axis represents the range of scores for a question, and the vertical axis represents the fraction of students receiving that score.

Figure 4

Score distribution of question 14 in (a) version $A$, and (b) version $B$. The horizontal axis represents the range of scores for a question, and the vertical axis represents the fraction of students receiving that score.

Figure 5

Mean scores of tests $A$ and $B$. The error bars represent standard errors (SE). The scores are scaled from 0 to 1.

Figure 6

Correlation of the average scores of each question between tests $A$ and $B$. The horizontal axis is the average score of a question in test $B$, and the vertical axis is the average score of the corresponding question in test $A$. The scores are scaled from 0 to 1. The error of the linear fit is $\mathrm{RMSE}=0.043$, which is at a similar level of the standard error of the average score of a question ($\mathrm{SE}\approx 0.05$).

Figure 7

Average scores on tests $A$ and $B$ of the top and bottom 50% of the students according to their midterm exam scores. The error bars represent standard errors. The scores are scaled from 0 to 1.

Figure 8

Mean scores of the 10 questions in test $B$ (without diagrams) and in test $C$ (with diagrams). The error bars represent standard errors. The scores are scaled from 0 to 1.

Figure 9

Mean scores of the 5 identical questions in tests $B$ and $C$. The error bars represent standard errors. The scores are scaled from 0 to 1.