Classification of open-ended responses to a research-based assessment using natural language processing

Surveys have long been used in physics education research to understand student reasoning and inform course improvements. However, to make analysis of large sets of responses practical, most surveys use a closed-response format with a small set of potential responses. Open-ended formats, such as written free response, can provide deeper insights into student thinking, but take much longer to analyze, especially with a large number of responses. Here, we explore natural language processing as a computational solution to this problem. We create a machine learning model that can take student responses from the Physics Measurement Questionnaire as input, and output a categorization of student reasoning based on different reasoning paradigms. Our model yields classifications with the same level of agreement as that between two humans categorizing the data, but can be done by a computer, and thus can be scaled for large datasets. In this work, we describe the algorithms and methodologies used to create, train, and test our natural language processing system. We also present the results of the analysis and discuss the utility of these approaches for analyzing open-response data in education research.

Figure 1

The RD probe of the PMQ. Reproduced from Ref. [23].

Figure 2

Distribution of student responses classified as set, point, and undefined for each probe for the entire human-coded dataset. The UR probe responses are almost all setlike and therefore this probe has been omitted from the NLP study.

Figure 3

A diagram of the process of analyzing student responses to the PMQ. The raw responses are preprocessed and then vectorized using NLP. These feature vectors are the input to the machine learning algorithm. The algorithm outputs probabilities that are used to predict in which category a response belongs.

Figure 4

An example ROC curve. This curve represents a model that is performing well, as it is to the upper left-hand corner. The plot suggests that there is a threshold of probability to classify something as the positive category that will result in a high true positive rate, and a corresponding low false positive rate. For comparison, the dashed line represents the expected ROC curve for a random classifier.

Figure 5

Average AUC for 100 random trials and 1 true trial. For each probe, this graphic shows, in blue, the AUC (averaged over all three categories) produced when we randomize the input labels to the logistic regression model. The red data point is the average AUC when the correct labels are used on the training data, showing significant improvement and true pattern recognition.

Figure 6

ROC curves for the logistic regression classifier. After running the logistic regression algorithm on the training data for each probe, we plot our true positive rate and false positive rate from predicting classifications on the testing data. Using one-vs-all classification, each curve represents the classification of a student response as being of the labeled category, or not of the labeled category (e.g., a response being setlike or not setlike).

Figure 7

AUC values and 95% confidence intervals (error bars) comparing the AUC of the NCU scheme to the one-vs-all (OvA) classification scheme. These results were determined from 500 different splits of the data into training and testing sets, and indicate whether the NCU scheme performs differently than the one-vs-all scheme.

Figure 8

Comparison of interrater reliability of the machine learning system and a human rater. Cohen’s kappa value ranges are shown comparing the machine learning system to the human categorization. Diamonds represent the kappa values between the original human categorization and a different human categorization. Generally, the machine learning system achieved the same level of agreement with the original human rater as the other human.

Figure 9

The high-magnitude-coefficient features and their unpenalized beta coefficients for each probe. The vertical axis lists different features, or words, that are important in categorizing a student response. The horizontal axis shows their relative weight, or beta coefficient. The words mc_a, mc_b, and mc_c correspond to the selection of multiple choice answer A, B, and C, respectively.