Evaluating IBM’s Watson natural language processing artificial intelligence as a short-answer categorization tool for physics education research

Recent advances in publicly available natural language processors (NLP) may enhance the efficiency of analyzing student short-answer responses in physics education research (PER). We train a state-of-the-art NLP, IBM’s Watson, and test its agreement with human coders using two different studies that gathered text responses in which students explain their reasoning on physics-related questions. The first study analyzes 479 student responses to a lab data analysis question and categorizes them by main idea. The second study analyzes 732 student answers to identify the presence or absence of each of the two conceptual themes. When training Watson with approximately one-third to half of the samples, we find that samples labeled with high confidence scores have similar accuracy to human agreement; yet for lower confidence scores, humans outperform the NLP’s labeling accuracy. In addition to studying Watson’s overall accuracy, we use this analysis to better understand factors that impact how Watson categorizes. Using the data from the categorization study, we find that Watson’s algorithm does not appear to be impacted by the disproportionate representation of categories in the training set, and we examine mislabeled statements to identify vocabulary and phrasing that may increase the rate of false positives. Based on this work, we find that, with careful consideration of the research study design and an awareness of the NLP’s limitations, Watson may present a useful tool for large-scale PER studies or classroom analysis tools.

Figure 1

Students were asked to provide their reasoning as to why they agreed or disagreed with the conclusions made in this hypothetical experiment.

Figure 2

Students provided their reasoning for their selection from this preflight question using physical concepts.

Figure 3

Watson’s accuracy versus confidence for data from the lab reasoning study compared to researchers’ agreement. Each point represents 50 samples. A total of 479 samples were analyzed.

Figure 4

Watson’s accuracy versus confidence when looking for the existence of vertical displacement in the preflight data. Each point represents 50 samples. A total of 732 samples were analyzed.

Figure 5

Watson’s accuracy versus confidence when looking for the existence of vertical velocity in the preflight data. Each point represents 50 samples. A total of 732 samples were analyzed.