Integrating artificial intelligence-based methods into qualitative research in physics education research: A case for computational grounded theory

[This paper is part of the Focused Collection on Qualitative Methods in PER: A Critical Examination.] Qualitative research methods have provided key insights in physics education research (PER) by drawing on non-numerical data such as text or video data. While different methods towards qualitative research exist, they share two essential steps: recognizing patterns in the data and interpreting these patterns. Although these methods have led to the development of rigorous theory, there are challenges: As such methods require a series of judgments by the analyst, they are difficult to validate and reproduce. Further, they are hard to scale so that they are unavailable to the analysis of large-scale data. In this way, important phenomena may remain inaccessible to qualitative analysis. Reacting to these challenges and leveraging the potential of emerging methods of artificial intelligence (AI) such as machine learning and natural language processing, sociologist Nelson has proposed the concept of computational grounded theory (CGT). CGT proceeds in a process of three consecutive steps: In the first step, one leverages the power of computational techniques, especially natural language processing and unsupervised machine learning techniques, for pattern detection in large datasets—those of a size and scope that may prohibit human-driven analysis from the outset. In the second step, one relies on the integrative and interpretative capabilities of human researchers to add quality and depth to the quantity and breadth of the first step. In the last step, one again uses computational techniques to test the extent to which the detected and refined patterns from the first and second step hold throughout the whole dataset under investigation. Interestingly, CGT does not aim at simply automating parts of the qualitative process by using AI, but rather aims at integrating AI into the human analyst’s workflow within a qualitative analysis. This leads to an analytical system that can do something that is quantitatively and qualitatively different from what a human or machine can do alone. In this way, CGT aims at addressing questions about validity, reproducibility, and scalability in qualitative research while preserving the theoretical sensitivity and unique inferencing capabilities of the human analyst. In this paper, we provide a primer on CGT, present how it can be used to investigate the physics problem-solving approaches of $N=417$ students based on textual data, and discuss CGT’s potentials and challenges in PER. In consequence, this paper can provide critical input to the discussion of how emerging AI technologies can provide new avenues in qualitative PER.

This article appears in the following collection:

Figure 1

Schematic representation (adapted from Kubsch et al. [13]) of the three steps of CGT based on Nelson [12].

Figure 2

Schematic representation of how natural language processing and machine learning serve as complementary tools for pattern detection in a sentence corpus.

Figure 3

Illustration of the vertical loop including the track of the sliding mass.

Figure 4

Visualization of the data preprocessing process which includes elements of NLP.

Figure 5

Frequency distribution showing the number of sentences per textual description.

Figure 6

Visualization of the process for obtaining an appropriate clustering solution using the dimension reduction procedure UMAP and the clustering procedure HDBSCAN.

Figure 7

Frequency distribution of the number of clusters within the 1000 different clustering solutions.

Figure 8

Representation of detected clusters within a two-dimensional embedding space. Overall, eleven clusters labeled from 0 to 10 and an additional noise cluster labeled as $-1$ were detected. Black bordered points represent cluster centroids.

Figure 9

Condensed tree plot of the final clustering solution.

Figure 10

Final five global clusters representing specific themes within the problem-solving process and how these global clusters relate to the initial detected clusters as indicated by the dashed black lines.

Figure 11

Relative frequencies of the five themes (and uncategorized sentences) within textual descriptions of Physics Olympiad participants and nonparticipants.