Thematic analysis of 18 years of physics education research conference proceedings using natural language processing

We have used an unsupervised machine learning method called latent Dirichlet allocation (LDA) to thematically analyze all papers published in the Physics Education Research Conference Proceedings between 2001 and 2018. By looking at co-occurrences of words across the data corpus, this technique has allowed us to identify ten distinct themes or “topics” that have seen varying levels of prevalence in physics education research (PER) over time and to rate the distribution of these topics within each paper. Our analysis suggests that although all identified topics have seen sustained interest over time, PER has also seen several waves of increased interest in certain topics, beginning with initial interest in qualitative, theory-building studies of student understanding, which gave way to a focus on problem solving in the late 2000s. Since 2010 the field has seen a shift toward more sociocultural views of teaching and learning with a particular focus on communities of practice, student identities, and institutional change. Based on these results, we suggest that unsupervised text analysis techniques like LDA may hold promise for providing quantitative, independent, and replicable analyses of educational research literature.

Figure 1

Example results from an LDA model trained on a dataset of roughly 9000 Huffington Post headlines from 2012–2018 [11].

Figure 2

Probabilistic matrix factorization interpretation of LDA.

Figure 3

Flowchart diagram of methods: (a) Data collection and preprocessing; (b) Repeated topic modeling across filtering parameters, number of topics, and $\alpha$ values; (c) Aggregation and clustering of topics; (d) Analysis of literature with final model.

Figure 4

Frequency of 20 most common words in the data corpus (29 026 words total). For each word, both the overall number of occurrences and the fraction of documents containing that word are shown.

Figure 5

Alphabetical list of most frequent words in data corpus, removed using frequency filter of 55%.

Figure 6

Graphs of elbow plot with different document-topic mixtures ($\alpha$ values).

Figure 7

Cumulative topic prevalence over time, with shaded areas based on a jackknife estimation of variance. A wider shaded area corresponds to a greater variance in the distribution of prevalence values, while a narrower shaded area indicates a more uniform distribution. The $Y$ axis corresponds to the effective number of papers each year on that topic.

Figure 8

Cumulative prevalence of topics over time, smoothed using a rolling three-year window.

Figure 9

Average prevalence of topics over time, smoothed using a rolling three-year window.