Could an artificial-intelligence agent pass an introductory physics course?

Massive pretrained language models have garnered attention and controversy due to their ability to generate humanlike responses: Attention due to their frequent indistinguishability from human-generated phraseology and narratives and controversy due to the fact that their convincingly presented arguments and facts are frequently simply false. Just how humanlike are these responses when it comes to dialogues about physics, in particular about the standard content of introductory physics courses? This case study explores that question by having ChatGPT, the preeminent language model in 2023, work through representative assessment content of an actual calculus-based physics course and grading the responses in the same way human responses would be graded. As it turns out, ChatGPT would narrowly pass this course while exhibiting many of the preconceptions and errors of a beginning learner. A discussion of possible consequences for teaching, testing, and physics education research is provided as a possible starter for more detailed studies and curricular efforts in the future.

Figure 1

Sample ChatGPT dialogues about homework problems. The entries labeled with a red “KO” are by the author, the entries labeled in green by ChatGPT. For mathematical expressions, LaTeX was used.

Figure 2

Text-based transcription of a graphical problem. The left panel shows the online version of a final exam problem in LON-CAPA (the graph would be parametrically randomized), and the right panel the transcription for ChatGPT, as well as the ensuing dialogue.

Figure 3

Logical error in an attempt to solve the transcribed question 19 of the Force Concept Inventory.

Figure 4

Surface-feature modification of a Force Concept Inventory problem. The left panel shows the original problem and the right panel a modification.

Figure 5

A late-night dialogue between a “stubbornly guessing” ChatGPT and a frustrated author.

Figure 6

Clicker items from a particular lecture [18]. Three of the items were presented twice, i.e., before and after peer discussion.

Figure 7

Dialogue about questions X3 and X6 in Fig. 6. ChatGPT got X3 correct; Peer Instruction was simulated by asking a confusing question, and the second iteration X6 was still counted as solved since ChatGPT did not deviate from its original answer.

Figure 8

Dialogue for a programming exercise in the second semester [19]. Due to a typo in the prompt, the mass of the third charge was left out, and ChatGPT assumed a mass of “1”.

Figure 9

Jupyter Notebook [35] output of the VPython program code generated by ChatGPT in Fig. 8.

Figure 10

Problems constructed by ChatGPT.

Figure 11

ChatGPT grading its own modified response to one of the problems in Fig. 1; the final answer has been changed from 4.9 to 4 h.

Figure 12

ChatGPT grading statements by Einstein, O’Brien, and Data.