Performance of ChatGPT on the test of understanding graphs in kinematics

The well-known artificial intelligence-based chatbot ChatGPT-4 has become able to process image data as input in October 2023. We investigated its performance on the test of understanding graphs in kinematics to inform the physics education community of the current potential of using ChatGPT in the education process, particularly on tasks that involve graphical interpretation. We found that ChatGPT, on average, performed similarly to students taking a high-school level physics course, but with important differences in the distribution of the correctness of its responses, as well as in terms of the displayed “reasoning” and “visual” abilities. While ChatGPT was very successful at proposing productive strategies for solving the tasks on the test and expressed correct reasoning in most of its responses, it had difficulties correctly “seeing” graphs. We suggest that, based on its performance, caution and a critical approach are needed if one intends to use it in the role of a tutor, a model of a student, or a tool for assisting vision-impaired persons in the context of kinematics graphs.

Figure 1

The histogram shows the distribution of 60 scores achieved by ChatGPT on the TUG-K. The vertical dashed lines mark the quartiles for ChatGPT’s distribution. Note that the distribution is much narrower than that reported in the publication by Zavala et al. [46], where Q1, Q2, and Q3 were 7, 12, and 17, respectively.

Figure 2

The histogram shows the distribution of the number of items against the percentage of correct responses on an item. We can observe the overrepresentation of ChatGPT at the extremes of the distribution, meaning that, when compared to the student sample, there are more items that ChatGPT consistently (always or almost always) got right or consistently got wrong.

Figure 3

The histogram shows the distribution of the total number of selected answer options against the relative frequency at which they were selected. We can see a clear overrepresentation of ChatGPT in the bracket of never, or almost never, selected options and an overrepresentation of ChatGPT in the brackets of options selected more than 70% of the time.

Figure 4

The histogram shows the percentage of correct answers on each item on the test, grouped by the test objectives (1-7) for ChatGPT and students (as reported in [46]). For a description of items and objectives and a more detailed comparison of students’ and ChatGPT’s performance, see Supplemental Material [47].

Figure 5

The image shows item 4 from the TUG-K, followed by a typical ChatGPT response, where it provides a good strategy and reasoning, but fails to answer the question correctly due to faulty visual interpretation of the graph.

Figure 6

The image shows the prompt created by transcribing item 4 into text, followed by a typical ChatGPT response. The response is longer, more complex, and more elaborate than what would typically be expected from a student, but it uses correct reasoning and reaches the correct conclusion. ChatGPT gave the correct answer for all the 60 repetitions of this prompt.

Figure 7

The figure shows item 17 from the TUG-K, followed by a typical response from ChatGPT. On this item, ChatGPT always correctly visually interpreted the graph but consistently presented incorrect reasoning.