Automated analysis of short responses in an interactive synthetic tutoring system for introductory physics

Computer-automated assessment of students’ text responses to short-answer questions represents an important enabling technology for online learning environments. We have investigated the use of machine learning to train computer models capable of automatically classifying short-answer responses and assessed the results. Our investigations are part of a project to develop and test an interactive learning environment designed to help students learn introductory physics concepts. The system is designed around an interactive video tutoring interface. We have analyzed 9 with about 150 responses or less. We observe for 4 of the 9 automated assessment with interrater agreement of 70% or better with the human rater. This level of agreement may represent a baseline for practical utility in instruction and indicates that the method warrants further investigation for use in this type of application. Our results also suggest strategies that may be useful for writing activities and questions that are more appropriate for automated assessment. These strategies include building activities that have relatively few conceptually distinct ways of perceiving the physical behavior of relatively few physical objects. Further success in this direction may allow us to promote interactivity and better provide feedback in online learning systems. These capabilities could enable our system to function more like a real tutor.

Figure 1

The Pathway Active Learning Environment interface: The lesson activities and relevant video clip can be seen on the left. The SI tutor can be seen in the center of the interface. At the right a sample video supplementing the tutor’s explanation is shown.

Figure 2

Flowchart indicating the general analysis scheme followed in this paper.