Alternative model for administration and analysis of research-based assessments

Research-based assessments represent a valuable tool for both instructors and researchers interested in improving undergraduate physics education. However, the historical model for disseminating and propagating conceptual and attitudinal assessments developed by the physics education research (PER) community has not resulted in widespread adoption of these assessments within the broader community of physics instructors. Within this historical model, assessment developers create high quality, validated assessments, make them available for a wide range of instructors to use, and provide minimal (if any) support to assist with administration or analysis of the results. Here, we present and discuss an alternative model for assessment dissemination, which is characterized by centralized data collection and analysis. This model provides a greater degree of support for both researchers and instructors in order to more explicitly support adoption of research-based assessments. Specifically, we describe our experiences developing a centralized, automated system for an attitudinal assessment we previously created to examine students’ epistemologies and expectations about experimental physics. This system provides a proof of concept that we use to discuss the advantages associated with centralized administration and data collection for research-based assessments in PER. We also discuss the challenges that we encountered while developing, maintaining, and automating this system. Ultimately, we argue that centralized administration and data collection for standardized assessments is a viable and potentially advantageous alternative to the default model characterized by decentralized administration and analysis. Moreover, with the help of online administration and automation, this model can support the long-term sustainability of centralized assessment systems.

Figure 1

(a) A schematic representation of the default model for the propagation of RBAs that has been used historically in PER. Here, light gray arrows denote rare or nonexistent connections. This model depends almost entirely on the instructor to drive the administration and analysis of their students’ results. There is also little to no mechanism for data from multiple courses to be centrally aggregated or compared. (b) A schematic representation of an alternative, centralized model for the administration and analysis of RBAs. This model depends almost entirely on a centralized administration system to facilitate administration and analyze students’ responses. This model also creates a centralized pool of comparison data that can be used for research and analysis purposes.