Reliability, compliance, and security in web-based course assessments

Pre- and postcourse assessment has become a very important tool for education research in physics and other areas. The web offers an attractive alternative to in-class paper administration, but concerns about web-based administration include reliability due to changes in medium, student compliance rates, and test security, both question leakage and utilization of web resources. An investigation was carried out in introductory astronomy courses comparing pre- and postcourse administration of assessments using the web and on paper. Overall no difference was seen in performance due to the medium. Compliance rates fluctuated greatly, and factors that seemed to produce higher rates are identified. Notably, email reminders increased compliance by 20%. Most of the 559 students complied with requests to not copy, print, or save questions nor use web resources; about 1% did copy some question text and around 2% frequently used other windows or applications while completing the assessment.

Figure 1

(Color online) Compliance rates in the pilot and full-scale study, broken down by individual classes and the average rate for each study. The four bars in each group are, left to right, the web-based precourse assessment, the paper-based precourse assessment, the web-based postcourse assessment, and the paper-based postcourse assessment. Shown are the compliance rates on each of the paper and web assessments, and the average for each of the studies. On the full-scale study, the small white bars on top of the web-based assessment represent the rates of students who attempted but did not complete the assessment.

Figure 2

(Color online) Frequencies of focus-lost events on both assessments compared to distribution for completely random events. Significant deviations from a random (binomial) distribution are probable indications of students using other applications while taking the assessment.