Effects of testing conditions on conceptual survey results

Pre-testing and post-testing is a commonly used method in Physics Education Research to assess student learning gains. It is well recognized in the community that timings and incentives in delivering conceptual tests can impact test results. However, it is difficult to control these variables across different studies. As a common practice, a pre-test is often administered either at or near the beginning of a course, while a post-test can be given either at or near the end of a course. Also, in conducting such tests there often is no norm as to whether incentives should be offered to students. Because these variations can significantly affect test results, it is important to study and document their impact. We analyzed five years of data that were collected at The Ohio State University from over 2100 students, who took both the pre-test and post-test of the Conceptual Survey of Electricity and Magnetism under various timings and incentives. We observed that the actual time frame for giving a test has a marked effect on the test results and that incentive granting also has a significant influence on test outcomes. These results suggest that one should carefully monitor and document the conditions under which tests are administered.

Figure 1

The pre-test averages, post-test averages, absolute gains, and normalized gains for the individual quarters in the comparison group. (The error bars denote standard errors.)

Figure 2

Pre-test total scores of the comparison group and the no-instruction group. (The percentages indicate average pre-test score percentages; the error bars denote standard errors.)

Figure 3

(Color) Pre-test item scores of the comparison group and the no-instruction group. The curves show the trend of item scores; the straight lines connect the average item scores in each range.

Figure 4

Pretotal and Post-total scores of the comparison group and the subsequent quarters. Note that the pre-test scores of the subsequent quarters are similar, but the post-test scores are rather different under different test timings and incentives. (The error bars denote standard errors.)

Figure 5

(a) Absolute gains of the comparison group and the subsequent quarters. Note that the pre-test scores for quarters from 2006 to 2007 are rather similar (see Fig. 4). (b) Normalized gains of the comparison group and the subsequent quarters. Note that the error bars denote standard errors.