Improving accuracy in measuring the impact of online instruction on students’ ability to transfer physics problem-solving skills

In two earlier studies, we developed a new method to measure students’ ability to transfer physics problem-solving skills to new contexts using a sequence of online learning modules, and implemented two interventions in the form of additional learning modules designed to improve transfer ability. The current paper introduces a new data analysis scheme that could improve the accuracy of the measurement by accounting for possible differences in students’ goal orientation and behavior, as well as revealing the possible mechanism by which one of the two interventions improves transfer ability. Based on a $2\times 2$ framework of self-regulated learning, students with a performance-avoidance oriented goal are more likely to guess on some of the assessment attempts in order to save time, resulting in an underestimation of the student populations’ transfer ability. The current analysis shows that about half of the students had frequent brief initial assessment attempts, and significantly lower correct rates on certain modules, which we think is likely to have originated at least in part from students adopting a performance-avoidance strategy. We then divided the remaining population, for which we can be certain that few students adopted a performance-avoidance strategy, based on whether they interacted with one of the intervention modules designed to develop basic problem-solving skills, or passed that module on their first attempt without interacting with the instructional material. By comparing to propensity score matched populations from a previous semester, we found that the improvement in subsequent transfer performance observed in a previous study mainly came from the latter population, suggesting that the intervention served as an effective reminder for students to activate existing skills, but fell short of developing those skills among those who have yet to master it.

Figure 1

An example of a zigzag plot, adapted from Ref. [10]. Each point represents the passing rate of students either before (“Pre”) or after (“Post”) being given access to the instructional material in each module. Passing rates in the Post stage of a module are cumulative with Pre stage attempts. See Sec. 2d for more details.

Figure 2

The sequence of OLMs designed for this experiment. Each OLM contains an assessment component and an instructional component. Students are required to make at least one attempt on the AC first, then are allowed to view the IC, and go on to make subsequent attempts on the AC. OLMs 1 and 4 were added for the 2018 implementation.

Figure 3

Comparison of performance on OLMs between students with different numbers of brief attempts: (a) Rotational kinematics and (b) angular momentum. The error bars represent standard error. Passing rates on the on-ramp module is not shown since it is irrelevant to the discussion of transfer.

Figure 4

Comparison of the performance on the pre and post attempts of module for students with 0–1 brief attempts. A subset of 2017 students were selected to match the background knowledge of 2018 students using a propensity score derived from exam scores. Passing rates on the on-ramp module is not shown. Data on example 2 from 2017 are absent because the module was added in 2018.

Figure 5

Comparison of the performance on the pre and post attempts of modules for students with 0–1 brief-attempts and different on-ramp performance (a) and (c): Pass On Ramp Pre. (b) and (d): Pass On Ramp Post. The student population in 2017 students was selected to match the background knowledge level of students in 2018 using a propensity score derived from exam scores. Passing rates from the on-ramp modules are not shown. Data on example 2 from 2017 is absent because the module was added in 2018.