Controlled study on the effectiveness of multimedia learning modules for teaching mechanics

We have investigated the impact of using multimedia learning modules (MLM) on the learning of students enrolled in introductory physics courses at California State Polytechnic University, Pomona. One hundred fifty-nine students were randomly registered in two sections of an introductory mechanics course, one of which featured the MLMs. Both sections had the same instructor, participated in class discussions on identical topics, and used the same problem-solving examples. The students in the multimedia group outperformed the students who did not experience the MLMs in a final course examination and across identical discussion questions.


I. INTRODUCTION
Research affirms that most students do not read their textbooks prior to class sessions [1][2][3][4]. These studies suggest that most introductory physics students do not spend the time required to read critically and to distill the important concepts from their textbook. In addition, recognizing the key ideas is a challenging task for novice learners of physics. Yet, without some basic physics content knowledge, meaningful engagement during a short class period is unlikely and opportunities for higher cognitive engagement (e.g., applying, analyzing, synthesizing, and evaluating [5]) are limited.
Given these challenges, the hypothesis is that by presenting the basic physics principles in a more appealing way prior to class meetings, and rewarding student activities, students would be more motivated to prepare for class activities. Previous research in the use of multimedia learning modules (MLMs) in its home institute [University of Illinois at Urbana Champaign (UIUC)] has shown positive effects on students' learning and attitudes towards introductory courses. For example, introducing MLMs has shown to better prepare students for learning during lecture and significantly improves student performances across all background and ability levels [6]. More interestingly, the addition of MLMs resulted in a decline in course difficulty level ranked by the students [7].
We have adopted multimedia learning modules developed in a large research institutional environment in a very different environment of a four-year institution. Previously, we have investigated use of MLMs in teaching hybridonline format courses in electricity and magnetism at California State Polytechnic University, Pomona (Cal Poly Pomona) [8]. In this study, we focus on the effect of adopting MLMs in traditional face-to-face courses. MLMs were used as online prelecture assignments for the treatment group (MLM group) to provide an initial introduction to the main ideas of the upcoming lecture prior to class meetings [9]. Subsequent class activities were built on this initial exposure using mini lectures, discussion questions via clickers, and problem-solving applications. Both the MLM and control groups received a textbook-reading assignment.
This paper first provides a brief description of the MLMs with empirical and theoretical arguments supporting the use of multimedia in education. Then, it discusses the study regarding the effectiveness of MLMs in enhancing the learning of students enrolled in introductory mechanics courses at Cal Poly Pomona.

II. MULTIMEDIA LEARNING MODULES
MLMs present essential content of one lecture using various media (e.g., text, narration, flash animations.) Each module is 12-15 min in length and includes two or three embedded formative assessment questions. Students must answer each question in order to access subsequent content. If students answer the questions correctly, the program gives the explanation to reinforce the concepts. For incorrect responses, the program provides feedback and a related question to guide students towards the correct answer. Once the related question is answered correctly, the original question is presented again.
Although the MLMs are not interactive, their design is aimed toward fostering student engagement in their own learning experience. For instance, the instructional philosophy underlying the inclusion of the formative assessment questions is guided by physics education research (PER) results and the topics are focused on known student difficulties with the material. In particular, a three-step instructional strategy is employed in the design of the formative questions [10]. First, the original question is used to elicit student difficulties by contriving a question in which students are likely to make an error that exposes a particular difficulty described in the research literature. Second, the immediate feedback to incorrect responses and the follow-up questions confront students with possible gaps in their knowledge. Finally, the related leading questions guide students towards acquiring the correct response and shaping their understanding to being more scientifically congruent by providing explanations that assist students to resolve the conflict.
In addition, MLMs employ three-dimensional graphical representations to illustrate scientific models, utilizing animation to illustrate the derivation sequences of mathematical models and problem-solving strategies. Furthermore, MLMs use informal conversational style audio to explain abstract concepts and to create personal effect with further direct reasoning. These visualizations and other artifacts provide a common point of reference for students' in-class collaborative discussions and anchor students' discourse, keeping it focused on content and on the scientific representation of the concepts [11].

III. MULTIMEDIA IN EDUCATION
Educational multimedia utilizes the findings of cognitive learning theory. Based on research in cognitive psychology, working memory includes independent auditory and visual channels [12] each with limited capacity (cognitive load theory) [13]. Meaningful learning occurs when a learner selects relevant information, organizes it into coherent representations, makes connections among corresponding representations in each channel [14], and builds mental representations from the words and pictures. A well-designed educational multimedia presentation incorporates the use of both auditory and visual channels in order to increase the memory input capacity and employs the multimedia design principles [15] to select and organize the relevant information to assist learners to build more coherent representations of the concepts.
Extensive research supports the assertion that multimedia instruction can foster learning and improve problemsolving skills; more specifically, the research indicates that multimedia-based instruction is more likely to trigger the cognitive processes associated with active learning than listening to a verbal explanation or reading the text [16]. For example, educational researchers found that if the verbal explanation or reading is not accompanied by visual representations, students are often unable to remember most of the key ideas or to apply the concepts to similar situations [17]. With multimedia tools, instructors can integrate the power of the visual and the verbal modes of presenting concepts to enhance student learning [18,19]. Furthermore, students learn more deeply when words are presented in conversational rather than formal style [16].
UIUC MLMs adhere to the principles of effective educational multimedia described above. For example, MLMs use synchronized animations and narrations to increase memory input capacity, present three-dimensional dynamic images to help learners build appropriate mental representations, and make explicit connections among key ideas to help learners organize concepts in more coherent fashion while avoiding unnecessary text and images to reduce student cognitive load. Furthermore, MLMs use conversational style verbal explanations that are shown to be more pedagogically effective.

IV. RESEARCH
We have been using MLMs in several introductory courses at Cal Poly Pomona. In the fall of 2008, we conducted a pilot study using only three electricity and magnetism modules leading to positive student feedback and a promising effect on student learning outcomes. In the spring of 2009, we studied the effectiveness of the full implementation of electricity and magnetism modules within the context of a hybrid-online course [8]. Finally, we implemented the mechanics modules during the winter of 2010 in a regular (face-to-face) calculus-based introductory courses. In this paper, we discuss the latter research study using the mechanics modules.
We used MLMs in one section of the first quarter course of calculus-based general physics (PHY 131). PHY 131 generally meets three times a week in lecture format (50 min each) along with a separate weekly three-hour laboratory session and without any recitation or problemsolving sessions. The course topics include the first 12 chapters of the textbook by Knight [20] and much of the homework is completed through an online homework system [21]. The students are mostly engineering and physics majors, yet often have great diversity in terms of their high school physics experience. Many students have very little physics background and for some, PHY 131 is their very first physics course. Table I shows the list of the MLM prelectures we assigned to students in this study, which included the topics of linear dynamics (six modules), conservation laws (seven modules), and rotational dynamics (seven modules) [22]. The modules in Table I worked very well with the content and pace of our quarter-length course syllabus at Cal Poly Pomona. However, due to the physics background of our student population, we often tend to spend more time on kinematics than only two lectures designed in MLMs.

A. Study design
Two PHY 131 sections served as our MLM (treatment) and control groups. The MLM group (N ¼ 108) viewed the prelectures and completed the preflights [online conceptual quizzes via just-in-time teaching (JiTT)] [23] prior to the class meeting. Students in the control group (N ¼ 51) completed the preflights only based upon their reading of the textbook and without the aid of the prelectures. Figure 1 summarizes the study design. Both groups were encouraged to follow the reading assignments on the schedule that accompanied the course syllabus and received points for completing online assignments (preflights and prelectures). All other aspects of the course were the same for both groups: the course instructor, discussion topics, homework assignments, clicker questions, and class examples. In this study, the dependent variables were student examination performance and their level of preparation for class discussions.

B. Measurements and results
To measure the impact of the MLM online prelectures, we compared student performance at three different stages: before attending each lecture via online preflights, during the lecture via use of common clicker questions in both the control and MLM groups, and third, on midterm and final examinations. In addition, we asked students about the usefulness of the different course components and their textbook-reading habits on a course survey. This paper will discuss the data from the course examinations, common clicker questions, and the course survey.

Course examinations
Students completed three closed-book exams: two onehour midterms in the fourth and eighth week of the quarter followed by a two-hour comprehensive final exam after the tenth week. The exams included both multiple-choice and open-ended problem-solving questions. The majority of the multiple-choice questions focused on concepts and reasoning rather than on calculations. The course instructor graded all the exams for both sections.
The lectures for both the MLM and control groups were held on the same day in a consecutive fashion; however, the classes met in two different buildings. Thus, the likelihood of student discourse with their peers from the other section was minimized. This allowed the instructor to use the same exam questions, some with modified numerical values, for both sections. Because of this schedule, the MLM students completed all their examinations prior to the control group. Table II shows the average group scores across the three course examinations. On all exams, the MLM group achieved higher scores when compared to the control group. On average, the MLM group scored more than 4% higher on the first and the second midterms and 6% higher on the final examination. A one-tail unpaired t-test indicates a significant difference in final examination performances for the two groups (p < 0:05).

Student preparation for class discussions
In addition to short lectures, pedagogical demonstrations, and problem-solving activities, students were also presented with 3-5 clicker questions. These formative assessment questions were used to initiate discussions, address common difficulties, measure conceptual understanding, and help students with identifying problemsolving steps (e.g., the construction of a free-body diagram in the application of Newton's second law). Clicker questions also focused on demonstration prediction, occasional factual recall, and simple symbolic calculation questions. To foster connections among the course components and to maximize the benefits of the just-in-time teaching strategy   FIG. 1 (color online). A schematic representation of the controlled study design. The students in the MLM group viewed an average of two MLMs per week, a total of 17 during the quarter.  to the extent possible, the lecture activities and clicker questions were informed by the feedback from student responses to the JiTT preflight questions. In terms of class discussions, the instructor first posed the questions and then asked students to discuss them with their neighbors. Next, students selected one of the responses and the instructor then displayed the class results to the students and asked their reasoning for each choice. Depending upon the percentage of correct responses, the instructor would determine the need for additional discussion. For example, if more than 70%-75% of the students responded correctly, the instructor typically did not continue with an extensive discussion of the topic. For questions with poor responses (e.g., less than 40% correct) the instructor allowed more time for discussion; for instance, the instructor either chose to repoll after reviewing the relevant concepts or asked students to consider additional ''follow-up and hint'' questions. The instructor often had prepared a series of 2-4 relevant questions in advance, and depending on the student responses would decide whether or not to present the follow-up and hint questions.
After each question was posed, an initial time limit was given to the students; if students requested more time (or if approximately less than 60% of the students had responded by the end of the time limit) the instructor would extend the time accordingly. The initial time limit was the same for identical questions in both classes. The instructor strongly encouraged discussion among students in each section and often a large fraction of the students would engage in discussion. However, due to the limited class time, the instructor did not regularly attempt a second polling as is recommended in Peer Instruction by Mazur [24].
When comparing the percentage of correct responses for the 39 common questions across both the control and MLM groups, the students in the MLM achieved an average of 69:2% AE 3:0% while the average score for the students in the control group was 61:7% AE 2:8%. Figure 2 shows that the MLM group achieved a higher percentage of correct responses on individual questions more often. We calculated a medium effect size of 0.4, which is the ''difference on a criterion measure between the two means divided by the Control group's standard deviation'' [25]. Moreover, the MLM group asked for time extensions for eight of the 39 questions (20%) compared to 14 questions (35%) for the control group.
The MLM group answered a total of 67 questions during the quarter. Of those, 39 were common for the two groups. Twelve of the remaining 28 questions were follow-up and hint types of questions while 16 were completely new questions. The control group answered a total of 61 questions; the noncommon questions (22) were largely composed of the follow-up and hint nature (see Table III). The lower percentage of correct responses on individual questions for the control group consequently resulted in a need for more discussions and use of follow-up and hint questions, which, in turn, limited the opportunities for presenting new applications questions.

Course survey
At the end of the quarter, students completed a course survey that was aimed towards gaining a better understanding of their experiences with MLMs [26] and their attitudes toward different course components. For example, we asked students to rate different course components (e.g., class lectures and activities, MLM prelectures, homework assignments, and the textbook reading) according to their usefulness in their learning physics on a fivepoint Likert-type scale. Students in both the MLM and control groups ranked the class lectures the highest in terms of usefulness; however, they had different ratings for the usefulness of discussion of clicker questions. In the MLM group, 75% of the students [27] indicated that the discussions were either ''very useful'' or ''useful'' (compared to 20% that were neutral and 5% that expressed unfavorable attitudes). In the control group, 60% of the students indicated a favorable attitude towards the usefulness of the clicker discussions (compared to 25%  that were neutral and 10% that expressed unfavorable attitudes). Prelectures received the second highest ranking in the MLM group. Many students indicated that MLMs are a useful component of the course: 70% of the MLM students ranked the prelectures as either very useful (55%) or useful (15%), while 20% expressed neutral attitudes and the last 10% reported unfavorable attitudes. Across all course components, textbook-reading assignments received the lowest ranking in terms of usefulness. Only 30% of the control group had a favorable attitude towards the usefulness of the textbook whereas 25% reported neutrality and 45% reported having unfavorable attitudes. The result of student rating of the textbook reading in the control group was comparable to the MLM group.

MLM completion
The online log data indicated that, on average, 78% of the MLM students completed 75% or more of the MLMs during the quarter. Conversely, student responses to the periodic question regarding the textbook-reading assignments placed at the end of the preflights indicated that only 20% of students reported reading ''all'' or ''most'' of the relevant topics prior to answering the preflight questions; see Fig. 3.
Finally, the instructor received slightly higher student evaluation scores from the MLM group than from the control group, indicating students' higher overall satisfaction with instruction and the course materials in the experimental course. It is important to note that smaller class size typically results in better evaluation scores for the instructors at Cal Poly Pomona. The MLM class was almost twice the size of the control class and this result is contrary to our Physics Department's historical records in which the same instructors have often received lower scores in larger size classes.

V. DISCUSSION
The purpose of this study was to evaluate the impact of MLMs on student learning in an introductory mechanics course. The results suggest that use of the MLM prelectures had a positive effect on student exam performance and on the level of student preparation for class discussion questions. In particular, students who viewed the MLM prelectures achieved higher final exam scores on common class discussion questions compared to the control group. In addition, students' initial exposure to the key ideas of each lecture topic through MLMs better prepared them for class activities, allowing the instructor to offer more examples focused on the application of concepts during discussion periods. By reducing the amount of time spent on restating the basic principles or on providing hints and follow-up clicker questions, more time was available for presenting additional clicker questions in the MLM class.
These additional examples during the class, along with earlier exposure to the MLMs, resulted in a higher average student performance on their final exam.
One can argue that students' reading of the textbook could have a similar effect; yet most students generally bypass text-reading regardless of the degree of instructor encouragement. Furthermore, proper monitoring and rewarding of student reading activities is not feasible. On the other hand, the MLM prelectures can be more easily enforced and rewarded via asynchronous student log data. Today's students demand everything to be immediate, fast, and effective, yet novice learners of physics require much more time to read and to construct the main ideas from a textbook. The reduced amount of time needed to view the prelectures (compared to reading) seems to motivate our students to complete the task more often. Most students ''want to learn, but they want to learn only what they have to learn, and they want to learn it in a style that is best for them'' [28]. The typical textbook content and format are not as attractive to today's students who often prefer watching movies and playing video games to reading.
Considering student traditional perception of lectures as the main content delivery venue, some could argue that moving content delivery to online lectures might affect student overall appreciation for the class time and the need for an instructor. However, in our study, the class attendance was very high in both sections (perhaps partly due to the small class participation points). More importantly, the class discussions and activities were rated higher in the MLM group, and the same instructor received better evaluations in the MLM class regardless of its larger size. This study is an example of successful implementation and dissemination of PER in teaching introductory physics courses, where course materials developed at a large research institution (UIUC) were adopted at a four-year teaching-focused institution (Cal Poly Pomona). The results of our implementations are in agreement with the findings of previous studies conducted by the PER group at UIUC [3,4,[6][7][8].
Evidently, some of the common systematic resistive forces of implementing PER informed curriculum were present [29] (e.g., department norms, expectations of content coverage, time structure, quarter versus semester curriculum, different student populations). While we made appropriate modifications when needed (mainly on the choice of topics and time schedule of the weekly prelecture), the simplicity of the logistics, well-designed format, and intuitive technology structure made the process seemingly effortless. More importantly, the endless support of the Physics Education Research Group at UIUC was the key element in successful implementation of the MLMs at Cal Poly Pomona. While we agree that ''major impediments to the spread of research-proven reforms are situational characteristics consistent with traditional instruction'' [30], we believe implementation efforts could be improved when instructors are not only provided with the information and tools to anticipate possible difficulties, but also receive timely feedback and support from the original developers of the material. In this case, the UIUC Physics Education Research Group knowledge and experience was a source of tremendous help and valuable insight during the implementation of the MLMs.
In summary, the study results indicate that the integration of supplementary materials in the form of the MLM prelectures offered an alternative to textbook reading that contributed to the learning of our students. Even if no difference existed between multimedia-driven learning and learning via print medium, the former appears to be more attractive to freshman students and more efficient in terms of both study time and in-class activities, thereby resulting in higher student achievement in the course.