Active Learning reduces academic risk of students with non-formal reasoning skills. Evidence from an introductory physics massive course in a Chilean public university

We present the findings of a pilot plan of active learning implemented in introductory physics in a Chilean public university. The model is research based as it considered a literature review for adequate selection and design of activities, consistent with the levels of students' reasoning skills. The level of scientific reasoning is positively correlated to student success. By contrast to a control group of students following traditional lectures, we observed a significant reduction in failure rate for students that do not yet posses formal scientific reasoning. This profile of student being the majority, we conclude that implementing active learning is particularly suited to first year of higher education in the context of a developing country. It fits the particularities of student profile and typical classroom size, leading to learning improvement and reduction of academic risk as well as being financially sound.


I. INTRODUCTION
Low student enrollment and high attrition rates in Science, Technology, Engineering, and Mathematics (STEM) education are part of the major contemporary challenges in higher education [1]. As a consequence, introductory physics courses usually becomes filter courses for numerous engineering students [2]. Indeed, this is reflected by our institution's historical approval rate in first semester introductory physics course taught for all engineering careers. In an effort to reduce academic risk, as well as to better prepare engineering students to 21st century, a methodological change in the teaching method [3] [4] was decided in our institution. Strong evidences accumulated over the past 35 years of significant gains in learning physics [5][6][7] using active learning motivated a pilot program of implementation of research based [8] active learning in introductory physics course.
The pilot program main challenges were making it realistically scalable for encompassing a large enrollment course (1600 students) and adapting innovative strategies to the context of our traditional and public university. Our institution has a voluntarily inclusive access policy that favors admissions to higher education of students with very heterogeneous profiles. Students originates from diverse ways of entry such as university selection test, and high school ranking, among others [9,10]. As deep understanding of physics concepts requires formal reasoning [11,12], it is essential to characterize the level of scientific reasoning of our introductory physics course students before tailoring a teaching sequence consistent with their profiles [13,14].
In this article we report on a professor training model implemented at piloting level to transform introductory physics courses for engineering programs. This model is inspired by previous similar experiences both in Latin America [15][16][17][18] and others countries [19][20][21]. By comparing the results of an experimental group following active learning and a control group, we found a 9.1% reduction in failure rate (see table I), statistically significant and coherent with previous reports [5]. In addition, students with transitional reasoning level, being the majority in our context, benefited most of the innovation. The level of scientific reasoning is well reported to be a determining factor in academic success in the first years of university science courses [12][13][14]. Methodologies that promote active learning, that is designing lectures where the student is intellectually active [22] have been found to improve scientific reasoning [23,24].
For designing classroom activities, we used active learning strategies focused on the need to enhance conceptual learning, problem solving skills, collaborative work and hypothesis generation among other skills required for the training of engineers in the 21st century. In particular, we used tutorials [25], interactive lecture demonstration [26], peer instruction [27], sense-making tasks [28] and collaborative Solving Problems [29] fitting the program of the course (introductory mechanics and statics). In supplementary material, we provide a list of the activities used for the active learning sessions.
The teaching material was designed by a coordination team, distinct from the group of teachers. Interactive lectures were given by a professor accompanied by a teaching assistant. The professors had already taught the same course in traditional way previously, and undergraduate students of our university served as teaching assistants were. Both professors and teaching assistants were present simultaneously during lecture to support the student's learning process. Implementing active learning strategies in the classroom requires a preparation work that we structured in a three-stage cycle ( fig.1 top). The first stage consists in the coordination team reviewing the literature to prepare the activities that would be used in the two class occurring two weeks after. In the second stage, the proposed activities are reviewed and adjusted, if necessary in a meeting between the coordination team, the professors and the teaching assistants. Finally, in stage three, teachers and assistants implement activities in the classroom. This process started two weeks before starting the semester and was repeated each week.
Classes are held in Scale-Up rooms [30], build for this project, of capacity 54 students. Their design promotes student-centered learning as the teacher is not the focal point. Five projectors provide good visibility of projected material for all students. The furniture consisted initially of normal tables and chairs, arranged in groups of 9 people to work in micro groups of 3 students, and latter large round desks that facilitate collaborative work were installed, as seen in figure 1 bottom. The entire wall of the room are covered with boards. In addition, blackboards of 60cm x 80cm are freely available in room for every group of students. Finally, the room layout allowed an easy circulation of students and professors between desks.
Following the flipped classroom model [31,32] material is provided to the students gradually: elements of theory and exercises to complement traditional classes, identical with those of the previous semester, were provided before the active session through a virtual institutional platform. Indeed this material is available to all the students, in active modality or not.
Such a methodological change may be associated to an increment in infrastructure and personal costs per student [33]. It is due to reducing sections sizes and thus increasing the number of sections in massive courses. Due to the layout and infrastructure of the university, it is not the case in our university: the average section of introductory physics is composed by 50 students (31 sections for approximately 1550 students in total). Scaling up the pilot plan would be realized at constant number of sections and professors.

A. Experimental design
The pilot program concerned 4 sections from a total of 31 were experimental (active learning) and 4 sections of equivalent historical results as control group. Both groups reasoning skills were characterized by taking the Lawson Classroom Test of Scientific Reasoning [34] at the start of the semester.
Both control and experimental groups followed the same weekly program of contents and had access to the same bibliography or online material. Indeed, online material and evaluations were not innovated and were similar to previous semesters. The experimental group followed two active lectures of 1h30 each and an active exercise session of 1h30. The control group had the same schedule but using traditional teaching. Three evaluations (identical for all students) were carried out with three problems each, prepared by a teaching committee not included in the implementation of the pilot plan.

B. Failure rate analysis
For consistency with the meta-analysis of Freeman [5], we compare failure rate of students defined as the number of student failing the course divided by the total number of students. We only consider students that attended all the evaluations during the semester forming a universe of 304 students, 146 of which followed traditional learning and 158 active learning. For determining the statistical significance of the difference in failure rate, we test the following hypothesis (H 0 ): "introductory physics students following active lecture are less likely to fail than students following traditional lecture". The null hypothesis is then that "introductory physics students following active lecture are as likely or more likely to fail than students following traditional lecture". Our statistical analysis is the following: failing or passing a course is a binary process. Thus, evaluating the failure rate is equivalent to estimate a so called "cut efficiency". Bayes analysis allows to theoretically calculate the uncertainty of efficiency measurement due to size effect [35]. If k is the number of positive cases and n the population, the probability distribution of the efficiency is a beta distribution of parameters (α = k + 1, β = n − k + 1). We can then calculate the density probability of the variation in failure rate (as shown in figure 3) and therefore estimate p(H 0 ).

C. Scientific reasoning diagnostic
During the two first weeks of the semester, students of the control and experimental groups took the Lawson Classroom Test of Scientific Reasoning [34]. The test is composed of 24 questions, organised by pairs. Bao [36] and Mashood [37] used the distribution of correct answer as an indicator of typical reasoning skills in first year of university, comparing different cultures. For our study, a total of 260 students took the reasoning test and were present in all evaluations of the semester (149 from active groups and 111 from control group).

A. Failure rate reduction
Failure rate was 45.2% for students following traditional learning, comparable to the historic rate (45.7%, considering only students that attended all evaluation). By contrast, active learning students failing rate was 36.1%, evidencing a 9.1% improvement (p(H 0 ) = 0.937) consistent with improvement reported in literature for physics and STEM [5]. We conclude that active learning methodology is particularly well suited in the Chilean context for teaching introductory physics contents.
Separating results by gender, one can note that while in both modalities, female failure rate was higher than male failure rate, female students benefited more of the innovation than male students (see table II): female students following traditional learning were 1.41 time more likely to fail than those following active learning (1.29 for male students). For comparison with previous cross cultural studies [36,37], in fig.2 top, we present the distribution of students as a function of the number of correct answer to the Lawson Classroom Test of Scientific Reasoning. The average reasoning level in our students universe was found significantly lower than those reported in USA, China and India (see table III) thus evidencing our local need to adapt to a profile of student with reasoning skills not yet fully developed. A pair analysis of answers allows to sorts the students in three reasoning level [34]: the so called "concrete", "transitional" and "formal" reasoning skills. At the concrete level (0-4 pairs) students are able to classify objects and understand conservation, but not yet able to form hypotheses. At the formal level (9-12 pairs), students can think abstractly and are able to control and isolate variables, among other similar tasks. At the transitional level (5-8 pairs), students are only capable of partial formal reasoning [14] In Fig.2 bottom we present the categorization of our introductory physics students. The majority of students are observed to be still in a transitional level of reasoning. In light of this diagnostic, obtained early in the semester (week 3), we orientated classroom activities to

FIG. 2. Results of the Lawson Classroom Test of Scientific
Reasoning for introductory physics students, experimental and control groups combined (N=260). Top: distribution of students as a function of the number of correct answers likely favor students with transitional or concrete reasoning skills. Indeed this raises the need to specifically adapt the first year of our engineering curriculum to progressively improve reasoning skills.  In table IV we present failure rates separated by reasoning skills for both the experimental and control groups. The reasoning skills level is correlated to the probability of passing the course: concrete students were approximately 1.7 less likely to pass the course than transitional students and 2.5 less likely than formal students. While we observed a null effect on concrete and formal students, active methodology strongly reduced failure rate of transitional students (-13.5%) as can be seen in table IV and figure 3. Results are statistically significant.

V. CONCLUSION
We implemented evidenced based active learning lectures in an introductory physics course for engineering, supported by a professor training model. Our study evidences specifically a lower academic risk for students that had not yet developed formal reasoning skills, which are the majority in our local context.
In light of the pilot plan, it is now planed to progressively scale up the methodology to all sections of introductory physics. Extrapolation of our findings indicate a potential increment of 150 students approving the course each year. In our local context, not exceptional in developing countries, implementing active learning does not increase the number of sections, making it financially sound: the initial investment of one classroom renovation being equivalent to 8 students fees, would be compensated in less than one year by the reduction of academic risk.