How gender and reformed introductory physics impacts student success in advanced physics courses and continuation in the physics major

[This paper is part of the Focused Collection on Gender in Physics.] Active-learning approaches to teaching introductory physics have been found to improve student learning and affective gains on shortterm outcomes [S. Freeman et al., Proc. Natl. Acad. Sci. U.S.A. 111, 8410 (2014)]; however, whether or not the benefits of active learning impact women to the same degree as men has been a point of concern [A. Madsen, S. B. McKagan, and E. C. Sayre, Phys. Rev. ST Phys. Educ. Res. 9, 020121 (2013)]. Further, the long-term impacts of active-learning experiences are also understudied. At Florida International University, a Hispanic-majority institution, we have implemented Modeling Instruction (MI) and the Integrated Science Learning Environment (ISLE) in introductory physics classes for the past decade. In this empirical paper, we report on a longitudinal investigation of student performance and persistence in upper level physics courses after having previously experienced MI or ISLE in their introductory physics courses, and disaggregate students by gender. Using survival analysis methods, we find women who declare physics as a major are more likely than men to graduate with a physics degree. Women are also just as likely as men to pass through the upper division courses, with the highest failure risk for both men and women occurring in the first semester of upper-division course taking. These results reinforce the need to expand considerations of performance outcomes to be longitudinal to measure the effectiveness of the entire physics experience.


I. INTRODUCTION
For decades, science, technology, engineering, and mathematics (STEM) disciplines 1 have seen an increase in the proportion of Bachelor's degrees earned by women, from around 17% in 1965 to about 36% in 2014 [1].However, this masks the trend in the past 15 years, over which time the rate of obtaining Bachelor's degrees for women has been stagnant and even decreasing for some disciplines, like physics.Additionally, the trends for women as a block do not highlight the trends in participation by minority women in STEM.Minority women in physics have faced greater struggles with participation and retention, as well as other traditionally underrepresented groups [2].To wit women account for only 21% of Bachelor's degrees in physics, a proportion that has decreased in the last 10 years [1] and, particularly, minority women on average earn only about 5% of the physics degrees awarded to women [2].
It is worthwhile to note that women now appear to persist in physics course taking at roughly comparable numbers to men throughout elementary, middle, and high school and, in fact, perform slightly better than men in high school physics based on their grades [3].There is a drop off in proportionate enrollment of women between high school and college, at least in terms of the students who declare physics as a major at the beginning of post-secondary studies.
The problem of persistently low diversity in postsecondary physics has given rise to many initiatives and efforts at local and national scales to help recruit and retain women in physics.For example, the American Physical Society (APS) has run several annual, regional conferences on Undergraduate Women in Physics (CUWiP) in order to support the professional development and persistence of women in undergraduate studies.Other national societies such as the National Society of Black Physicists (NSBP), National Society of Hispanic Physicists (NSHP), and the Society for the Advancement of Chicanos & Native Americans in Science (SACNAS) have specific outreach efforts for women of color in physics in their conferences.
It has now been clearly established that active learning environments lead to significant improvements in the learning gains of students and, similarly, the likelihood of passing a STEM course [4][5][6][7][8].Freeman et al. reported increased learning and course performance in active learning STEM classrooms versus traditional lecture courses in their meta-analysis of 225 studies.
Much attention has been paid in reformed classroom environments to performance gaps between men and women and how these gaps are affected by the nontraditional instruction in physics [9][10][11][12].In some cases women have been seen to perform lower than men both before and after instruction, with the performance gap increasing afterwards [12,13].Although Lorenzo et al. [9] claimed to have reduced the gender performance gap with their interactive engagement reform efforts, their results have not been clearly reproducible in other institutional or classroom contexts [10,11], and have since been reanalyzed to show no effect or a negative effect of gender on performance [11,14].
Educators should also be cautious of focusing solely on such performance gaps, as they do not tell a complete story of what has transpired in a classroom, let alone the impacts on individual students [14,15].The normal practice of analyzing performance gaps during or shortly after a semester-or year-long intervention carry some limitations.One particular limitation of this practice is that the longerterm impacts of such interventions on other performances and persistence (e.g., subsequent course taking) are relatively unknown.The current work seeks to address this gap in research by investigating the longer-term impacts of an active-learning introductory physics experience, along with any persistent gender effects.This is relevant to both researchers and educators who wish to understand how pedagogical reforms impact individuals in an active learning classroom.The latter may face the common criticism that active-learning environments address only certain short-term outcomes (e.g., performance on a concept inventory) and fail to "cover" other elements necessary in the long term, thus disadvantaging these students in advanced course taking.Longitudinal analysis of success can address these critiques.
The common practice to compare men and women's prepost performance scores and the results that have been found this way, as well as broader societal stereotypes, has led to a widespread belief that women's potential in STEM fields is somehow inferior to men's when, in fact, this may be an unfounded belief.With these perceptions in mind, Hyde and Linn [16] performed a meta-analysis to compare psychological traits and cognitive abilities between genders.They identified that 78% of the 46 analyzed reports found a negligible or small effect of gender on a spectrum of psychological traits such as mathematical, verbal, and spatial abilities; aggression; leadership effectiveness; self esteem; and computer use.The distribution of effect sizes for both genders was overlapping by 85.3% and demonstrated that gender may not be such a major discriminator on math and science ability as is commonly believed.Therefore, it is more appropriate to think of men and women as predominantly similar in their math and science abilities.The few exceptions in the meta-analysis were on activity levels and physical aggression, in which males were more aggressive and physically active than females.
Women and men are cognitively similar and yet the myth and broader stereotypes perpetuates the belief that women are more likely to fail a course or not persist in a STEM program based on performance differences on standardized tests.Many times, small differences in performance between genders are so widely publicized that there is a significant risk of reinforcing subtle, persistent biases [16] and anxiety levels in women, thus activating stereotype threats [17].One result of widely held societal beliefs and biases is that males are more likely to show interest in certain STEM fields, including physics, which may result in a greater proportion of men entering such college majors [8].We are concerned that biases impact decisions about admission, hiring, promotion, and persistence.

II. PHYSICS EXPERIENCE AT FIU
At Florida International University (FIU), a large public urban Hispanic-serving institution (HSI), the physics education research group established Modeling Instruction (MI) as an optional introductory physics course sequence, starting in 2004.The Investigative Science Learning Environment (ISLE), complementary to MI, has been offered since 2010.Both MI and ISLE are interactive studio-based pedagogies that feature student-centered activities, integrated "lecture" and "lab" courses, and little to no time spent presenting material through lecture.Students typically interact with each other in groups of three with the help of one or two learning assistants and the instructor in classes of 30 to 35 students.Students that elect to be in a Modeling or ISLE classroom contact the faculty and/or department, are placed in a lottery system, and selected at random to fill each available section.Impact on students that have taken Modeling Instruction and ISLE courses include a 6.73 times greater odds of succeeding in their introductory physics courses, when compared to regular lecture courses [12].FIU Modeling Instruction courses have also reported positive attitudinal shifts towards physics by students in these courses [2].Students that take these reformed introductory physics courses also outperform students taught in a lecture format on a physics conceptual inventory by a 14% difference in scores [12,13].Even though the reformed courses are better at retaining more students in physics and achieving various learning goals, it has been found that performance gaps between male and female students in MI persist [13].
After the completion of introductory physics courses, our goal in the present work is to explore the trajectory of our majors all the way to graduation as a longitudinal measure of their persistence in physics.Persistence in physics starts with physics majors and other interested students (which is taken as an elective by many BS students but mandatory for BA students).Note that for our data set and analyses, both BA and BS students are reported together because the longitudinal analysis of their persistence did not differ based on their different sequence of courses.

III. RESEARCH DESIGN A. Research questions
This paper has three main research questions: (1) Are there any gender differences in declared major intentions and ultimate graduating degrees for students that attempted any upper division physics course in the years 2005 to 2014? (2) How does gender and/or reformed introductory course experience of declared physics majors impact the time that they experience the highest risk of failing an upper level physics course for the first time?(3) How does gender and/or reformed introductory course experience impact the likelihood of physics majors successfully graduating with a physics degree (BS or BA)?

B. Data sample
This study uses institutional data from Florida International University's student database that includes student demographics, financial, and academic information.We examined data for students registered in upper level physics courses over the years 2005 to 2014.Academic data included enrollments, grades, and semester for the courses, as well as intended and declared majors, graduation data, and degree earned.We also identified every student's introductory physics course experience as either a reformed (Modeling Instruction or ISLE course) or a traditional introductory course sequence.
The sample used in this paper is limited to students who enrolled in at least one upper level physics course beginning with Modern Physics I in 2005, as 2004 was the first year FIU offered Modeling Instruction.We selected Modern Physics I as the first upper division physics course students enrolled in.A comparison of our data sample demographics with national and general FIU data demographics is shown in Table I.FIU demographics are very diverse when compared to national ethnic backgrounds in physics.As a Hispanic Serving Institution, our physics student ethnic majority is mostly Hispanic and different from the typical national white non-Hispanic ethic majority in physics.It is important to note these differences when interpreting our results and its generalization [18].
The data sample has a unique student list of N ¼ 272 students over the ten year period, including 20% women and 80% men with 21% of students having a reformed introductory experience.We group all the women from different ethnic backgrounds together for the purpose of statistical analysis as our numbers are insufficient for analysis through an intersection of both gender and ethnicity.
Out of those 272 students that have enrolled in at least one upper division physics course, 130 have graduated, 100 men and 30 women.The first research question takes into account these graduating students.The second and third research questions pertain to declared physics majors only.Therefore, the data sample was cut to include any declared physics majors taking upper division courses or graduating with a physics degree.The sample of physics majors was found to be N ¼ 177; 24% of majors had taken reformed introductory physics courses and 76% had traditional introductory physics courses with a similar ethnic breakdown as the overall study sample.

C. Analytical methods
To study the longitudinal effects of reform and gender on student performance, we primarily use survival analysis.Survival analysis is a collection of statistical procedures to study the time it takes for an "event" to occur [19].Survival analysis is often used in the health sciences to measure the time it takes for patients to relapse under a medical treatment.In the social sciences the method of recording when events happen over time is called event history data [20].Both survival analysis and event history analysis describe the same method of collecting data of when an event happened and possible predictor variables that could help explain how or why an event happened.For the remainder of the paper, we will refer to these methods as survival analysis.
In higher education, survival analysis is effective in analyzing longitudinal data of student persistence [21][22][23][24].For this paper, the "events" of interest are student persistence in their upper division physics courses and student persistence in the major up to graduation (e.g., the furthest point in the trajectory towards graduating in physics).We coded two variables for persistence.The first dichotomous variable is assigned to students to identify if they have persisted (0) in an upper division physics course by having received a letter grade of C or above or failed (1) the course.Course failure is defined as having received a course grade of C-or lower, and includes drops and withdrawals.Our failure definition is based on programmatic criteria for continuing on to subsequent courses in the major trajectory.
A second dichotomous variable is assigned to students to indicate if they are continuing in their physics program and have graduated (0) or have failed to graduate with a physics degree (1).Failing to graduate with a physics degree is defined as graduating with a different degree, and/or not having attempted any physics course for three consecutive semesters.For many students in the data sample the event of graduating is "censored," indicating the event has not occurred yet.Censored data occur when a subject does not experience the event of interest within the study period [19].Though our data are retrieved from institutional data and not individual responses, data of student graduation are censored due to the fact that about 50% of our data sample has not had time to graduate yet.For example, a student who began taking Modern Physics I in 2013 could not have graduated by 2014, which is the end of our study period.If a student's graduation has not yet occurred then it is considered censored and labeled as (0), continuing to persist in the major.For both of these event variables of persistence, the independent variable is time, as measured by the number of semesters since first attempting Modern Physics I (up until graduation or failure to graduate).The first time index on the x axis of the survival graphs is the first semester of attempted upper level physics course.
The statistical model underpinning the survival analysis is guided by the survivor function SðtÞ ¼ PðT > tÞ, defined as the probability that the survival time T is greater than a specific value t.Our goals in survival analysis is to estimate the survivor function SðtÞ from the survivor data, compare survivor functions of different groups (through inferential statistical comparison), and assess the relationship of possible explanatory variables such as gender to survival time.Survival time in this study is the time it takes for the persistence event (as defined previously) to occur counted by the number of semesters of advanced course taking.
We use the Kaplan-Meier method, a nonparametric estimation using the product limit, to estimate the survivor function [19].The advantage of this method is that it is based on the calculation of a risk set every segment of time at which at least one event of interest occurred.For example, the risk set in the first semester of upper level physics includes everyone that enrolled in upper division courses, and if one failed to survive or persist then an event of failure is recorded and the leftover students are analyzed for the next risk set in the second semester.This way, the information contained in the set of events is optimally used [20].Each time an event happens the survival curve changes similar to a step function.To compare two groups and their respective survival curves, we use the log-rank test to check for statistical equivalence.The log-rank test is a chi-squared test, which provides an overall comparison of two Kaplan-Meier curves.The test makes use of observed versus expected event counts over the event times in the entire data set [19].

A. Physics pathways
Our first research question pertains to major intentions and the pathways to a physics degree for graduated students that attempted an upper division physics course between 2004 and 2014.The pathways and degrees are graphed separately for men and women in a Sankey diagram, shown in Figs.1(a) and 1(b).These Sankey diagrams show the flow of individuals from one major intention through their graduating degree, where the width of the arrows are proportional to the number of individuals in each flow.We selected those students that have graduated and set aside those that are still on the trajectory towards their degree.In the years from 2004 to 2014 we had a total of 130 students graduate with some degree, 100 are men and 30 are women.
The list of intended majors for 130 graduates included over 100 different majors that were grouped into six categories.Engineering intended majors included students with mechanical, industrial, civil, computer engineering, and biomedical engineering majors.Other STEM majors included chemistry, biology, mathematics, environmental science majors, and all other that were not physics were grouped in the non-STEM major intentions.Physics intentions were Physics BA, Physics BS, and General Physics intentions.We analyze all physics intention collectively as there was no difference in our data set and analysis of persistence between BA and BS students.For men, 50 students (50%) graduated with either a BS or BA physics degree and 29 of them had initial physics intentions.Therefore, 58% of men that graduated with a physics degree persisted in the major, while 18% were specifically recruited from non-STEM majors such as art, philosophy, and business.
For women, we have a total of 23 (76%) physics degree graduates and 17 of them had initial physics intentions.A total of 74% of the women that graduated with a physics degree persisted within the major and 8% were recruited from non-STEM majors.From the total proportions of men and women that graduated with a physics degree, 50% men and 76% women, the difference in these proportions is statistically significant, indicating that more women persist to graduation than men.
The Sankey diagram of major pathways shows the diversity of students that end up with a physics degree and gives insight into whom physics educators might recruit from our introductory classes.As a qualitative picture of persistence, the Sankey diagrams also illustrate the difference between genders persisting within the physics major at FIU.When compared to men, there is a greater percent of women who have committed to the major that stay in the major.

B. High risk semester
During the first semester in the upper level physics trajectory, it is suggested that students take Modern Physics I Lab and Thermodynamics, and a required math course.As required courses, Modern Physics I and Modern Physics I Lab have the highest enrollments out of all the upper level physics courses; yet, Thermodynamics is the course with the highest risk of failure, with a 41% failure rate over the years between 2005 and 2014.Modern Physics I and Modern Physics I Lab have 32% and 34% average class failure rates, respectively, over the 9 academic years in the study.Figure 2 shows the probability of students to pass their upper level physics courses for each semester enrolled.Figure 2(a) shows the survival curves comparing the probability of surviving (or passing) each semester of upper division physics course taking for students that had a reformed introductory physics course, including Modeling Instruction and ISLE labeled "modeling," and those that were in traditional introductory lecture class under "nonmodeling."This survival comparison shows that both modeling students and Nonmodeling students are equally likely to pass their first semester of upper level physics with about a 60% probability.This places the first semester of upper level physics as the highest risk semester for students since, for the rest of the semesters enrolled in upper level physics, the differential survival rate is much less than the initial 60% in the first semester.The comparison between the modeling and Nonmodeling introductory physics groups shows no significant difference, as assessed by a log-rank test, with a chi square of 1.4 and p value of 0.243.The statistical equivalence of the two groups would suggest that reformed introductory students are just as successful in upper division courses and have the same skills and content knowledge to succeed as traditional lecture students.
Figure 2(b) shows the survival curves for female and male students.The figure indicates a 72% probability of surviving their course taking in the first semester for women and only a 55% chance for male students.The statistical difference between the two survivor curves for each gender is not statistically significant as indicated by the log-rank test, with a chi square of 0.5 and p value of 0.489.The log-rank test considers the entire set of data over the entire period of analysis, and not just the difference in the first semester.For both men and women, the highest risk semester is the first semester of upper level physics.
Both survivor curves share the common trends of having the highest risk occurring in the first semester.The relatively high probability of failing a course in the first semester cannot be attributed to any one course by this survival analysis alone.Yet, Thermodynamics and Modern Physics I and the associated lab have among the highest raw failure rate of all upper division courses and many students take these courses in their first semester of upper level physics.Over the period of the study, several different instructors rotated in teaching these three courses and we find it reasonable to exclude simple instructor effects as a major factor in creating this high risk semester.Low survival in the first semester may also be attributed to the combined pressure of success for students now taking two to three physics courses instead of just one physics course a semester, each of which involves more advanced physics ideas and mathematical complexity.

C. Graduating with a physics degree
The third research question investigates how the introductory physics experience or gender impacts the likelihood of graduating with a physics degree.Figure 3 shows the survival probability curves to graduate with a physics degree.Figure 3(a) shows how students that had a modeling introductory physics experience compare to students that had a traditional introductory physics experience in their likelihood of graduating with a physics degree.The log-rank test of significance shows these two survival curves to not be statistically different, with a chi squared of 3.3 and p ¼ 0.07.For modeling students, their survival probability to graduation is 85% by their fourth semester in upper level physics courses and suffers no further risk.For the traditional students, FIG. 2. Majors' survival by semester.These survival curves show the probability of surviving the upper level physics courses with each semester.The highest risk of survival is the biggest difference of one semester to the last.In (a) both modeling students and nonmodeling students have the same statistical risk to their survival, where the biggest risk is found in the first semester of upper level physics.(b) Shows that men and women have the overall same statistical risk to surviving their physics courses, where the biggest risk to surviving their courses happens in the first semester of upper level physics.
their survival probability to graduate with a physics degree is 70% by their fifth semester taking upper level physics courses and suffers no further risk.
For the comparison of men's and women's probabilities to graduate we refer to Fig. 3(b).As mentioned earlier, there were 177 declared physics majors in the years of 2005 to 2014, where 20% were women.Figure 3(b) shows that for women the likelihood of survival to graduate is 89% by their second semester without further risk, and men's likelihood of graduating with a physics degree drops to 70% by the end of their fifth semester in upper level physics course.The difference between the men's and women's survivor curves is statistically different with a chi squared of 4.2 and p ¼ 0.04.This is statistically significant at the 5% level, an estimate that may be limited by the size of our female student population of 20.Six semesters of upper level physics correspond to a four year graduation rate, presuming students took their introductory physics courses in their first year of college and immediately preceding their first upper level physics course of Modern Physics the next fall.
The data in the survival curves for graduation use both students that have graduated and also data from students that are continuing to persist towards graduation and have not finished.Survival analysis methodology uses the information from both groups to generate the appropriate risk assessment for their survival probabilities in the given semester.High persistence or survival probabilities are generated by fewer failure events of switching out of the major or dropping out of school occurring during the study period.
For men and women particularly, in the long-term outcome of graduating with a physics degree, women outperform men by almost 20% (significant difference with p value of 0.04) by the end of their fifth semester in upper division courses.This result challenges the preconceived social bias that women are more likely to not persist in a STEM career because of the performance gaps in diagnostics assessing semester-long interventions.Further analysis of longitudinal measures like graduation in physics and different measures of success other than semester long changes in learning, may lead to equitable research designs and interpretations that do not include social bias against women [14].

V. DISCUSSION AND CONCLUSION
Our analysis of survival probabilities of passing upper division physics courses shows that students who experience our reformed introductory physics classrooms are just as likely to succeed in upper division physics courses as students with traditional lecture introductory classrooms.We also found that reformed students are also just as likely to graduate with a physics degree as nonreformed students.Our analysis of longitudinal outcomes of success addresses the common criticisms of active learning environments in physics addressing only certain short-term outcomes appearing on concept inventories and not "covering" all the introductory material, thereby disadvantaging students in advanced course taking.
At Florida International University, students that had a reformed introductory physics experience in Modeling Instruction or ISLE have 6.73 higher odds of succeeding in their introductory physics course as compared to lecture students [12].Combined with the same likelihood of passing upper division physics, reformed students have a net benefit in their education.The high retention in reformed introductory physics also feeds into a diverse pool of recruits for the physics major.From our analysis of major intention pathways, men and women differ in their persistence in the physics major such that a smaller percentage of women choose to leave physics than men after completing their introductory physics experience and declaring their intention to major in physics.Of those that we do recruit, 18% of the men that graduate with a physics degree switch from non-STEM majors and 8% of the women do similarly.This result sheds light on who we should consider as being relevant to advise into the physics majors, which is to say many students are potential physics graduates, not just those who declare an intended physics majors in their freshman year.The success story of women in physics at FIU is even more compelling.We find that women are persist within the major at greater rates than men, survive their high risk semesters of upper division physics course taking just as well as men, and have a higher likelihood of graduating with a physics degree than men.The persistent bias that women face in the expectations about their supposedly inferior performance or likelihood to persist compared to men may be perpetuated by an overattention on short-term performance gaps and unfounded societal stereotypes.Women of color suffer compounded stereotypes of both their gender and their races or ethnicities.The demographic breakdown of our student population in this study speaks to the success of Hispanic women in particular, since they are the predominant group in our subsample of women (as well as for the men in our data).
While there still may be persistent gender performance gaps in introductory physics courses in some classrooms or contexts, women's performance in upper division physics and graduation is, in our programs, better than that of men.This result reinforces the need to expand our consideration of performance outcomes to be longitudinal and to measure the effectiveness of the entire physics experience.
One limitation of the current work is that we cannot address the causal mechanisms by which women outperform men at FIU.There are many possible factors that can have an effect on positive outcomes for women including college preparation, study habits, community support, peer and other mentoring, and the impacts of other nuanced experiences.Another reason FIU women persist could be from a sense of grit acquired throughout the physics experience despite the gendered expectation about their ability and gender roles in society and within the physics culture.Our data are limited by small numbers and cannot simultaneously disaggregate by gender and race or ethnicity.Yet, our results beg the question of how the interactions between gender and racial or ethnic identities impact women's experiences within our physics classrooms.Further research should address the experiences of Hispanic students in their physics classroom as it relates to them being a majority in our classrooms but a minority in the broader physics community.Comparatively, it would be interesting to investigate the experience of students from nationally well-represented groups in physics as it relates to their experience in FIU physics classrooms in which they are a minority.
The study results are also limited by the source of our data collection coming from retrospective institutional data.Identifying students' "real" major intentions at every moment in their studies is difficult since we do not have complete knowledge of students' week-by-week intentions (e.g., a student may become interested in pursuing a physics degree for a period of months or even years without declaring it to their institution, or having taken upper division courses).The study is also limited by small sample size of physics majors, though given the size of undergraduate physics nationally, there are only a relative handful of programs which produce even this number of physics Bachelor's degrees.
(e.g., other science or engineering students) enrollment in upper level physics courses.Modern Physics I is a gateway course to all other upper level physics as it is a corequisite or prerequisite to most of the other upper-level physics courses.The BS program has required upper level physics course curriculum that includes Modern Physics I & II, Thermodynamics, Mechanics I & II, Electromagnetism I & II, Quantum Mechanics I & II, and a few physics electives.The BA program is similar with Modern I & II, Thermodynamics, then Widely Applied Physics I & II

FIG. 1 .
FIG. 1.(a) Sankey diagram depicting major intentions and pathways of men that attempted an upper level physics course and graduated on the right.50% of men that graduated received a physics degree.(b) Sankey diagram depicting major intentions and pathways of women that attempted an upper level physics course and graduated on the right.76% of women that graduated received a physics degree.

FIG. 3 .
FIG.3.Survival towards graduation for majors.These survival curves indicate the probability of persisting in the major and graduating with a physics degree.In (a) modeling students and traditional students have the same statistical likelihood of persisting towards graduation.(b) Women are statistically more likely to persist towards graduation than men by their 5th semester in upper level physics.

TABLE I .
Study demographics and national and FIU comparison demographics.Other in National Physics demographics refers to AIP's definition of ethnicities not represented in the previous four and non-US citizens.Other in FIU demographics refers to students that decided to pick multiracial, or their ethnicity is not directly labeled as one of the previous four.