Math remediation intervention for student success in the algebra-based introductory physics course

Pretesting and early intervention measures to identify and remediate at-risk students were implemented in algebra-based introductory physics to help improve student success rates. Pretesting via a math and problem-solving diagnostic exam administered at the beginning of the course was employed to identify at-risk students based on their scores. At-risk students were encouraged to utilize an online math tutorial to increase their chances of passing the course. The tutorial covers the same math topics covered by the diagnostic exam. Results from 643 students enrolled in the course showed that the 61 at-risk students who successfully completed the math tutorial increased their odds of passing the course by roughly 4 times those of the at-risk students who did not. This intervention is easily implemented, short term, and can be administered concurrently with the course. Based on these results, the Department of Physics has implemented the math tutorials in all sections of the introductory algebra as well as the calculus-based physics courses.


I. INTRODUCTION
Much research has been done on the influences on student success in introductory physics courses.Among these influences, students' mathematics skills have been shown to be one of the most important (see, for example, Refs.[1][2][3][4][5][6][7][8][9][10][11][12]).Many studies emphasize the importance of math skills for introductory physics [1][2][3][4][5][6].Several studies have shown that preparation in high school is correlated with success in introductory college physics, where more rigorous preparation in math and physics predicts greater course success [1,2,7,8].Correlations between SAT math scores and physics course grades have also been demonstrated [1,2,9].Additionally, several studies report correlations between diagnostic math tests given at the beginning of a physics course and success in the course or learning gains [10][11][12].All of these studies emphasize the effect of math preparation on students' final grade in introductory physics courses.Based on these studies, we devised a strategy to identify students with a math deficiency and implement an intervention to improve their math skills in order to increase their odds of success in introductory physics courses.
This study addresses students' math knowledge and the effects of a short term, concurrent math intervention on their overall performance in the first-semester algebrabased introductory physics course at a large urban university in the south central United States.A math diagnostic exam was employed to identify students "at risk" of failing the course, i.e., students with math deficiencies.At-risk students, as well as all other students in the course, were afforded the opportunity to improve their math skills within the first 10 days of the course through an online math tutorial.In this paper, results of the math intervention are presented comparing student outcomes across six sections of the course, where the math tutorials were offered in three of the sections.

II. BACKGROUND AND CONTEXT
The algebra-based Introductory General Physics I course at the studied university is a core course taken by approximately 1400 students in an academic year; it is required by the degree plans of twelve majors in the departments of biology, technology, architecture, and health and human performance.The typical lecture class size is approximately 200, where the students are diverse in terms of race or ethnicity, family income, year in college, transfer status, and first generation in college status, all of which are factors that correlate with student success [1,2,13].At the time of this intervention, the student success rate in this course at the studied university was about 65%, where success is defined as completing the course with a grade of D or higher.This was low in comparison to the success rate for other introductory level science courses at the same university, i.e., an 80% success rate for the Introductory Chemistry I course.Faculty in the Department of Physics have implemented many researchbased, interactive pedagogical approaches in an attempt to address student success, including Peer Instruction [14], Just in Time Teaching (JiTT) Warm-Up assignments [15], and Interactive Lecture Demonstrations (ILDs) [16].Combinations of these approaches were used in the sections in this study, as described in Sec.III.While students and faculty gave positive feedback about the interactive approaches, student success rates did not improve.
Since the interactive teaching strategies used all address students' conceptual learning, it was decided that additional factors should be considered in order to achieve improvements in the student success rate for the course.Based on the previous studies showing correlations between students' math scores and success rates in physics courses [1][2][3][4][5][6][7][8][9][10][11][12], this study focuses on students' mathematical knowledge and preparation and their effects on student success, i.e., to what extent did a math tutorial intervention increase the probability of passing the course for at-risk students.
The prerequisite for the Introductory General Physics I course is passing college precalculus with a grade of at least a D-.However, not all students who meet this requirement possess the necessary math and problem-solving skills for the course.Possible reasons for this include the student having (i) taken a precalculus course with insufficient rigor, (ii) taken the course several years ago, and/or (iii) completed the course with a low grade.
In 2007, the Physics Department implemented a math and problem-solving skills diagnostic exam to assess students' math knowledge and to identify at-risk students with insufficient math and problem-solving skills for the course.Students were given recommendations on their preparedness for the course based on their diagnostic scores.For students identified as at risk, remediation (self or external to the Department of Physics) was recommended.In the current study, a more direct approach was taken.At-risk students were advised to seek remediation through an online math tutorial offered by the Department of Physics to improve their math skills and their chances of successful completion of the course.

A. Participants
Participants in the study were 643 students enrolled in one of six algebra-based introductory physics sections at a large urban university in the south central United States, which is designated as a Hispanic-and Asian-serving institution.Only students who both took the diagnostic exam and earned a grade in the course were evaluated.819 students were initially enrolled in the six sections; 176 either withdrew from the course without earning a grade, did not take the diagnostic exam, or both.Students scoring below 65% on the diagnostic exam were designated as at risk based on previous analysis of the exam, as discussed in the next section.Demographic information of the participants is shown in Table I.Information about student SAT scores, which are related to educational background, is presented in Table II.
All students in three sections of the course (taught over two semesters by two instructors) were offered the math tutorial.These sections will be referred to as A, B, and C. Student outcomes from three other sections of the course, in which the math tutorial was not offered, were also included in the study and will be referred to as D, E, and F. Sections were taught independently and students enrolled in only one section.Courses A, B, and F were taught by the same instructor using JiTT and ILDs.Courses C, D, and E were taught by a second instructor using JiTT and Peer Instruction with clickers.

B. Diagnostic exam
The diagnostic exam, developed and implemented in 2007, is designed based on the mastery of prerequisite unit criteria discussed by Larkin and Brackett [17].It consists of twenty algorithm formatted multiple-choice questions covering topics in college precalculus (or earlier) math courses, such as exponents, scientific notation, basic and symbolic algebra, order of operation, geometry, trigonometry, and logic word problems.Sample questions from the diagnostic exam are included in Appendix A. In the process of developing the diagnostic exam, exam question effectiveness was determined to ensure the exam's assessment longevity using discrimination and facility parameters [18,19].Discrimination compares the number of correct answers for each question in the upper and lower 27% of the total responses.For example, if a question has a greater number of correct responses from the lower 27% of testers versus the upper 27% of testers, the question is considered ineffective.The upper and lower 27% rule is commonly used in item analysis [18].Discrimination was effective for all of the exam questions.Facility is the percentage of students obtaining the correct answer on a multiple-choice question.A facility of <30% is a difficult question, 30%-75% is a satisfactory question, and >75% is an easy question.Questions that fell into the difficult or easy category were replaced.In addition, the overall diagnostic exam was evaluated for efficiency using % test efficiency based on %Test efficiency ¼ No: of satisfactory questions Total No: of questions : This evaluation ensured the quality of the exam as an indicator of mathematics skills.Question effectiveness and test efficiency ratings were important for determining the skills students did or did not possess and providing a remediation guide.The diagnostic exam scored a 70% test efficiency rating which indicated that it was very effective for the criteria intended.Further details of the exam evaluation are given elsewhere [20,21].
The diagnostic exam is required of all students enrolled in the course, approximately 1400 per academic year.A student's score on the diagnostic exam counts as 3% of their final course grade.Recommendations for the students are made based on their scores for the diagnostic exam, and are included on the course syllabus.Students scoring above 70% should have the necessary preparation to pass the course, students scoring between 51% and 70% should review math skills or topics identified as a weakness, and students scoring 50% and below should consider dropping the course and/or seek immediate remediation to improve their math and problem-solving skills.These recommendations were instituted based on correlations between diagnostic exam score and final course grade for all sections of the course from Spring 2007 to Fall 2008 (∼2800 students), shown in Table III [20].Of the students scoring 50%-70% on the diagnostic exam, 56% were successful in the course, and of students scoring below 50% on the diagnostic exam, 28% were successful.While Voight [20] considered students scoring below 50% on the exam as at risk of being unsuccessful in the course or in need of remediation, in the current study the definition of at-risk students was expanded to those scoring below 65% (i.e., 60% and below) on the diagnostic exam in order to encourage more students who might benefit to take the math tutorial (only at-risk students were given a grade incentive to complete the math tutorial, as explained in the next section).

C. Math tutorial
At-risk students, as well as all other students, in three sections of algebra-based introductory physics were afforded the opportunity to complete a math tutorial during the first two weeks of class to improve their math skills.The tutorials were implemented through MyMathTest [22], which is an online program designed to test students' math skills and provide targeted reviews for their areas of weakness.The tutorial consisted of three subtests and a final test over the same math topics covered by the diagnostic exam.The test questions were selected by us from those available in MyMathTest.MyMathTest questions are written by "qualified higher education math instructors" [23].The questions are free response; students type their answers into text boxes on the computer.Samples of the MyMathTest questions are shown in Appendix B. Upon completion of each subtest, the program identifies the student's areas of weakness and generates a custom study plan with additional problems from those topics.A score of 80% or above must be achieved on the first subtest in order to advance to the next subtest, etc., and, subsequently, to the final.The goal was to have students use the math tutorial as a remediation tool to improve their math skills early on, to better prepare them for the course.While all students in sections A, B, and C could take the math tutorial, at-risk students (those scoring below 65% on the diagnostic exam) were given the incentive that completing all tutorial tests with a score of at least 80% would result in their diagnostic exam grade being increased to 65%, the score on which counts towards 3% of their final course grade.

D. Statistical analysis
In order to assess the extent to which the math tutorial intervention increased the probability of passing the course for at-risk students, a comparison of performance was made between the students in sections in which the math tutorial was offered (sections A, B, and C) to those in sections in which the math tutorial was not offered (sections D, E, and F).Importantly, when students enrolled in the course, they were not aware that some sections offered the tutorial, so it is expected that the course composition of students would be similar between sections A, B, C and sections D, E, F.
As preliminary analyses, chi square statistical tests were performed (i) to determine which student groups might be at risk for failure based on their diagnostic scores and (ii) to assess the association of the demographic data with the likelihood of completing the math intervention.The chi square test tests the null hypothesis that two categorizations are independent of each other; i.e., that PðA and BÞ ¼ PðAÞ × PðBÞ, where PðA and BÞ is the probability of events A and B both occurring, PðAÞ is the probability of event A, and PðBÞ is the probability of event B. Lower chi square values indicate a better fit between the data and values expected under the assumption of independence.The p value is the probability of obtaining chi square results at least as extreme as the results in the data, under the assumption that the null hypothesis is true.A p value less than 0.05 is considered evidence to reject the null hypothesis of independence; that is, the categories are related.When demographic characteristics are related to the likelihood of participating in the intervention, these characteristics should be controlled for in subsequent analyses [24].
To investigate the effectiveness of the math tutorial intervention, generalized estimating equations (GEE) logistic regression models were used.GEE models are extensions of the generalized linear model, a class of models that include both linear and logistic regression.In cases in which a response variable, such as a final grade in a course, is normally distributed, a linear regression model can be fit to predict course performance, as in In logistic regression, the logit, or natural log of the odds ratio (OR), is used to predict the probability of success, as given by logitðPÞ ¼ log P In Eqs. ( 2) and (3), P is the probability of success of a dichotomous outcome for an observation with characteristics x 1 through x k , which may be continuous or categorical predictors.Y i is the response of the ith individual, and β i is the coefficient that relates the predictor x i to the response.β 0 is referred to as the intercept.Effects can be reported in terms of the odds ratio, OR, which indicates the amount that a predictor, i.e., the math tutorial intervention, differentially increases the odds that a student will pass the course compared to a student who did not complete the intervention.Odds ratios greater than 1 indicate that a predictor increases the probability of success, and odds ratios less than 1 indicate that a predictor decreases the probability of success.Note that ORs are different from, but related to, probabilities.If the probability of an event is P A for group A, and the probability is P B for group B, the OR is The basic linear and logistic regression models are based on the assumption that observations are independent.GEE models are an extension of the generalized linear models that are used when the response variable (outcome) is correlated or clustered, for example, in the case of students in different sections of a course.GEE models maintain the same link function and linear predictor setup as the generalized linear models, but also specify a covariance to account for correlated responses [26].To investigate whether the math tutorial intervention was associated with increased probability of passing the course, comparing six course sections taught by two instructors, a GEE logistic regression model with student success (i.e., pass or fail) as the primary dependent variable was employed.A course grade above 50% was considered passing for this study.While the primary effect of interest in this study was whether students completed the math intervention, other covariates included diagnostic exam score, course section, and gender (as gender was associated with completing the math intervention, as discussed below).

IV. RESULTS
Overall, 643 students across the six sections completed the diagnostic exam and earned a final grade for the course (i.e., did not withdraw).231 students (36%) were considered at risk (scored less than 65% on diagnostic exam).125 of these were enrolled in sections A, B, and C where the math tutorial was offered; 61 (49%) of the 125 completed the math tutorial.Scores on the diagnostic exam ranged from 5% to 100% (mean M ¼ 67.54%,Standard deviation SD ¼ 17.99%).Final course grades for all 643 students ranged from 2.97% to 101.95% across the six sections (M ¼ 62.58%, SD ¼ 17.97%), where some sections offered extra credit points leading to final grade percentages greater than 100%.The mean values for the diagnostic exam scores, final grades, and SAT scores for the 231 atrisk students who completed the course are presented in Table II.
Several analyses were performed on the data to demonstrate the effects of the math tutorial on course performance.In the preliminary analysis, the relationship between demographic characteristics and participation in the intervention was explored.Chi square statistical tests of the demographic information shown in Table I indicated that women were no more likely to have at-risk scores on the math diagnostic test than men, χ 2 ð1; N ¼ 643Þ ¼ 3.42, p ¼ 0.064; however, they were more likely to complete the math intervention when eligible, χ 2 ð1; N ¼ 231Þ ¼ 10.0, p ¼ 0.002.In contrast, race or ethnicity was associated with being at risk, χ 2 ð4; N ¼ 643Þ ¼ 17.0, p ¼ 0.002, with African American students being at the greatest risk.However, race or ethnicity was not associated with the likelihood of completing the math intervention when eligible, χ 2 ð4; N ¼ 231Þ ¼ 2.52, p > 0.10.Based on these results, gender was included as a covariate in the logistic regression model of student success rate, but ethnicity was not.
The primary set of analyses using GEE logistic regression investigated whether math tutorial completion was related to the odds of receiving a final grade greater than 50%, hence passing the course.Results of the analysis are shown in Table IV.Overall, 83.6% of the at-risk students who completed the math tutorial received a final grade greater than 50%, in contrast to only 59.4% of those who did not complete the intervention.After entering the covariates, for the at-risk students, the OR for completing the math tutorial was 4.24 (p < 0.001), indicating that the odds of scoring over 50% in the course were over 4 times higher for students who completed the math intervention versus those who did not.In addition, for two course sections (A and B), a second intervention awarding students points for visits to teaching assistant tutors (TA visits) was implemented; therefore, this was entered as a covariate variable as well.Gender was not significantly related to course outcome (p ¼ 0.088), but course section, diagnostic exam score (OR ¼ 1.04 for a 1 point increase in diagnostic exam score, p ¼ 0.023), and TA visits (OR ¼ 2.34, p ¼ 0.023) were associated with passing the course.
To further investigate the effect of the math tutorial intervention, an analogous GEE multiple linear regression was run with final numerical grade as the outcome measure, as opposed to the dichotomous pass or fail.As discussed above, final course grades ranged from 2.97 to 101.95 out of a possible 100.As in the logistic GEE model, this model estimates a linear regression with a covariance structure.This model is used to predict the average gain in final score expected for students who completed the math tutorial.In a linear regression, such as given by Eq. ( 2), the coefficient β i represents the amount of change in the outcome (i.e., final grade) for a one-unit change in the predictor (i.e., completion of the intervention).Results of the GEE multiple linear regression are shown in Table V.The coefficient for completing the math tutorial was 10.96 (standard error ¼ 1.98), which means that, on average, completing the math tutorial resulted in roughly an 11 point gain in final course grade, after controlling for gender, course section, diagnostic exam, and TA visits.This is roughly equivalent to one letter grade.In contrast to the GEE logistic regression results shown in Table IV, gender was significantly related to final numerical grade (p ¼ 0.019), with male students outscoring female students, but the raw diagnostic exam score was not significantly associated with course grade (p ¼ 0.218).The decrease in correlation between diagnostic exam score and course grade may be due to the effect of the invention, as after the intervention the diagnostic exam would be less predictive of final course grade.
Both analyses indicated that students who completed the math tutorial intervention were more likely to earn a final grade of at least 50%.This may be due to the effective intervention of the math tutorial; however, an alternate explanation could be that students who completed the intervention were simply more motivated, or otherwise academically better prepared.This hypothesis was not tested directly due to the unavailability of additional data such as homework completion records or scores, or class participation or attendance records.Instead, students' SAT scores were examined as a proxy in order to assess this hypothesis since research shows that SAT scores are a good assessment of academic preparation [9,27,28], and several studies report a relationship between SAT scores and traits such as conscientiousness and "grit" [29,30].SAT scores of the students who completed the intervention versus those who did not were compared using t tests [24] (see Table II).The t test is used to test for differences between two means.
For the 231 students with diagnostic scores below 65%, there were no statistically significant differences between the total SAT scores of the students who completed the math tutorial and those who did not (p > 0.10), although students who completed the intervention had significantly lower diagnostic exam scores [tð229Þ ¼ 2.10, p ¼ 0.036].
Any preexisting motivational differences between the students are not apparent from higher SAT scores.If anything, who completed the math tutorial tended to have lower math and lower overall SAT scores; though these differences were not statistically significant, they are in opposition to the hypothesis that students who completed the math tutorial were more motivated or better prepared academically.On both measures of the SAT and diagnostic exam, at-risk students who completed the tutorial were less prepared mathematically at the beginning of the course, yet outperformed at-risk students who did not complete the tutorial.Additionally, sensitivity analyses were performed to assess the motivational theory.In this analysis, comparison groups were chosen that might be expected to either maximize or minimize the confounding effect of student motivation.First, the data were reanalyzed including only the course sections for which the intervention was an option.This should maximize the impact of any motivational component since it directly compared students who completed the tutorial with similar students who did not complete the tutorial, and had the opportunity to do so.In this case, the odds ratio for completing the math tutorial was 4.30, only slightly higher than the original OR ¼ 4.24 (p < 0.001).Second, students who completed the math tutorial were compared to at-risk students from other sections that did not have the option.This should minimize the impact of any motivational component since it excludes all students who were given a choice to complete the intervention, but chose not to.In this case, the OR dropped to 3.68 (p < 0.001).These sensitivity analyses show only a slight variation in OR between comparisons chosen to maximize or minimize the potential effect of student motivation, with the math tutorial OR remaining approximately 4. Thus, SAT and sensitivity analyses support the primary conclusion that the increase in at-risk students' odds of success is due to the math tutorial intervention.

V. CONCLUSION
Analyses of 643 students in six sections of algebra-based Introductory General Physics I show that an online math tutorial intervention was effective in improving students' likelihood to pass the course.Students in three sections were given the opportunity to complete an online math tutorial during the first two weeks of class to improve their math skills.The math tutorial was available to all students in these three sections, but students identified as at risk of failing (diagnostic exam score below 65%) were strongly encouraged to complete it.GEE logistic regression modeling showed that at-risk students who successfully completed the online math tutorial increased their odds of passing the course by roughly 4 times those of the at-risk students who did not.Examination of student characteristics, a GEE multiple linear regression, and sensitivity studies all support this finding.This online math tutorial intervention is easily implemented, short term, and can be administered concurrently with the course.Therefore, the Department of Physics has implemented the math tutorials in all sections of the introductory algebra as well as the calculus-based physics courses.
Future efforts will focus on increasing the number of atrisk students who complete the math tutorials by (i) communicating results of this study via email and internet to students enrolling in the course.This communication will include the information on the diagnostic exam date schedule, math tutorial registration, and statistics on success rates for the course; (ii) evaluating or modifying the math tutorial, such as evaluating the problems or number of problems on the tutorial as well as the passing score required for completion of the tutorial; and (iii) making the math tutorial available prior to the beginning of the semester, giving students a minimum of 6 weeks to enroll in and complete the tutorial.This would give the students the opportunity to complete the tutorial before taking the diagnostic exam, which could improve their diagnostic exam score, and hence reduce the number of at-risk students in the course.

TABLE I .
Demographic composition of sample and at-risk students.

TABLE II .
Mean SAT scores, diagnostic exam scores (%), and final numerical grades (%) for the 231 students scoring below 65% on the diagnostic exam.

TABLE IV .
GEE logistic regression predicting at-risk students passing the course.Note: Reference groups are as follows: Gender ¼ female; Course Section ¼ A; Math Intervention ¼ did not complete.Coefficient estimates and odds ratios are specified in Eqs.(2) and (3).p values represent the probability of finding the observed values if the null hypothesis of no effect is true.

TABLE V .
GEE multiple linear regression predicting at-risk students' final grades.Note that reference groups are as follows: Gender ¼ female; Course section ¼ A; Math intervention ¼ did not complete.Coefficient estimates as specified in Eq. (2).p values represent the probability of finding the observed values if the null hypothesis of no effect is true.