Departmental support structures for physics graduate students: Development and psychometric evaluation of a self-report instrument

High attrition rates in science, technology, engineering, and mathematics disciplines are an ongoing problem. Graduate student attrition, in particular, is understudied. Most past studies have focused on students ’ attributes, undergraduate preparation, and mentoring relationships. Emerging results from the implementation of the American Physical Society Bridge Program (APSBP) suggest that departmental support structures could lead to increased retention. However, there are no validated instruments to measure students ’ experience of the departmental support structures. This paper describes the development and psychometric evaluation of the aspects of student experience scale (ASES). Items were developed based on prior literature and the APSBP recommendations, revised based on APSBP feedback, and subjected to psychometric evaluation. Principal components analysis of data from 397 students at 19 physics graduate programs (2 M.S. and 17 Ph.D.) across the U.S. resulted in four distinct factors: (a) mentoring and research experience, (b) professional development, (c) social and academic integration, and (d) financial support. ASES meets the standard criteria for divergent validity, discriminant validity, and internal consistency. Results of students ’ ASES response ratings are discussed, along with comparisons between students enrolled at APSBP and non-APSBP affiliated graduate programs. On average, we find that physics graduate students experience adequate mentoring and financial support; however, they report experiencing limited social and academic integration and a lack of professional development. We also find that students in APSBP affiliated programs report better experiences on all four factors than students in non-APSBP affiliated programs, with higher differences in social and academic integration and financial support. This study is a starting point for the refinement process of this instrument. DOI


I. INTRODUCTION
Retention rates for graduate students in science and engineering doctoral-granting departments are only 59% [1]. Understanding the reasons for this high attrition is an important priority for researchers [2][3][4][5][6]. A larger concern is that attrition disproportionally affects underrepresented students (e.g., women, students of color) [6]. It is substantially important to identify the factors that lead to attrition. Students invest a significant amount of energy, money, and time, before and after their acceptance in their chosen program. Not completing their degree can be detrimental to their emotional and financial well being [7,8]. However, neither departments nor institutions are left unaffected by this outcome. The impact of graduate attrition on departmental resources is also high [8].
Despite increased attention, the underlying factors leading to attrition are still not fully clear. A substantial limitation to approach this phenomenon is the lack of a systematic way of recording the reasons that lead students to depart. Those students tend to leave silently instead of staying and voicing dissatisfaction about what eventually contributed to the decision to depart from the program [7]. Additionally, there is a clear disconnect between the factors that faculty believe lead to attrition and research outcomes related to attrition. Faculty attribute attrition to students' academic incompetence (see, e.g., Refs. [9][10][11][12]). However, if GRE scores can be considered a measure of academic competence, most studies examining academic achievement scores as predictors of graduate attrition across many disciplines found no significant relationship, except for advanced GRE scores [13]. A recent study focused on science, technology, engineering, and mathematics (STEM) disciplines found no effect of GRE scores on female students' program completion and a negative effect for male students, meaning that those with higher GRE scores were more likely to depart [4]. A qualitative study surveying graduate students and faculty from many disciplines showed that students attributed attrition at about 30% to departmental reasons, placing it as the secondhighest reason. These include inadequate advising, financial support, faculty attrition, and departmental politics. At the same time, none of the faculty members attributed attrition to any departmental factors; they believed that student factors caused attrition [10].
Similarly, most past studies examining attrition placed disproportionally more emphasis on the student, neglecting external aspects of students' academic experience. The work of Lovitts [12] shifted that direction. Since then, more studies have started to examine the effect of students' experience of departmental and institutional factors as predictors of attrition.
For many years, students' demographic characteristics, mentoring relationships, and academic competency monopolized a substantial literature. Since then, the research focus has expanded, including factors such as departmental climate and socialization processes, which are found to be correlated to attrition [12]. This leads us to the conclusion that graduate attrition is a multifactorial phenomenon. The recent outcomes from the application of key departmental recommendations from the American Physical Society Bridge Program (APSBP) add value to this notion.
Women and students of color are significantly underrepresented in physics graduate education [14][15][16]. At the same time, students of ethnic and racial minority backgrounds are more likely not to complete their program than white students [5]. The APSBP was founded to help increase racial or ethnic diversity in physics graduate education. Through this program, minority students are placed in selected graduate programs. The APSBP has also developed several key recommendations (e.g., mentoring, research engagement, monitoring student progress) that those programs were to implement to help better support student experience. Data show that those students affiliated with the APSBP have a 92% retention rate [17]. This surprising, but yet early record indicates that experiencing supportive departmental practices increases graduate retention. Based on the recommendations of the APSBP for creating supportive and inclusive departmental practices, we developed an instrument to quantify students' experience of such departmental practices. This instrument could, in turn, be used to test whether students' experiences of such supports predict retention.

II. THEORETICAL BACKGROUND
The purpose of the APSBP is to increase the diversity in physics graduate education by increasing the number of underrepresented racial minority (URM) students. Students of URM background apply to the APSBP and receive funding to start their graduate education at selected departments across the U.S. In parallel, to increase the chances of student success at these competitive programs, the APSBP has a list of key recommendations that departments are asked to apply to the bridge students. Those recommendations include admission practices, building a physics graduate student association to promote social relationships, mentoring and advising, induction (social integration practices), research engagement, monitoring student progress, resources for professional development, and practices to achieve APSBP sustainability [18]. In the following, we review the literature on the impact of some of these APSBP practices on graduate retention and students' overall experience.

A. Social integration
Tinto's influential research on undergraduate attrition [19] asserts that attrition results from the interaction between the individual student and their institution. Some recent studies focused on graduate-level education adopted this perspective. Lovitts pointed out that "the reasons for attrition are deeply embedded in the organizational culture of graduate school and the structure and process of graduate education" (Ref. [12], p. 2). Results from the study indicated that a lack of academic and social integration are predictors of attrition. Lovitts claimed that departments' physical space can help build a cooperative and social environment where integration can naturally occur. However, she pointed out that there should be a balance between task and social integration. A student who is too heavily task oriented may fail to integrate into the community's social culture, which might lead to a sense of isolation and, in turn, lead to frustration and attrition. On the other side, a similar outcome may result if a student is much more heavily integrated socially than academically. More recent studies confirmed that experiencing social isolation at graduate school predicts attrition [9]. Also, it is important to note that women are more likely to experience social isolation in STEM departments than men [20]. Unfortunately, graduate programs often exhibit two characteristics that contribute to a sense of social isolation: an unfamiliar environment to most incoming students and its stressful nature [21]. Another aspect that seems to contribute to attrition is the mismatch in how students and departments view student roles and responsibilities [9,22,23]. Therefore, as part of the socialization process into the department, faculty and administrative support are essential to help set clear expectations and discuss guidelines [23,24]. Along these lines, the APSBP acknowledged the importance of the social integration aspect and recommended that departments create opportunities that could foster a friendly and supportive environment for incoming students. This set of recommendations is listed as the APSBP induction component.

B. Academic integration
Another aspect that was shown to be correlated with attrition is the lack of support within the graduate program to help less prepared students cope with the degree requirements [8]. Golde pointed out that across the four studied departments (two science and two humanities), there was the notion that underprepared students would catch up on their own. Therefore departments lacked strategies to help support those students. Elaborating on this lacking structure, Golde concluded that (p. 686) "the pressures to demonstrate quite quickly that one is capable of course and research work meant that those who needed 'remedial' work were likely to be counseled to leave if they were unable to help themselves." The author urged us to consider how this Darwinian perspective could affect students from nonelite backgrounds. Building on this idea, we can imagine how the existence of those structures (e.g., tutoring, individualized coursework plan) could increase diversity, particularly in STEM fields. The APSBP specifically encourages departments to create an individualized coursework plan for each incoming student that will be determined based on a placement test. Under this component, other suggestions are the guided group-work sessions to help address any academic or personal issue a student might face and tutoring opportunities. The APSBP refers to this component as monitoring student progress.

C. Mentoring and research experience
The most highly cited external factor related to the student experience and retention is the relationship with advisors [22]. This is because the advisor is usually the department representative with whom graduate students interact the most during their graduate program [12,25,26]. The impact of advisors' practices influences students' experiences in the program related to their research activities, opportunities for professional development, the integration to the professional field of study, and the development of students' confidence through encouragement and praise [12,24,25]. Golde [8] found that inharmonious advising relationships, characterized by a lack of interaction, trust, and intellectual support, accounted for the attrition in two science departments. A longitudinal study of Santiago and Einarson [27] examined the influence of specific departmental structures (funding, being a member of the research group, mentorship) on science and engineering graduate students' self-efficacy and expected career outcomes upon the completion of the degree. They found that students who reported positive faculty-student interactions in their program had higher academic self-efficacy ratings. However, participation in a research group was found to negatively impact one's expectations of finding a field-related job by 65%. In this context, a field-related job may involve continuation in academia or research-focused employment by governmental or industrial laboratories. These paths are related to students' research experience, which is heavily related to their perceived mentoring experience. Four reasons may explain this latter outcome.
First, through participation in a research group, students compare themselves with other competent peers and realize the competitive nature of academia. Second, the exposure to research life reveals to a large extent the nature of faculty life, which is characterized by intense amounts of work and limited work-life balance [8,28]. Third, students might hold incorrect views on the nature of science and, in particular, of experimentation. Research often results in a high failure rate of experiments, which might lead to frustration and disappointment. Golde [8] showed that students exhibited feelings of disappointment and discouragement when they realized that scientific research and scholarship work is "incremental and slow," which was reported as one of the reasons that students in that study decided not to complete their program. And lastly, particularly in STEM disciplines, students' research projects are "assigned" by their faculty advisor. However, students in STEM fields depend on the faculty's research grant for funding their education [8,29] and, therefore, might not leave the research group right away, even if they do not find themselves to be interested in the assigned project.
These points suggest that attention should be given to how students view the nature of faculty life. Also, faculty and graduate students should openly communicate the nature and process of conducting research early on, instead of finding them out at later stages of the program. Finally, the research rotation method helps students learn about research opportunities and identify their interests. Findings from the review study of Sverdlik et al. on student-faculty supervision relationships [22] suggested that students who chose their supervisors instead of being allocated one by the department were more likely to be satisfied with their degree programs. Furthermore, it was emphasized that during the dissertation stage, students are still uncomfortable working in ill-defined environments that are different from the well-structured coursework they were experiencing as undergrads. In this case, it is the supervisors' role to guide, redirect, and monitor students' progress in the dissertation process.
The APSBP recognizes the value of effective mentoring and its effect on student experience; thus, it created a list of key points that could promote an effective student-mentor interaction. More specifically, the APSBP recommends that the department should designate multiple mentors to each student (such as peer, academic, and research mentors). The APSBP suggests that the mentor should meet regularly with the student, integrate them into the program and the physics community, guide them in selecting courses, and help them develop and complete their academic plans. These recommendations for an effective mentoring model are included under the mentoring and advising APSBP component.
The APSBP has created a list of practices to guide departments into providing a fruitful student research engagement. This component includes the suggestion to create the opportunity for research rotations, helping students identify their research interests. Also, departments are encouraged to provide resources and help the student integrate into the research group culture and encouraging them to attend and give presentations at group meetings.

D. Professional development
Departmental factors include providing opportunities for student professional development. A study by O'Meara et al. [30] explored the ways that STEM departments facilitated student career advancement by reinforcing a sense of student agency. They pointed out that a sense of agency over career advancement could contribute to degree completion, and concluded that departments should develop structured opportunities to support student agency.
The professional development component of the APSBP recommends that departments develop practices to help students develop time-management skills and learn about best practices for effective teaching. Also, departments are encouraged to create training to prepare students for the role of postdoc, develop mentoring skills, learn how to organize the laboratory, and develop effective networking skills.

E. Financial support
Students' financial situation has been found to contribute to doctoral persistence [31,32]. Students who receive financial support in any form (fellowship, graduate assistantship, scholarship) have higher chances of completing their degree than those who are self-supported. However, depending on the student's life condition, financial aid, if present, may not always be adequate. In some cases, financial aid is uncertain, or absent [33]. In other cases, students are financially dependent on their research advisors' external grants. This dependence could negatively affect student experience if they do not find interest in the advisor's research project. In this situation, the student may decide to continue working on the subject only because of the financial support or else discontinue their degree.
The APSBP provides direct funding to the bridge students for two academic years. After that, the department is responsible for providing or finding external support to students until they complete their degree.
Graduate students' life is intense in terms of time spent on coursework, teaching, and research responsibilities. This experience can be overwhelming even for the most motivated and competent student. As discussed above, there are a plethora of research studies suggesting that departments can design interventions to better support students in this process. While most of these studies focus across many disciplines (see, e.g., Refs. [7,8,12,28]), there is very little work focusing solely on STEM disciplines [29], and in particular, on the holistic experience of STEM students. Distinctive aspects of the nature of science departments suggests the need to study attrition of science disciplines in particular. Also, there is a lack of research-validated instruments to measure students' experiences regarding the existing structures in their programs. Most quantitative studies examining graduate experience focus solely on mentoring relationships (see, e.g., Ref. [27]). Following the recommendations of the APSBP to physics graduate programs on creating supportive departmental structures, we started the process of developing an instrument to quantify students' experiences. Researchers can use this instrument to examine whether and to what individual degree factors contribute to retention. Administrators can help identify lacking or at their programs from students' perspectives. This study serves as a first step toward the development process of this survey instrument.
In the following, we describe the methodology of instrument development and the results from its administration to a sample of physics graduate students from 19 programs across the U.S. We also report the results of the survey's administration to students enrolled at APSBP and non-APSBP affiliated programs, as well as interview data from the chairs or graduate advisors of those programs, which is used as a further validity check of the instrument.

A. Developing the aspects of student experience scale (ASES)
In the Fall of 2017, we reviewed prior literature on the factors that affect graduate student retention. Looking more closely at physics, we found that students enrolled in the APSBP have a much higher retention rate than the national average [18].
The APSBP has developed several recommendations to departments to foster a supportive environment. These recommendations span the space from individual advisor's actions (e.g., mentoring) to more broad administrative measures. They also include a wide chronological window, from admission practices to opportunities for professional development and research experience. We decided to develop a study to identify and quantify the outcomes for student success from the implementation of such recommendations. To do so, we needed a way to measure the departmental environment.
Inspired by the APSBP key recommendations, we developed several items about departmental factors that students could report on. For instance, the APSBP recommends that once a student issue is identified (personal or academic), departments should designate at least one individual who will be responsible for monitoring progress on addressing the issue [18]. To capture whether students experience this practice, we included the following item: "Whenever I face(d) a challenge succeeding on coursework, someone from my department helped me overcome it." None of the recommendations for departments that students could not report upon were included (e.g., do not use GRE scores to eliminate any students from consideration).
We then shared the developed items with the APSBP group and received their written feedback. We also met with the APSBP and discussed each of the items to ensure that those captured the intended recommendations. We revised the items based on their feedback and retained 35 items aimed to measure aspects of the student experience at their graduate program. We then recruited two physics graduate students and asked them to read each item and think aloud of how they approach each question. The purpose of this step was to examine whether the students understood the questions as intended and to identify biased responses through the students' answers. This helped improve survey comprehension. Three more graduate students were asked to respond to the survey to help us identify the estimated time required to respond to the items. Finally, we asked those three students to reflect on the clarity of the items and the response scale. Those retained 35 items were measured on a five-point scale ranging from 1 (strongly disagree) to 5 (strongly agree). All items were framed and coded in the same direction; thus, no reverse coding was required. Finally, we named this instrument the aspects of the student experience scale (ASES).

B. Procedure
In the Spring of 2019, we administered an online survey (15 minutes via Qualtrics) that elicited participants' demographic information, perceived experiences of departmental structures through ASES, self-efficacy, and intention to persist. In this paper, we solely focus on students' responses to the ASES and use their demographic information.
The APSBP helped us identify a list of APSBP sites. Using the data from the American Institute of Physics [14], we created a list of non-APSBP sites that shared similar characteristics with the APSBP sites (region, the number of students enrolled, rank). All the sites in the list were research-intensive physics graduate programs. We also identified the email addresses of the department chairs and the graduate advisors of those programs. The first step was to invite the chairs of those departments to participate in our study by providing us with a list of all their graduate students' contact information and sending their students an email encouraging them to take the survey. However, some programs that agreed to participate would not provide the list of students. In these cases, the chair or graduate advisor sent the invitation to the survey (on our behalf) and an anonymized link to the survey. The students were expected to click on the survey link to read the consent form and then decide whether they chose to participate. To help increase program participation, we promised to share a summary report (sent to the chair) of student responses to the survey and comparison data from the other programs in the study. The graduate committees could use this information to improve their practices. We contacted 60 physics graduate programs across the U.S. (sending two follow-up emails), and 20 programs agreed to participate. Among these 20 institutions, one had a low response rate (less than 5%) and was excluded from the results. At 14 departments, we contacted the students via direct personalized emails, which resulted in an average response rate of 28%. In comparison, at the remaining five departments where we contacted the students through the chair who sent the anonymized link to the survey, the corresponding response rate was 15.3%. We sent two weekly follow-up emails (direct or via the chair or graduate advisor) to increase participation.
The email invitation to students ensured the confidentiality of data responses. The average response rate from all the 19 departments is 23.4%. The obtained response rate is slightly lower than the reported rate for email or online surveys [34,35]. Table I   I. Characteristics of each of the 19 physics graduate programs. Note that the average response rate is 23.4%. The response rate is the ratio of the number of students that completed more than 90% of the survey items to the total number of students enrolled in the program. We purposefully do not disclose the exact number of students enrolled at each program to avoid program identification. The size refers to the total number of students enrolled at the program, where (a) ≤50, small; (b) 51-100, medium; (c) ≥101, large. Similarly, the program's ranking is presented in the following categories: (a) 1-50, high; b) 51-100, medium; c) ≥101, low. Information on the size of each program was obtained from the American Institute of Physics [14], and the ranking from the U.S. News & World Report [36].

C. Sample of participants
A total of 397 physics graduate students responded to more than 90% of the ASES items. Forty-six students either attempted the survey (completing less than 5%) or completed only the demographic items and maybe some ASES items. Those 46 students were excluded from the analysis. Of these 397 students who completed almost all the survey items, 86% are seeking a Ph.D. degree and 14% a master's degree. 61% of the respondents have passed the fourth semester (second year) of their studies, and 86.5% belong to a research group. 25.4% self-identified as cis-women or nonbinary and 74.6% as cis-men. 72% were U.S. citizens, while 28% were international students. Of the U.S. citizens, 54%, 8%, 2%, 6%, and 2% self-identified as racially White, Asian, Black/African American, ethnically Latino/Hispanic, and with more than one race, respectively. Finally, 9% are first-generation college students, 7% are parents, and 6% identified as LGBTQIA. According to the national data on physics graduate education of the American Institute of Physics [14][15][16], students that self-identified as cis-women or nonbinary are slightly overrepresented in our sample (25% vs 20%), non-U.S. citizens are significantly underrepresented (28% vs 43%), and among the U.S. citizens, URM students are also slightly overrepresented in our sample (8% vs 5.7%).

D. Data analysis 1. Principal components analysis (PCA)
All statistical analyses were conducted using the SPSS version 26.0 software. We performed a principal component analysis (PCA) using the 397 student responses to uncover the underlying structure of the 35 items that were designed to capture the extent to which students experience a supportive departmental environment. First, we ran PCA on the 35 items, which resulted in 8 retained components with eigenvalues greater than one [37]. Then, using Horn's parallel analysis (PA) [38] on all the 35 items, we retained those components whose eigenvalues were smaller than the eigenvalues of those previously obtained from PCA. The PA approach is widely recommended for factor retention in PCA [39]. This approach supported a four-factor solution. Table II shows the eigenvalues from PCA and PA and the decision for factor retention. We then reran PCA requesting four extracting factors. To identify the appropriate matrix of rotation, we requested an oblique rotation to obtain the factor correlation matrix. The factor correlation matrix for correlations was lower than 0.32, suggesting that the data are orthogonal (uncorrelated). Therefore, we reran PCA requesting now a Varimax rotation, which is appropriate for orthogonal data [40,41]. We found that four items did not load on any factor. These four items were omitted from further analysis. We then reran PCA using Varimax rotation to obtain the final model.

Reliability and construct validity
We used the composite reliability index (CRI) and Cronbach's α [42] to assess the internal consistency for each of the components. Internal consistency is a measure of the extent that all the items in a test measure the same underlying concept or construct [43]. This measure ranges from 0 to 1. CRI should be above the acceptable level of 0.60 [44], while Cronbach's α values should surpass the acceptable level of 0.70. Also, convergent and discriminant validity were both used to assess the construct validity of the instrument. Convergent validity was assessed using the average variance extracted (AVE). The squared intercomponent correlation values were used to assess divergent validity by meeting the criterion of the AVE values to be higher than the squared intercomponent correlation values.

Hedge's g effect size
The instrument was designed to measure many of recommendations of the APSBP. Therefore, as a further validity check, we used students' responses enrolled at APSBP institutions and compared those with students' responses from similar institutions (in terms of size and ranking) that are not related to the APSBP. In particular, we attempted to use data from programs that apply the APSBP recommendations across all students, such that we can test if students' responses capture the intended constructs. We also aimed to interview the graduate advisors of those programs to get their perspectives on the department's implemented practices. In our sample, 9 out of 19 departments are affiliated with the APSBP. Of these nine APSBP programs, two are known to apply the APSBP recommendations widely, across all students enrolled at the program. Here, we use students' responses and interview data from the graduate advisors of these two APSBP sites that apply to APSBP recommendations across all students. We compare those data with student survey responses and graduate advisor's interviews from two non-APSBP sites of similar size and rank. The two APSBP affiliated programs are Depts. 1 and 7, and the two non-APSBP affiliated are Depts. 3 and 10. Table III summarizes students' demographic characteristics from the two types of programs. We also found that the same percentage of the respondents have completed the second year of their studies (31.6% and 31.7% at APSBP and non-APSBP sites, respectively). While the respondents at both types of programs are at the same stage at their programs, we found that 94.7% of students at APSBP mentioned having a mentor, while this number is only 78.3% for students at non-APSBP.
The magnitude of difference in the mean ratings of the ASES between respondents in the two APSBP and the two non-APSBP affiliated sites were assessed using Hedges' g effect size (ES) analysis and confidence intervals (CIs) [45].
Researchers and editors suggest using effect size statistics with confidence intervals to report and interpret results (see, e.g., Refs. [46,47]), as the commonly used test statistics using p value can easily be misleading [48,49]. P value statistics are sensitive to sample size and do not indicate meaningful or practical significance. ES is a standardized measure of the magnitude or strength of the difference in outcome between two groups [45,48]. We use the adjusted Hedge's g for small sample sizes. CIs are used to evaluate the precision of the findings [50], which is an indicative measure of uncertainty.
Bosco et al. [51] showed that ES fluctuate across research domains, constructs, and measures. Cohen [52] proposed benchmarks for interpreting power for ES. An ES of 0.20 is "small" in magnitude, 0.50 is "medium," and values of 0.80 and above are "large." However, he noted that these values should be used carefully as a general rule of thumb, especially when there is no previous empirical evidence in the particular research area to compare to Ref. [50]. Based on previous empirical evidence for attitude factors in applied psychology, Bosco et al. [51] suggested medium ES ranging from 0.18 to 0.39. Hence, in this paper, any ES values equal to or higher than 0.18 are considered meaningful.

Interviews
We conducted in-depth, semistructured interviews with the chairs or graduate advisors of those four programs. The interviews were online, lasting for about one hour. The interview protocol included topics such as the vision for student outcomes, program admission processes, program practices to support students, departments' culture, attrition, and reasons leading to it, and alumni's trajectory. The interviews were audio and video recorded and were transcribed verbatim. Pseudonyms were assigned to protect participants' identity.
We used thematic analysis using a deductive approach [53]. Thematic analysis is a qualitative method for identifying, analyzing, and reporting patterns in the data [54]. The analysis captured the main themes in participants' interviews related to selected graduate programs' supports or practices. In particular, we wanted to explore how these existing support structures look from the department chairs or graduate advisors' perspectives. We identified themes at a semantic level, where we solely focused on the content of data at a surface level without examining underlying ideas or nuances. In this paper, we present the qualitative results to complement the quantitative data on the two components, where we found meaningful differences across the two types of programs.
The analysis followed the six phases described by Braun and Clark. These six phases are familiarizing with data, generalizing initial codes, searching for themes, reviewing themes, defining and naming themes, and producing the report [53]. Here, we report the results focused on the practices to support students'social and academic integration.

A. Principal components analysis
The Bartlett's test of sphericity, χ 2 ¼ 5038.15, df ¼ 496, p < 0.001, and a Kaiser-Meyer-Olkin measure of sampling adequacy, KMO ¼ 0.883, indicated that the resulting matrix of correlations was appropriate for factor analysis. PCA was used to determine the items that fall into each of the four identified factors by meeting the criterion of loading at least 0.4 on their respective factor. Table IV displays the loading factors from the rotated four-factor Varimax solution. The resulted four factors are (a) mentoring and research experience (MRE), (b) professional development (PD), (c) social and academic integration (SAI), and financial support (FS). The four-factor solution accounted for 48.66% of the total variance in the 31 items. More specifically, MRE (n ¼ 11) accounted for 24.95% of the variance in the data, PD (n ¼ 9) accounted for an additional 10.74%, SAI (n ¼ 8) accounted for an additional  Figure 1 shows how each APSBP corresponds to each ASES component. Table V displays the descriptions of each of the four ASES components.

B. Psychometric evaluation
Evidence for substantial internal consistency was found for all four retained factors. Using CRI, we found that all the components are above the acceptable value of 0.6 [44]. To assess construct validity, we computed both convergent and discriminant validity. Convergent validity was assessed using the AVE. We found that the values for MRE, PD, and SAI are below the acceptable value of 0.5. According to Fornell and Larcker [44], values of AVE below 0.5 are adequate, if the CRI value is above 0.6 [55]. All four components satisfy this criterion; thus, we can claim that we found support for relative convergent validity. Moreover, we computed the squared intercomponent correlation values. Meeting the criterion of the AVE values to be greater than the squared intercorrelation values, we also found evidence of substantial divergent validity. Table VII shows the AVE, squared intercomponent correlation values, and CRI values. Thus, we conclude that ASES exhibited substantial internal reliability and acceptable construct validity within this context.

C. Descriptive statistics
Descriptive statistics were established for each of the four retained components. The results are presented in Table VI. These results suggest that while students report experiencing components of MRE and FS at a somewhat adequate degree (74.3% and 77.4%, respectively), there is a lack of experiencing practices related to PD and SAI (only 46.5% and 55.5%, respectively).

D. Results from the use of the instrument on APSBP and non-APSBP sites
The mean values of students' responses from the APSBP and non-APSBP affiliated programs to the four ASES components are presented in Table VIII. Figure 2 shows the effect sizes and 95% CIs on the four ASES components. We found that Hedge's g shows a meaningful ES of 0.17 and 95% CI of ½−0.24; 0.58 between the two APSBP affiliated and nonaffiliated departments on PD, and an ES of 0.36 with 95% CI of ½−0.05; 0.77 on the MRE. At the same time, we found a very large effect size of 1.71 and 95% CI of [1.24, 2.19] on the SAI component. Moreover, we found a large effect size of 0.65 and 95% CI of [0.24, 1.07] on the FS component. These measures indicate that students at the two APSBP affiliated departments reported being better supported than students at the two non-APSBP affiliated departments on all components and with the stronger signals for SAI and FS components.
Next, we used chairs' or graduate advisors' interview data on aspects related to items included on the component of SAI, on which we found a highly large effect size between the two types of departments. We explored further for differences in the two types of departments' (APSBP affiliated and not) approaches on practices related to helping students integrate socially and academically in the program.

Social integration
The two graduate advisors of the non-APSBP departments reported that there are no formal structures designed by the departments in encouraging students' socialization processes. The graduate advisor of Dept. 10, Noah, mentioned: There are these introductory meetings of the TA training and things like that. Otherwise, not much. I will be on honest, but yeah.-Noah Similarly, the graduate advisor of Dept. 3, Anderson, mentioned that there are no formal structures initiated by the department to help student social integration. He described that while there are practices to support socialization among graduate students, especially incoming students, those are mostly organized by the local graduate student organization. More specifically, he mentioned: So, we have an orientation, a week before classes start. There's not a ton that's officially organized; the graduate student society that runs in the department is the graduate association of physics (GAP) [students]. As part of that orientation week, they do some social events for the incoming class, so that the incoming students can feel a little bit more part of the department. Then, throughout the school year, we have weekly colloquial on seminars that students can go to be part of the department. One of the nice things that GAP does is monthly mentoring with the incoming class throughout their first year. So, once a month, they meet with the incoming class or the new class, I should say, and talk about issues they might be having with classes.
[…] And then, I usually meet with GAP not long after they have these mentoring things, so I can be aware of anything going on. That requires my attention. So yeah, that's about it.-Anderson While at the non-APSBP affiliated departments we found that there are not rigidly formal structures to support students' socialization, the contrary exists for the APSBP affiliated departments. In particular, Michael, the chair of Dept. 1, mentioned that the department has an official peer mentoring program to support incoming students' socialization and blending into the department. The department also offers a room where the different graduate students' societies can host their meetings to support incoming students. More specifically, Michael, mentioned: When they come, there is an orientation.
[…] we have a mentoring program that the senior students mentor the newcomers. We have, for the graduate teaching assistants, we have this room where they sit, we have a graduate student association. We have a women in physics society at the department. There are a couple of physics societies that help new students for everything.

-Michael
Similarly, Keith, the graduate advisor of Dept. 7, described how the department makes sure that each incoming student is assigned a senior student as a mentor to help them better integrate into the department's culture. He also mentioned that there are social activities among graduate students. Keith described these opportunities for student socialization:  We have a small program, and it's a very tight group, even though we have a big university, and we start with matching them [incoming students] with the existing students. Everyone gets a peer mentor before they arrive on our campus. So, we encouraged [incoming students] to get in touch and talk about the program, provide help with local engagements, things like that. We pay much attention to make sure that the [incoming] students are immediately integrated with the existing students. There are lots of social activities, and we do a lot of one-onone meetings on the academic side, but we also encourage incoming first-year students to ask advice from the second-and third-year students.-Keith As shown above, the APSBP affiliated departments have developed structures to boost the social integration of incoming students. In particular, we saw that those structures involve peer mentoring programs, student societies, and one-on-one faculty and student meetings. These qualitative data from the department leadership complement the quantitative observed difference from student responses to the departments' approach to help student social integration.

Academic integration
Similarly to the social integration aspect, we found meaningful differences in the approach of the two types of programs related to student academic integration. More specifically, we found that the non-APSBP departments do not have a formal plan to help students with the core courses. For example, Anderson, the graduate advisor of Dept. 3, described how it is up to the teaching assistants (TAs) in these graduate core courses to offer office hours. He also mentioned that the department makes a personalized coursework plan available only to some women students. More specifically, Anderson, mentioned: The core physics courses, each core physics course has a TA. So it'd be a more senior graduate student. Those senior grad students will most likely have office hours that students can come and ask them. However, those TAs are not necessarily chosen based on their pedagogical prowess.
[…] So, not all, I would say, it hasn't necessarily worked out that those TAs are particularly good mentors. Some are, but some are not.-Anderson

And continued:
So we're actually in the midst of changing things. Basically, up to now, all incoming grad students normally would take the same course load in their first year. The only exception would usually be when women are coming in. You might have an unusual background who would go through the senior [undergraduate] courses, but then the following year takes that same suite of grad courses.-Anderson As shown in the above quotes, this non-APSBP department does not offer targeted practices to assess students' academic preparation and design a plan to increase the chances of academic success for its students. Contrary to this program, the leadership of both APSBP departments mentioned that at their departments, there are formal structures to offer an individualized coursework plan based on incoming students' preparation. Also, both noted that there are built-up practices, such as problem-solving recitations, to help students cope with the coursework material. For example, Michael described these two aspects: I meet individually with each of the new graduate students, and we look at their background, academic background, and we do a placement test, which is to see where they are, and I discuss with them. And then if I see that there is some weakness, I say, "okay, don't take three-credit core courses this semester, take just two and spread things over two years instead of doing one year and then you are going to be burnt out and not perform well." So, I tried to spot people who will struggle too much and might fail.-Michael

And continued:
We have those recitations, and for each course, there's some recitation for problem-solving. Some senior students help with the problem-solving.-Michael Similarly, the graduate advisor of Dept. 7 informed us that the department offers tutoring sessions and encourages incoming students to consult with senior students. Moreover, students at this department receive an individualized coursework plan based on their incoming academic standing. Keith, here, described these departmental supports: We do a lot of one-on-one meetings on the problemsolving side, but we also encourage first-year students to ask second and third-year students for advice. […] Every student has an individualized course plan.
[…] It is not unusual for me to suggest multiple courses for a student.

-Keith
These interview data suggest that there are substantial differences in the approach that these two types of graduate programs take concerning student social and academic integration. These results are in agreement with the quantitative effect size differences between these programs in the ASES's component of SAI.

V. LIMITATIONS
We identify the following limitations in the current study and discuss how future research directions could help eliminate them. First, although the psychometric evaluation of the developed instrument suggests that ASES exhibits acceptable construct validity and substantial internal consistency within this context, the process of instrument development is complex and more tests are required to refine further and improve this emerging instrument. More specifically, future work is needed to administer ASES to a different population of physics graduate students and use confirmatory factor analysis to test whether measures of the four identified constructs in this study are consistent with the new data. A test-retest is also needed to establish the internal consistency of the constructs further.
Moreover, the differences in the response rates among graduate programs (ranging from 11% to 66%) suggest that self-selection might exist that, in turn, could introduce response bias. In particular, we must note that students' response rate in medium to small departments was higher than in large-size departments. We can expect that nonresponse bias might be present in some departments. Furthermore, although we ensured respondents' anonymity in the email text and the consent form, we expect that some students could have avoided taking the survey in fear of identity exposure. This might have contributed to nonresponse bias, especially from students that are unhappy at their programs.
Finally, we administered the instrument to a population mostly enrolled in large and highly ranked programs. This population does not necessarily capture a representative sample of the physics departments in the country. Future work should elicit students' responses from a diverse range of departments in terms of rank and size and compare them to the results of this work.

VI. DISCUSSION
This study was designed to develop and test the aspect of student experience scale that measures students' selfreported experiences of supportive structures in their physics graduate program. ASES has 31 items and four components.
The application of principal components analysis identified four ASES components. The loadings of the clustered items under their extracted factors and the total variance explained by each factor are within the acceptable thresholds, indicating statistical significance and practical importance of the retained items, and therefore can be used in future studies. ASES's components initially demonstrated acceptable construct validity after the application of PCA through the examination of the total variance and the measures of convergent and divergent validity. We also found that ASES exhibits substantial internal consistency. However, an instrument's psychometric properties are established through a pattern of results across multiple studies and not through a single study. This work is only the beginning of the refinement of this instrument.
The application of ASES in 19 graduate programs suggested that, on average, students experience adequate mentoring and research experience and financial support, but report a lack of support in aspects related to professional development and social and academic integration. Interviews with chairs and graduate advisors of the two American Physical Society Bridge Program departments and the comparison with interview data of similar non-APSBP departments supported the validity of ASES. It confirmed the observed quantitative difference in SAI (g ¼ 1.71). Those interview data also suggested that the APSBP departments have practices to support students' social and academic integration more rigidly than similar departments.
Finally, it is worth mentioning that the four interviewees reported very different average departmental retention rates. More specifically, the two APSBP departments reported an average of 90% or more retention rates. In contrast, one of the two non-APSBP reported 60%, and the other mentioned that it always used to be 50% and only recently has increased to 80%. We see that the observed difference in students' reports of experiencing SAI and/or FS aspects could be correlated with increased retention. This is consistent with prior work that found a link between social isolation and attrition [9]. A lack of social and academic isolation is more likely to predict early attrition in the program's first two academic years. At that stage, a typical graduate student has not yet started their research. Their primary interaction during the first two years is with their peers and instructors. Similarly, the mentoring and research experience component is expected to have a high value after the second year when students focus heavily on their research. Testing this hypothesis will be the focus of a future publication.

VII. IMPLICATIONS FOR PRACTICE AND RESEARCH
The ASES was designed to help graduate programs assess students' experiences. We note that the purpose of ASES is not to evaluate the effectiveness of the graduate program. However, a pattern in student responses might indicate areas of strengths and weaknesses of the program per se. Through ASES, we intended to provide a tool for program administrators to conduct end-of-semester or year evaluations of the student experience. For example, the program might have practices to support student socialization and academic integration. However, the students might not take advantage of those valuable resources. Thus, the program would need to reflect on the resources offered and talk with students to understand why they were not being used as intended. As this is an early stage in the survey development process, we encourage graduate program administrators to view the individual item responses as the essential data provided by the instrument.
Use of the ASES might be particularly valuable for large graduate programs where student-faculty interaction is less frequent at early stages in the student program, and isolation is more likely to occur. Similarly, ASES's purpose is not to evaluate the provided mentoring from the research advisor, instead, ASES assesses whether the student received adequate mentoring regardless of the source (mentor could be from inside or outside the department).
ASES is also intended for use in scholarly work. ASES has been validated in the set of 19 institutions reported here. However, as noted earlier, further assessment of the psychometric properties in additional contexts is needed to fully evaluate this instrument's usefulness. A paper and online version of the ASES is available upon request from the authors.

VIII. CONCLUSIONS
Results from the implementation of the American Physical Society Bridge Program suggest that physics departments can create structures that help improve the student experience and result in increased retention. In this study, we developed an instrument based on the APSBP recommended practices. The psychometric evaluation of the developed instrument suggests that the aspects of student experience scale exhibited acceptable construct validity and substantial internal reliability within this context.
In our effort to further examine the validity of aspects of ASES, we used quantitative data from student responses and qualitative data from two APSBP and two non-APSBP departments. We found trends in both types of data that support the validity of the ASES. For example, the large effect size on students' experience of SAI was also indicated in the administrators' responses to those structures offered in their programs. We also found an indication that departmental factors could be correlated with increased average departmental retention. ASES can be used by researchers in graduate education to identify areas associated with student success. Also, practitioners can use ASES to identify the lack of practices at their departments and develop structures to support the student experience. Changes in department practices have the potential to increase retention and contribute toward a constructive and valuable experience for all students.