Validation study of the Colorado Learning Attitudes about Science Survey at a Hispanic-serving institution

The Colorado Learning Attitudes about Science Survey (CLASS) has been widely acknowledged as a useful measure of student cognitive attitudes about science and learning. The initial University of Colorado validation study included only 20% non-Caucasian student populations. In this Brief Report we extend their validation to include a predominately under-represented minority population. We validated the CLASS instrument at Florida International University, a Hispanic-serving institution, by interviewing students in introductory physics classes using a semistructured protocol, examining students' responses on the CLASS item statements, and comparing them to the items' intended meaning. We find that in our predominately Hispanic population, 94% of the students' interview responses indicate that the students interpret the CLASS items correctly, and thus the CLASS is a valid instrument. We also identify one potentially problematic item in the instrument which one third of the students interviewed consistently misinterpreted.


I. INTRODUCTION
2][3][4][5][6] These instruments complement the large body of research on conceptual physics understanding and problem solving, adding a new dimension that addresses students' perceptions about science, a dimension that can be considered critical in creating a scientifically literate society.The Colorado Learning Attitudes about Science Survey ͑CLASS͒ has recently been developed, aiming to improve on previous instruments by using statistically derived categories. 7,8Validation is critical to the interpretation of these instruments to ensure that they are applied and interpreted accurately.
The CLASS improves on previous survey instruments that explore student cognitive beliefs and attitudes toward physics and learning physics in two main ways. 8First, the design utilizes questions that are both clear and concise with only one interpretation for both novices and experts.This was validated by interviews with students and faculty.Second, exploratory factor analysis was used to find category measures with high statistical reliability. 7The CLASS has been validated through both interviews and a rigorous statistical analysis during its development. 7However, previous validation studies included only 20% non-Caucasian student populations. 8In this Brief Report, we explore the validity of the interpretation of the individual CLASS items at Florida International University ͑FIU͒, where nearly 60% of the total student population is Hispanic, providing insight into the use of the CLASS with traditionally under-represented student populations. 9Furthermore, these data provide insight into our recent positive attitudinal gains measured by the CLASS in a modeling Instruction introductory physics class at FIU. 10 In the process of collecting these data, we conducted validation interviews.This Brief Report reports results of those interviews.
The CLASS contains 42 statements, to which students respond in agreement or disagreement on a five point Likert scale.Thirty-six of the 42 statements are scored by comparing the student's response to the expert's response.The other six questions are not included in the scoring because they did not load on any factors in the CLASS validation, five of these six questions are currently being rewritten.One statement, No. 31, is used to eliminate surveys from students not carefully reading the statements.Adams et al. 8 identified overall expert responses as well as eight subcategories ͑identified in Table I͒.
FIU is a large urban research university serving 38 614 students in Miami, Florida.FIU is a Hispanic-serving institution, responsible for granting the largest number of bachelor's degrees to Hispanic students in the United States.Introductory physics students at FIU may choose to enroll in one of two types of courses: a traditional format lecture course or a reform-based modeling course.The modeling course is centered on the modeling theory of science, in which students focus on building and applying models that are developed inside the classroom. 11,12At FIU the modeling sections of introductory physics consist of three sections of 30 students, which integrate both laboratory and lecture in a studio format.The focus on models leads to an emphasis in this class on conceptual understanding, problem solving skills, and the nature of science.

II. METHOD
The validation of the CLASS with our predominately Hispanic population was completed through two semesters of small group interviews with 30 students who had previously completed the CLASS. 7In the Spring 2008 semester volunteers were solicited from one section of the modeling class, 13 students volunteered and 7 students were interviewed, in three groups, based on schedule convenience at the end of their introductory electricity and magnetism course.Additional volunteers from a preservice physics teacher program were solicited and resulted in four more participants being interviewed, in two groups, at the end of their introductory mechanics course.A total of 10 of the 11 students interviewed were concurrently enrolled in a modeling section of introductory physics.The other student was enrolled in a more traditional lecture and laboratory structured introductory class at the time of the interview.In the Fall 2008 semester volunteers were requested from a section of the modeling class, as well as from the traditional lecture format classes via an online survey.Sixty-nine students volunteered to participate and 19 were chosen based on schedule convenience.Six of these 19 students were from the modeling class, 4 of whom were interviewed individually, and 13 students were from the traditional format classes, 5 of whom were interviewed individually.All 19 of these students were interviewed within the first 3 weeks of their mechanics course.Of all the 30 students who were interviewed 67% were non-Caucasian, 47% were Hispanic, and 47% were female.All students were assigned gender specific pseudonyms.
All interview participants completed the CLASS on paper prior to the interviews.An interview protocol was developed including 15 background questions and 20 of the 42 items from the CLASS instrument.The 20 items were selected based on two criteria.Either the statement was one where we expected the most student difficulty with statement interpretation based on prior experience with Maryland Physics Expectations Survey ͑Ref.3͒ or these were items where we particularly wanted to see what insights could be gained on students' cognitive attitudes and beliefs.The individual interviews were video taped and lasted approximately 40 min.The small group video-taped interviews consisted of two to four students at a time and lasted roughly 1 h.During both the individual and group interviews, one participant would first read the statement aloud, report their response from the previously administered CLASS, elaborate on their response, and provide rationale for their response.They were also given the opportunity to change their response.In group interviews, the remaining participants then gave their responses and elaborated.The initial responder was alternated in subsequent questions.Among the students who were interviewed in groups, there was a low incidence of changed answers ͑8 of 420 possible͒ as a result of the group interactions.This does not include the nine students who were interviewed individually.In analysis of the video-taped interviews it was found that only six interviewees were possibly influenced to change their answers by their group members on items 3, 8, 17, 23, and 37. Two researchers independently read each student response and compared the response to the intended meaning of the item.The developers of the CLASS designed and tested the survey such that the wording of each item would be "subject to only a single interpretation by both a broad population of students and a range of experts." 5 The researchers for this study coded each response as consistent, inconsistent, or undetermined.The two researcher's codings were then compared and an initial inter-rater reliability was established by calculating the percent agreement between researchers.The initial inter-rater agreement was found to be 92%.The researchers then met to evaluate responses on which they differed and came to consensus; all differences were resolved in this manner.Student responses that diverged from the intended meaning resulted from either a complete misinterpretation of the statement or misreading a negatively worded item.The number of students whose responses were both consistent and inconsistent with the intended meaning was recorded for each statement.Occasionally a student would not provide a response to the item that could be interpreted as consistent or inconsistent.In these cases, the consistency rate was calculated only for those students who provided a clear response.Results were compiled for each item, sorted into each of the eight categories, and are provided in Table I.Item 4 was included in the interview protocol, though it is not currently used in the scoring of the CLASS.However, as this Brief Report is intended to investigate the surface validity only of the survey, this item remains in the overall category in Table I.
The University of Colorado group developed the categories for the CLASS using a rigorous statistical methodology.Exploratory factor analysis determined statement grouping; these grouped statements were then labeled as a category.Once the categories were developed, they were only kept if they proved to be statistically robust in their correlations to the responses between statements. 8The categories found through this method were Real World Connections, Personal Interest, Sense Making/Effort, Conceptual Connections, Applied Conceptual Understanding, Problem Solving-General, Problem Solving-Confidence, and Problem Solving-Sophistication.These interviews address at least 50% of the questions in each category except the Problem Solving categories, in which only 33% of the questions are addressed.

III. RESULTS AND ANALYSIS
We find that overall the student responses were consistent with the intended meaning of the CLASS items 94% of the time.The responses given by at least 90% of students matched the intended meaning of the items in 16 of the 20 items investigated.Of the remaining four items, 3 and 13 had a consistency rate over 80%, while items 6 and 21 were at rates of 74% and 62%, respectively.Table I shows these consistency ratings listed by category.The statistical analysis of the four questionable items will be addressed in Sec.III A.

A. Real world connections
Items in this category focus on links between the physics students learn in class to that which they see in their everyday life.We collected interview data on three of four statements in this category and found that the vast majority of student responses were consistent with the intended meaning of each of these statements.For example, in response to item 28, learning physics changes my ideas about how the world works, Charles said, "I'm neutral to that.To a certain degree, it changes the way I look at the world, like whenever I do stuff, like objects being thrown or movement or motion, I think of-it's physics related or I can think of the basics of where that ball will go now, just think about it in a different way.And at the same time I don't, it's physics.I just go about my daily business.If a ball happens to fall, I'm not going to sit there and be like oh wow, that's 9.8 m / s 2 right there."Clearly in this case, Charles evaluates the statement emphasizing the change physics has had on how he sees the world working.

B. Personal interest
Again, in the Personal Interest category, students tended to be consistent with the intended meaning of the statements.Also, the students exhibited a strong connection between the physics they learn in the classroom and what they experience in their personal lives.Responding to item 3, I think about the physics I experience in everyday life, Michelle answered, "I put ͓strongly agree͔….I really do think about the physics of my everyday life….I mean it goes down to me driving my car and I'm looking at how fast I'm going and thinking about how long it's going to take."

C. Sense making/effort
The Sense Making/Effort category had the highest overall consistency rate on the items investigated.The students' responses overall to these items seemed to emphasize the need to understand the underlying mechanics of the physics before continuing with their work.Katie, for example, in response to item 23, in doing a physics problem, if my calculation gives a result very different from what I'd expect, I'd trust the calculation rather than going back through the problem, answered, "I tend to not trust calculations very much.What I was taught at a good school by a really good teacher was before looking at the answers try and think what you think the answer should be based on what you know…that's why I put strongly disagree."

D. Conceptual connections and applied conceptual understanding
As seen in Table I, the two categories with the lowest overall agreement, Conceptual Connection and Applied Con-ceptual Understanding, both include the same statements in the CLASS: item 6, Knowledge in physics consists of many disconnected topics and item 21, if I don't remember a particular equation needed to solve a problem on an exam, there's nothing much I can do (legally!)to come up with it.On the CLASS item 6, only 74% of the students interpreted the statement consistently with the intended meaning of "the topics inside of the subject of physics are disconnected."Six of the 23 students, three from individual interviews and three from two different group interviews, whose interpretation deviated from the intended meaning understood the statement to be "the topic of physics is disconnected from subjects outside of physics," as is evident from the example responses seen in Table II.Similarly, on item 21, 10 of 26 students interpreted the item to be in reference to coming up with the answer to the question, as opposed to the intended meaning of coming up with the equation that was forgotten, as seen in Table III.
In the other items in these two categories, however, we found that most of the students' interpretations were consistent with the items' intent.In item 1, a significant problem in learning physics is being able to memorize all the information I need to know, Joe responded, "So ͓strongly agree͔….You need to be able to memorize pretty much all the formulas and all the concepts behind it to really understand it."

E. Problem solving
The Problem Solving category focuses mostly on the methods students use when faced with a problem in physics.Ranging from statements on the use of equations to the ability of all students to complete physics problems, the items break down into three different groupings: Sophistication, Confidence, and General Problem Solving.Within this category, the student interpretations were consistent with intended meanings at least 83% of the time.If we exclude item 21 from this analysis, the rate of consistency rises to 96%.In response to item 13, I do not expect physics equations to help my understanding of the ideas; they are just for doing calculations, Michelle replied, "I put five, strongly agree because I think, uhm, the equations are something that help you get the mathematical answer, but…they're just for doing calculations."Though Michelle's answer is not expertlike, her response indicates that she understands this item as being about the difference between using equations for the math and using them as understanding of the concepts.Similarly, as an example of a more expertlike response John answered, "I said strongly disagree because there's a lot that we can do with the equations.I mean you could see that dimensionally they match if we have the same units.I mean granted you don't have to use the equations for everything.But for Newton's laws, for example, using each of the equations you could come to an understanding of what's really happening."In this statement, John is considering what the equations mean beyond their numerical significance, exactly the interpretation the item was meant to elicit.

F. Statistical analysis of items 3, 6, 13, and 21
In order to examine whether items 3, 6, 13, and 21 were in fact significantly different in their consistency rate from the others in the study, we performed a Fisher's exact probability test to check for independence in the distribution of consistent to inconsistent responses when compared to the average distribution of a 94% consistency rating.The results are shown in Table IV.
These results indicate that only item 21 is significantly different than the average distribution of consistent to inconsistent responses.The elimination of this item from the analysis would yield an overall consistency level of 95% for the CLASS items examined in this study.

IV. DISCUSSION AND CONCLUSIONS
We found that overall 94% of all student responses in this study indicated agreement with the intended meaning of the CLASS items.While FIU's student population is nearly 60% Hispanic, the data sample examined was only 47% Hispanic, which is within four students of being representative of FIU, but it is still appreciably larger than the 25% required to be considered a Hispanic-serving institution. 13tem 21 was the only statement with an alternate interpretation that resulted in a significant misunderstanding that we identified in this study.As discussed, this item was misinterpreted by several students to mean coming up with the answer to the problem rather than deriving the equation.This is an interesting result, as the CLASS categories discussed by Adams et al. show that the Conceptual Connections and Applied Conceptual Understanding categories have lower "robustness" scores than any of the other categories. 8This misinterpretation of item 21 may be a contributing factor to this lower statistical score.Since students do not all interpret this question correctly, the contribution of this item should be considered with caution.
Finally, in another paper, Brewe et al. has shown positive shifts in the attitudes of these students as measured by the CLASS as a result of a modeling class. 10This remarkable result required further investigation on the part of FIU to ensure accurate interpretation of the instrument.However, the results from this Brief Report suggest that these positive shift results indicate a real and significant improvement in the students' cognitive beliefs and attitudes for learning physics.
Although we identified a potentially problematic interpretation of the CLASS item 21, the evidence in this Brief Report suggests that the Colorado Learning Attitudes about Science Survey is valid for use with a predominately Hispanic student population.

TABLE I .
Percentage of students whose interpretation of the CLASS item agreed with that of the authors, listed by category.

TABLE II .
Interviewee responses from two different groups to the CLASS item 6 that differed from the intended meaning.

TABLE III .
Interviewee responses from two different groups to the CLASS item 21 that differed from the intended meaning.

TABLE IV .
Fisher's exact probability test results for distribution of consistent and inconsistent results when comparing items with consistency level less than 90% to the average 94% distribution.