Theoretical model and quantitative assessment of scientific thinking and reasoning

Abilities in scientific thinking and reasoning have been emphasized as core areas of initiatives, such as the Next Generation Science Standards or the College Board Standards for College Success in Science, which focus on the skills the future will demand of today’s students. Although there is rich literature on studies of how these abilities develop in students across grade levels, the research community has not reached consensus on their definition, modeling, or assessment. To advance research in this important area, a coherent theoretical model of scientific reasoning is needed for practically guiding instruction and assessment. For decades, the only instrument available for large-scale application was the Lawson’s Classroom Test of Scientific Reasoning, but the instrument has demonstrated validity weaknesses and ceiling limitations, and its design is missing an explicit modeling framework for justifying the included skills. As a result, there is an urgent need for the development of a comprehensive modeling framework of scientific reasoning and a valid scientific reasoning assessment that targets the wide-ranging skills required for 21st century learners. This paper reports on the development of a modeling framework of scientific reasoning along with a new assessment instrument, adding to the research literature in a much needed area. The modeling framework integrates research in scientific and causal reasoning and operationally defines the skills and subskills that underlie the reasoning for knowledge development through scientific inquiry. Subsequently, this framework is used to guide the development of an assessment instrument on scientific reasoning. The validity and reliability of the instrument, which have been established based on large-scale testing, will also be discussed.

Figure 1

Essential elements and processes contributing to developing an understanding of a causal relation, which describes the process underlying time ordered events. Through observation, covariation data patterns can be generalized to provide empirical evidence of causation in terms of the covariation attributes. Meanwhile, the hypothesized mechanism can provide mechanistic explanations of why and how the causal relation and the covariation may exist. Both are needed to form a complete understanding of a causal relation.

Figure 2

Schematic of the DMCR framework for scientific reasoning to support causal decision making and knowledge formation. The variables are illustrative and do not represent any specific examples. Variable A represents features that exist in both initial and final states, which may have changed. Variable B exists only in the initial state, while variables C and D appear only in the final state. Other variables include those that are either controlled, ignored, or hidden (unknown). The solid arrows connecting the variables in the initial and final states represent the possible known and unknown temporal evolution and contextual interactions of the variables. The dashed arrows represent observations that contribute to developing MCRs and DCRs.

Figure 3

Causal networks for relations among different variables. Variables marked with an asterisk (*) represent the ones that are varied and may exhibit covariation or correlation relations. The solid lines represent covariation relations and dashed lines represent controlled effects that do not impact the covariation. (a) bivariate relational network, (b) one-level multivariate relational network, (c) multilevel multivariate relational network, (d) a complex system of multiple interconnected networks with cross-coupling and recursive feedback. Note that (e) is a special case representing a correlation between ${\mathrm{DV}}_1$ and ${\mathrm{DV}}_2$.

Figure 4

Complexity of causal reasoning processes.

Figure 5

A conceptual diagram of the IDEA-Loop model of reasoning functions.

Figure 6

Special cases of the IDEA-Loop reasoning model. (a) IDEA-Loop for hypothetical deductive reasoning and theory-evidence coordination. (b) SDDS as multiple parallel nested IDEA-Loop cycles. Double-headed arrows represent the interconnected dual-space cyclic pathways, which occur in both directions.

Figure 7

The development of knowledge and reasoning in formal and informal education settings within a specific content domain. A learner at a specific age (with some variations) can be in different stages for different content domains, while learners at different ages can also be in similar or different stages for the same or different content domains.

Figure 8

iSTAR assessment framework of scientific reasoning.

Figure 9

Modified versions of the fruit fly questions from LCTSR, both of which were adapted for iSTAR. The percentage distribution of answers are based on the college population discussed in the next section. The correct answer is marked with an asterisk (*).

Figure 10

Example questions on data analytics in iSTAR. The percentage distribution of answers are based on the college population discussed in the next section. The correct answer is marked with an asterisk (*).

Figure 11

Example questions on causal decision making in iSTAR. The percentage distribution of answers are based on the college population discussed in the next section. The correct answer is marked with an asterisk (*).

Figure 12

Score distributions of college students on iSTAR and LCTSR.

Figure 13

College students’ dimensional scores on iSTAR and LCTSR.

Figure 14

Scores on iSTAR and its subskills for graduate and undergraduate students. The error bars reflect standard errors.

Figure 15

Wright map of iSTAR.