Comparison of performance on multiple-choice questions and open-ended questions in an introductory astronomy laboratory

When considering the variety of questions that can be used to measure students’ learning, instructors may choose to use multiple-choice questions, which are easier to score than responses to open-ended questions. However, by design, analyses of multiple-choice responses cannot describe all of students’ understanding. One method that can be used to learn more about students’ learning is the analysis of the open-ended responses students’ provide when explaining their multiple-choice response. In this study, we examined the extent to which introductory astronomy students’ performance on multiple-choice questions was comparable to their ability to provide evidence when asked to respond to an open-ended question. We quantified students’ open-ended responses by developing rubrics that allowed us to score the amount of relevant evidence students’ provided. A minimum rubric score was determined for each question based on two astronomy educators perception of the minimum amount of evidence needed to substantiate a scientifically accurate multiple-choice response. The percentage of students meeting both criteria of (1) attaining the minimum rubric score and (2) selecting the correct multiple-choice response was examined at three different phases of instruction: directly before lab instruction, directly after lab instruction, and at the end of the semester. Results suggested that a greater proportion of students were able to choose the correct multiple-choice response than were able to provide responses that attained the minimum rubric score at both the post-lab and post-instruction phases.

Figure 1

Percentage of students meeting the minimum criteria (a) PRE-LAB, (b) POST-LAB, and (c) POST-COURSE. We denote when students choosing the correct multiple-choice response are statistically related to the students attaining the minimum rubric score with a dagger.

Figure 2

Percentage of students meeting minimum criteria when asked about the apparent motion of stars near Polaris with respect to the horizon ($n=26$).

Figure 3

Percentage of students meeting minimum criteria when asked about the length and direction of their shadow over a two week period ($n=20$).

Figure 4

Percentage of students meeting minimum criteria when asked about the portion of the celestial sphere visible from different latitudes ($n=21$).

Figure 5

Percentage of students meeting minimum criteria when asked whether or not the same stars are visible in an observer’s night sky six months apart ($n=21$).

Figure 6

Percentage of students meeting minimum criteria when asked about the Sun’s apparent motion with respect to the stars over a one day period ($n=19$).

Figure 7

Percentage of students meeting the minimum criteria when asked about the cause of the Moon’s phases ($n=23$).

Figure 8

Percentage of students meeting minimum criteria when asked to predict the phase of the Moon in two weeks given the current time and phase ($n=23$).

Figure 9

Assessment used to measure students’ ability to predict the number of stars that go below the horizon when given the position of Polaris. The distributions of students choosing the correct multiple-choice response and attaining the minimum rubric score can be found in Fig. 2.

Figure 10

Assessment used to measure students’ ability to identify how the direction and length of their shadow change over a given time period. The distributions of students choosing the correct multiple-choice response and attaining the minimum rubric score can be found in Fig. 3. This question was adapted from the Center for Astronomy Education exam bank.

Figure 11

Assessment used to measure students’ ability to identify the portion of the celestial sphere that is visible from various locations on Earth. The distributions of students choosing the correct multiple-choice response and attaining the minimum rubric score can be found in Fig. 4. This question was adapted from Paul Green’s Peer Instruction for Astronomy.

Figure 12

Assessment used to measure students’ ability to provide a description of how and why the night sky changes over the course of six months. The distributions of students choosing the correct multiple-choice response and attaining the minimum rubric score can be found in Fig. 5. This question was adapted from Paul Green’s Peer Instruction for Astronomy.

Figure 13

Assessment used to measure students’ ability to describe the Sun’s daily motion versus annual motion. The distributions of students choosing the correct multiple-choice response and attaining the minimum rubric score can be found in Fig. 6. This question was adapted from the Astronomy Diagnostic Test.

Figure 14

Assessment used to measure students’ understanding of Moon phases as a result of Earth’s perspective of the Moon’s half-lit side. The distributions of students choosing the correct multiple-choice response and attaining the minimum rubric score can be found in Fig. 7.

Figure 15

Assessment used to measure students’ ability to predict the phase of the Moon in two weeks given the current time and its current position on the observer’s horizon. The distributions of students choosing the correct multiple-choice response and attaining the minimum rubric score can be found in Fig. 8. This question was adapted from the Center for Astronomy Education exam bank.