Assessing learning in small sized physics courses

We describe the construction, validation, and testing of a concept inventory for an Introduction to Physics of Semiconductors course offered by the department of physics to undergraduate engineering students. By design, this inventory addresses both content knowledge and the ability to interpret content via different cognitive processes outlined in Bloom’s revised taxonomy. The primary challenge comes from the low number of test takers. We describe the Rasch modeling analysis for this concept inventory, and the results of the calibration on a small sample size, with the intention of providing a useful blueprint to other instructors. Our study involved 101 students from Oklahoma State University and fourteen faculty teaching or doing research in the field of semiconductors at seven universities. The items were written in four-option multiple-choice format. It was possible to calibrate a 30-item unidimensional scale precisely enough to characterize the student population enrolled each semester and, therefore, to allow the tailoring of the learning activities of each class. We show that this scale can be employed as an item bank from which instructors could extract short testlets and where we can add new items fitting the existing calibration.

Figure 1

Probabilistic model of the interaction person question. The probability of giving a correct answer to an item $i$ depends nonlinearly on the difference between the item’s difficulty and person’s ability (${\beta }_i-{\theta }_{\nu }$). The parameter ${\beta }_i$ has positive large positive values for difficult items and low negative values for easy items. The $s$-shaped logistic curves in this diagram are giving the probabilities of answering correctly for persons of different abilities ${\theta }_{\nu }$. The person ability is equal to the difficulty of the item with a 50% probability to be solved correctly by that person. The three persons in this cartoon have the abilities ${\theta }_1=-0.5$, ${\theta }_2=0.5$, and ${\theta }_3=1.5$. All the persons agree that the item of ${\beta }_3=1.5$ is the most difficult because it has the lowest probability to be solved correctly. If taking these persons in the increasing order of their ability, their probabilities to solve the item of 1.5-logit difficulty are 12%, 27%, and 50%, respectively. Similarly, all these persons agree that the item of ${\beta }_1=-0.5$ is the easiest because they all have the highest probability to solve it correctly; these probabilities are 50%, 73%, and 88%. Only the items that are ranked in the same order, from the easiest to the most difficult, by all the persons in the sample, are retained in the Rasch invariant scale.

Figure 2

The logit-unit axis simplifies the predictions with Rasch calibrated tests. Two persons, Anna and Bob, and two items, piece of cake and tough nut, are shown. From each persons’ location on the common logit axis at the top, we know that Anna has a lower probability than Bob to solve correctly any item. We know that Anna has a 50% chance to solve correctly the item piece of cake. We need Eq. (1) for calculating that Bob has an 88% chance to get piece of cake correctly, but only a 27% to solve tough nut. Anna’s chance with tough nut is of 4.74%. Alternately, we could refer to two separate probability axes going in opposite directions, one for persons and one for items. Instead of reading “Bob’s ability is of 1 logit,” one may prefer to read on the middle axis that “Bob has a 73% chance to solve correctly all the items in the test”. On the lower axis going from right to left, the probabilities to correctly solve the items are shown as percentages. An average person receiving the basic training in the field measured by the test has a probability of 73% to solve the item piece of cake but only of 12% to solve the item tough nut.

Figure 3

Four items in the PSCI. (a) An item at application level for the concept category 4-Energy Bands. The diagram for item 4.1 is an adaptation of the Figure 2.3.2 in Van Zeghbroeck’s online textbook (Ref. [54]), (b) An item at application level for the concept category 6-PN junction I–V plot, (c) An item at application level for the concept category 8-Statistics, (d) An item at evaluation level for the concept category 7-Quantum. The cartoon is inspired by the PhET interactive simulation for the “Photoelectric Effect” (Ref. [50]). All these items require procedural knowledge: to translate verbal constraints into mathematical language and to interpret diagrams and charts.

Figure 4

Rasch calibration of the items in PSCI-ALPHA. Thirty items in field testing fit the Rasch unidimensional model. The observed logits fall on a $-1$ slope line. The horizontal dashed line represents the empirical zero-logit level for the 50% item popularity. The vertical dashed line marks the intercept with the empirical zero-logit level, at $\tilde{\theta }=-0.558$ logit, and suggests that the persons in field testing had, on average, less than a 50% chance to solve correctly the entire 30-item set.

Figure 5

Likelihood ratio goodness-of-fit with median-split criterion for PSCI-ALPHA. In order to check the invariance of the Rasch scale, the item parameters are plotted for two subgroups of the sample in field testing. The item parameters estimated from the answer patterns of 41 persons with raw scores lower than median are plotted on the horizontal axis. The item parameters estimated for 41 persons with scores higher than the median are plotted on the vertical axis. The two 95% confidence intervals of each double-estimated parameter give an ellipse. The items invariant to sample split fall on or close to the bisector of slope 1 shown as a solid black line. The items situated farthest from the bisector have the poorest fit.

Figure 6

Person-Item map for PSCI-ALPHA. The black dots represent item locations. Each black dot corresponds to a tick on the upper horizontal axis. The gray bars at the top mark person locations ${\widehat{\mathrm{\theta }}}_{\nu }$ on the common logit axis. The height of a gray bar is proportional to the number of persons at that location (histogram). The 19 persons repeating the test have two distinct estimated abilities, at pre- and postinstruction; therefore, a total of $64+19$ persons are depicted in the histogram person parameter distribution. The median ability is $-0.585\pm 0.391$ logit, below the 0 logit of the test difficulty. The mode of the ability distribution is $-0.741\pm 0.399$ logit.

Figure 7

Subsets in the PSCI-BETA-Instrument. The items in each subset are represented in the increasing order of their difficulty. The horizontal dashed line stands for the zero-logit median difficulty. The shaded box PAIRS contains the labels of the six pairs of items. The items paired together are indicated by arrows starting from that pair’s label in the shaded box.

Figure 8

Assignment to a knowledge-coherence category. The person parameters estimated in the partial credit model are plotted against the raw scores for the item-subset PAIRS. The maximum possible score is 12. The dashed lines are at 1 standard error distance from the point estimate. The assignment to one of the three knowledge-coherence categories listed in the legend depends on the person’s location on the scale PAIRS.

Figure 9

Quick diagnostic chart. The name codes of the students in postinstruction testing are listed on the horizontal axis. Each student is represented by a symbol. The shape and color of the symbols denote the knowledge-coherence category. The vertical coordinate gives the PSCI-estimated ability parameter $\widehat{\mathit{\theta }}$. The highlighted band represents the 95% confidence interval of a zero (median) ability. The persons located below the highlighted band have less than 30% chance to solve correctly the entire PSCI-BETA. The persons located above the highlighted band have a chance greater than 70% to solve correctly the test.

Figure 10

Ability summary chart. The chart gives the estimated ability parameters for each of the students listed on the horizontal axis. The horizontal broken lines mark the 95% confidence interval around the median of the ability scale. The black thick dashes mark person locations on the full PSCI reference scale. The thin dotted dashes represent the persons’ locations on the subscales Application (Ap), Analysis (An), Evaluation (Ev), and PAIRS. A location at the top of the chart denotes good command of the concepts and skills addressed by the items in that subset, or high knowledge-coherence if the subset is PAIRS. A location at the bottom of the chart signals lack of appropriate knowledge to solve the items in that subset, or lack of knowledge-coherence if the subset is PAIRS. For the 35 students on this chart, there are numerous locations below the median of the Application-abilities. The instructor could infer that these students are not familiar with the concepts addressed by the set Application.