Development of the Concise Data Processing Assessment

The Concise Data Processing Assessment (CDPA) was developed to probe student abilities related to the nature of measurement and uncertainty and to handling data. The diagnostic is a ten question, multiple-choice test that can be used as both a pre-test and post-test. A key component of the development process was interviews with students, which were used to both uncover common modes of student thinking and validate item wording. To evaluate the reliability and discriminatory power of this diagnostic, we performed statistical tests focusing on both item analysis (item difficulty index, item discrimination index, and point-biserial coefficient) and on the entire test (test reliability and Ferguson’s delta). Scores on the CDPA range from chance (for novices) to about 80% (for experts), indicating that it possesses good dynamic range. Overall, the results indicate that the CDPA is a reliable assessment tool for measuring targeted abilities in undergraduate physics students.

Figure 1

Item 1 assesses whether or not students know that the measurement with the larger uncertainty should carry less weight in the answer to the question. A weighted mean could be used, but the difference in uncertainty is large enough that a student could also arrive at the correct answer by simply discarding the poorer measurement. The distractors include (a) use of the unweighted mean, (c) discarding the more precise measurement, (d) using a value of $105.5\mathrm{mL}/\mathrm{s}$, which is the value centered in the gap between $90+12\mathrm{mL}/\mathrm{s}$ and $110-1\mathrm{mL}/\mathrm{s}$, and, (e) added to balance the options. The concern might be raised about whether students worry about systematic error, and in the method of student B, in particular. This concern reflects a particular kind of expert point of view, but is not one that we believe to be valid from the student perspective. (Over the course of our interviews, we have not encountered any evidence of a student worrying about systematic error for this question; that is not to imply that it does not happen, only that it does not appear to be common.)

Figure 2

Item 2 probes a student’s judgment of an appropriate use of significant figures. The distractors include (a) the product with no rounding of significant figures, (b) the product rounded to the same precision as the multiplicand, and (d) the product rounded to an intermediate precision.

Figure 3

Item 3 assesses a student’s ability to interpret data displayed as a straight line on a semilog plot. The answer is most efficiently reached through applying knowledge that straight lines on semilog plots reflect exponential behavior, along with a basic knowledge of logarithm algebra and dimensional analysis. The distractors include (a) a straight line, ignoring the logarithm entirely, (b) the common problem of students attempting to work this out algebraically but failing to correctly manage the intercept in the calculation, and (d) the correct functional form but with incorrect units of the coefficient in the exponential. This question contains a hint to the correct answer in the form of physics content rather than pure knowledge of functions and graphs; that is, students might be helped by remembering that population growth is very often described by an exponential function.

Figure 4

Item 4 assesses a student’s ability to interpret data displayed as a straight line on a log-log plot. The answer is most efficiently reached through applying knowledge that straight lines on log-log plots reflect power-law behavior (with the power given by the slope), along with a basic knowledge of logarithm algebra and dimensional analysis. The distractors include (a) the common problem of treating the intercept on the graph incorrectly in their algebraic conversion of the logarithm, (b) in which the logarithms are ignored entirely and the formula for a straight line is given, and (d) the correct functional form but with incorrect units for the coefficients. This question contains a hint to the correct answer in the form of physics content rather than pure knowledge of functions and graphs; that is, students might be helped if they are cognizant of the Stefan-Boltzmann law.

Figure 5

Item 5 tests the student’s ability to interpret uncertainties displayed graphically and how they should be employed in weighting a straight line fit. The distractors include (a) the problem where students ascribe greater importance to the prominence of the data with larger error bars, (c) comes from striving to have an equal number of points above and below the line as well as touching the error bars of the least important, but most prominent, data, and (d) which is close to a correct weighting, but still gives too much weight to the poorest data. A chi-square statistic of the numerical data clearly shows that fit (b) is best. These data might seem contrived but can actually be produced under very natural circumstances. For example, a semilog plot of the voltage decay across a capacitor in a RC circuit [i.e., $\ln (V)$ versus $t$] results in data and error bars of this sort.

Figure 6

Item 6 is a further test of interpretation of graphically represented uncertainty and data fitting. The distractors include (b) which appears to hit two points “exactly” and balances three points each above and below the line, as well as coming very close to four prominent points arranged close to a straight line, (c) which touches the maximum number of data points as possible within their error bars, and, (d) which possesses an equal number of data points above and below the line. A chi-square statistic of the numerical data clearly shows that fit (a) is best. It is possible that students select the correct answer to this question for the wrong reason, i.e., because they believe a line of best fit should pass through the first and last data points in a data set (what the University of Cape Town group would likely classify as “point reasoning”); however, we have not encountered any evidence with our students that those (very few) who select the correct answer for this question are doing it for incorrect reasons. To eliminate this weakness, a next iteration of this question should make the second to last point have the extremal minimum uncertainty.

Figure 7

Item 7 tests the student’s ability to identify a power law in graphically displayed data. The distractor options include the tendency to think that any rapidly falling function is exponential decay, with (a) incorrect and (b) correct dimensional units in the exponential, and (d) which gives the correct power law but mishandles the units. This question contains a hint to the correct answer in the form of physics content rather than pure knowledge of functions and graphs; that is, students might be helped by recalling the ideal gas law.

Figure 8

Item 8 probes the student’s ability to identify a power law in a column of numbers. The distractor options include (b) and (c) which both present the correct power law but mishandle the units, and, (d) which again checks on the tendency to default to exponential decay. This question contains a hint to the correct answer in the form of physics content rather than pure knowledge of functions and graphs; that is, students might be helped by remembering that the inverse-square law generally applies when some conserved quantity is radiated outward radially from a point source.

Figure 9

Item 9 assesses both the understanding of rounding error in a digital instrument and the propagation of that error through a simple multiplication. The distractor options include (b) which arises from ignoring the uncertainty in the mass of a single rod, and then attaching an uncertainty of 0.01 mg to the meter, (c) attaches a 0.01 mg uncertainty to the initial measurement, rather than a 0.005 mg rounding uncertainty (note that some experts will do this since real meters often have instrumental uncertainty greater than the rounding error), (d) ignores uncertainty entirely, and, (e) which does not treat the rounding uncertainty symmetrically.

Figure 10

Item 10 assesses the student’s ability to propagate uncertainty through a simple division, either by knowing the appropriate technique for handling multiple uncertainties or with a judgement that one of the uncertainties is completely dominant. The distractor options include (a) which arises from dividing the uncertainty in the numerator by the measured quantity in the denominator, (b) which arises from dividing the uncertainty in the numerator by the uncertainty in the denominator, (c) which arises from addition of the uncertainties, and (e) which arises from multiplication of the uncertainties.