$p$-values for model evaluation

Deciding whether a model provides a good description of data is often based on a goodness-of-fit criterion summarized by a $p$-value. Although there is considerable confusion concerning the meaning of $p$-values, leading to their misuse, they are nevertheless of practical importance in common data analysis tasks. We motivate their application using a Bayesian argumentation. We then describe commonly and less commonly known discrepancy variables and how they are used to define $p$-values. The distribution of these are then extracted for examples modeled on typical data analysis tasks, and comments on their usefulness for determining goodness-of-fit are given.

Figure 1

Example data set for the case $N=25$ with Gaussian fluctuations. The fits of the four models are superposed on the data.

Figure 2

$p$-value distributions based on $R_G$. The parameters of the models discussed in Table  are fitted to $N=25$ data points and allowing parameter values in the small range. The $p$-values correspond to $N-n$ degrees-of-freedom, where $n$ is the number of fitted parameters.

Figure 3

$p$-value distributions based on $R_G$. The parameters of the models discussed in Table  are fitted to $N=25$ data points and allowing parameter values in the large range. The $p$-values correspond to $N-n$ degrees-of-freedom, where $n$ is the number of fitted parameters.

Figure 4

$p$-value distributions based on $R_G$ for model IV. The parameters of the model are fitted to $N=25$ data points and allowing parameter values in the two ranges range. The $p$-values correspond to $N-n$ degrees-of-freedom, where $n$ is the number of fitted parameters.

Figure 5

$p$-value distributions based on $R_J$. The parameters of the models discussed in Table  are fitted to $N=25$ data points and allowing parameter values in the small range. The $p$-values correspond to $N_b-1$ degrees-of-freedom, where $N_b=5$ is the number of bins.

Figure 6

$p$-value distributions based on $R_{sr}$ and $R_{fr}$. The parameters of the models discussed in Table  are fitted to $N=25$ data points and allowing parameter values in the small range.

Figure 7

Joint distribution of the $p$-values for success and failure runs. The results are for $N=25$ data points. Marker sizes in each bin are chosen relative to the bin with the highest probability of the individual model. Bins with probability less than $3.5\times {10}^{-3}$ have been excluded from the plot for the purpose of clarity.

Figure 8

$p$-value distributions based on $R_G$. The parameters of the models discussed in Table  are fitted to $N=100$ data points and allowing parameter values in the small range. The $p$-values correspond to $N-n$ degrees-of-freedom, where $n$ is the number of fitted parameters.

Figure 9

$p$-value distributions based on $R_{sr}$ and $R_{fr}$. The parameters of the models discussed in Table  are fitted to $N=100$ data points and allowing parameter values in the small range.

Figure 10

Example fits to one data set for the four models defined in Table  for $N_b=25$ and the small parameter range. The lines show the fit functions evaluated using the best-fit parameter values.

Figure 11

$p$-value distributions based on $R_P$, $R_N$, and $R_C$. The parameters of the models discussed in Table  are fitted to $N_b=25$ bins and allowing parameter values in the small range. The $p$-values correspond to $N-n$ degrees-of-freedom, where $n$ is the number of fitted parameters.

Figure 12

$p$-value distributions based on $R_L$. The parameters of the models discussed in Table  are fitted to $N_b=25$ bins and allowing parameter values in the small range. The $p$-values correspond to $N_b$ and $N_b-n$ degrees-of-freedom, where $n$ is the number of fitted parameters.

Figure 13

Typical data set used for the fitting of the exponential model. The individual event times are shown, as well as the binned contents as defined in the text. The best-fit exponential from the unbinned likelihood is also shown on the plot, normalized to the total number of fitted events.