Statistical evaluation of forecasts

Reliable forecasts of extreme but rare events, such as earthquakes, financial crashes, and epileptic seizures, would render interventions and precautions possible. Therefore, forecasting methods have been developed which intend to raise an alarm if an extreme event is about to occur. In order to statistically validate the performance of a prediction system, it must be compared to the performance of a random predictor, which raises alarms independent of the events. Such a random predictor can be obtained by bootstrapping or analytically. We propose an analytic statistical framework which, in contrast to conventional methods, allows for validating independently the sensitivity and specificity of a forecasting method. Moreover, our method accounts for the periods during which an event has to remain absent or occur after a respective forecast.

Figure 1

A prediction method that raises an alarm [thick (red) vertical line] can be validated only if the event has to occur in a specific time window [solid (blue) box], the OP of length $T_o$, after the alarm was triggered. The period between the alarm and the onset of the OP is the IP of length $T_i$, in which intervention systems can be initiated [dotted (green) box]. (a) A true positive alarm is then defined as an alarm which is followed by an event in the OP. (b) If an event remains absentin the OP, the alarm is a false positive.

Figure 2

Negative predictions refer to the IP [dotted (green) box], since the time points after the IP could belong to an occurrence period of a positive prediction after the negative prediction. (a) A true negative prediction refers to a negative prediction after which no event occurs within the IP. (b) A false negative prediction refers to one after which an event occurs at the end of the IP.

Figure 3

The prediction method simulated here is based on a linear function of the linkage $c$ ($x$ axis) of predictions $a$ [solid (red) or dashed (blue) line] to the probability of an event $x=1$ ($y$ axis). While the probability of an event increases linearly with the linkage strength for positive predictions [$a=1$; solid (red) line], it decreases for negative ones [$a=0$; dashed (blue) line]; see Eq. (11).

Figure 4

Percentage at which the simulated prediction-observation time series for linkage strengths $c$ were statistically tested to be significantly better than chance. The solid (red) line shows the coverage ($c=0$) and power ($c>0$) of true positives. The dashed (blue) line shows the coverage and power of the true negatives. For this, 1000 repetitions of prediction-observation time series consisting of $10000$ data points were simulated for each linkage strength. Inset: The first 500 of these $10000$ data points. Positive predictions, i.e., alarms, are shown by long (red) vertical lines; positive observations, i.e., events, by short (blue) vertical lines.

Figure 5

Percentage at which the simulated prediction-observation time series for linkage strengths $c$ were statistically tested to be significantly better than chance. The solid (red) line shows the coverage ($c=0$) and power ($c>0$) of true positives. The dashed (blue) line shows the coverage and power of the true negatives. For this, 1000 repetitions of the prediction-observation time series consisting of 10 000 data points were simulated for each linkage strength with $\mathrm{IP}=2$, $\mathrm{OP}=4$, and $\mathrm{RP}=2$. Inset: The first 500 of these 10 000 data points. Positive predictions, i.e., alarms, are displayed by long (red) vertical lines; positive observations, i.e., events, by short (blue) vertical lines.