Information-theoretic model selection for optimal prediction of stochastic dynamical systems from data

In the absence of mechanistic or phenomenological models of real-world systems, data-driven models become necessary. The discovery of various embedding theorems in the 1980s and 1990s motivated a powerful set of tools for analyzing deterministic dynamical systems via delay-coordinate embeddings of observations of their component states. However, in many branches of science, the condition of operational determinism is not satisfied, and stochastic models must be brought to bear. For such stochastic models, the tool set developed for delay-coordinate embedding is no longer appropriate, and a new toolkit must be developed. We present an information-theoretic criterion, the negative log-predictive likelihood, for selecting the embedding dimension for a predictively optimal data-driven model of a stochastic dynamical system. We develop a nonparametric estimator for the negative log-predictive likelihood and compare its performance to a recently proposed criterion based on active information storage. Finally, we show how the output of the model selection procedure can be used to compare candidate predictors for a stochastic system to an information-theoretic lower bound.

Figure 1

The information atoms quantifying the relationship between $X_t$ and $X_{t-\ell }^{t-1}$. Because the order-$\ell$ entropy rate and order-$\ell$ active information storage sum to the marginal entropy $h\left[X_t\right]$, maximizing the active information storage is equivalent to minimizing the entropy rate.

Figure 2

Three possible behaviors of the active information storage for a dynamical system. (a) For an order-$p$ Markov model, the active information storage attains its maximum at $\ell =p$. (b) For a process with finite entropy rate but not finite Markov order, the finite past active information storage monotonically approaches the infinite past active information storage. (c) For an observation from a chaotic dynamical where the embedding theorem [13] applies, the active information storage diverges to infinity at a finite value $p\le 2d_B+1$, where $d_B$ is the box counting dimension of the attractor associated with the full state of the dynamical system. Here we use the convention that $\mathcal{A}\left(0\right)=0$.

Figure 3

The sampling distribution for the active information storage estimator with $k=5$ ($\widehat{\mathcal{A}}$, top) and the negative log-predictive likelihood estimator ($\widehat{\overline{h}}$, bottom) using $S=100$ realizations from the stochastic logistic map for varying time series lengths. The figures show the sample mean $\pm$ the sample standard deviation of the $S=100$ realizations. The dashed horizontal lines correspond to the exact active information storage (top) and entropy rate (bottom) for the stochastic logistic map.

Figure 4

The sampling distribution for the active information storage estimator with $k=5$ ($\widehat{\mathcal{A}}$, top) and the negative log-predictive likelihood estimator ($\widehat{\overline{h}}$, bottom) using $S=100$ realizations from the SETAR(2; 2, 2) model for varying time series lengths.

Figure 5

The mean-squared errors for the linear autoregressive (LAR), additive autoregressive (AAR), and $k$-nearest-neighbor ($k\mathrm{NN}$) predictors relative to the floor determined by the estimated entropy rate, ${\widehat{E}}_{\text{MSE},c}-\frac{e^{2\widehat{\overline{h}}}}{2\pi e}$, for the stochastic logistic map across 100 realizations. Note the break in the vertical axis.

Figure 6

The mean-squared errors for the linear autoregressive (LAR), additive autoregressive (AAR), and $k$-nearest-neighbor ($k\mathrm{NN}$) predictors relative to the floor determined by the estimated entropy rate, ${\widehat{E}}_{\text{MSE},c}-\frac{e^{2\widehat{\overline{h}}}}{2\pi e}$, for the stochastic Lorenz system across 100 realizations. Note the break in the vertical axis.