Training a neural network to predict dynamics it has never seen

Neural networks have proven to be remarkably successful for a wide range of complicated tasks, from image recognition and object detection to speech recognition and machine translation. One of their successes lies in their ability to predict future dynamics given a suitable training data set. Previous studies have shown how echo state networks (ESNs), a type of recurrent neural networks, can successfully predict both short-term and long-term dynamics of even chaotic systems. This study shows that, remarkably, ESNs can successfully predict dynamical behavior that is qualitatively different from any behavior contained in their training set. Evidence is provided for a fluid dynamics problem where the flow can transition between laminar (ordered) and turbulent (seemingly disordered) regimes. Despite being trained on the turbulent regime only, ESNs are found to predict the existence of laminar behavior. Moreover, the statistics of turbulent-to-laminar and laminar-to-turbulent transitions are also predicted successfully. The utility of ESNs in acting as early-warning generators for transition is discussed. These results are expected to be widely applicable to data-driven modeling of temporal behavior in a range of physical, climate, biological, ecological, and finance models characterized by the presence of tipping points and sudden transitions between several competing states.

Figure 1

(a) Schematic of an echo state network. To make a prediction $\tilde{\mathit{u}}(t+▵t)$, the flow state $\mathit{u}\left(t\right)$ at the previous time step and reservoir state $\mathit{r}\left(t\right)$ are passed to the randomly generated reservoir (green box) where they are nonlinearly transformed to yield the prediction. (b) Training time series for Reynolds numbers $\mathrm{Re}=250$, 275, 300, 500 obtained by time integration of the MFE model and shown in the form of the time evolution of the flow kinetic energy. Only shadowed parts were used for training. (c) Flow prediction made by the echo state network trained at $\mathrm{Re}=300$ (bright blue curve) and a representative turbulent trajectory of the MFE model computed at the same Reynolds number (light blue curve).

Figure 2

Lifetime distributions for Reynolds numbers from $\mathrm{Re}=200$ to $\mathrm{Re}=350$ shown in the form of survival functions. Dark (resp. light) colors correspond to the distributions generated by echo state networks (resp. MFE model). At $\mathrm{Re}=275$, the ESN result was constructed based on a distribution of survival functions computed for 100 ESNs: the squares and the dashed curve denote the median values, and the shaded area denotes the $80\%$ confidence band.

Figure 3

Prediction of turbulent-to-laminar transition at $\mathrm{Re}=500$ based on the calculation of the probability of turbulent-to-laminar transition $P_{\text{T}\to \text{L}}$ which is estimated using an ensemble approach for five initial states (red dots) at times $t_j=13840+100(j-1)$, where $j=1,\cdots ,5$. Every 10th ensemble member of the prediction generated by the echo state network is plotted in bright blue. The true flow trajectory obtained by time integration of the MFE model is plotted in light blue.

Figure 4

Laminarization probability as a function of the kinetic energy of random perturbations plotted for the MFE model (light blue curve) and the ensemble of 100 echo state networks (bright blue curve denotes the median, light blue shadowed area denotes the interdecile range) at $\mathrm{Re}=500$.

Figure 5

Predictions made by echo state networks (right) each of which was trained using the corresponding shadowed part of the time series shown on the left. Trajectories of different colors on the right were obtained using random initial conditions.

Figure 6

An example of turbulent trajectory used for finding an optimal combination of hyperparameter values. Short-term predictions are made within each colored block.