Beyond the edge of chaos: Amplification and temporal integration by recurrent networks in the chaotic regime

Randomly connected networks of neurons exhibit a transition from fixed-point to chaotic activity as the variance of their synaptic connection strengths is increased. In this study, we analytically evaluate how well a small external input can be reconstructed from a sparse linear readout of network activity. At the transition point, known as the edge of chaos, networks display a number of desirable features, including large gains and integration times. Away from this edge, in the nonchaotic regime that has been the focus of most models and studies, gains and integration times fall off dramatically, which implies that parameters must be fine tuned with considerable precision if high performance is required. Here we show that, near the edge, decoding performance is characterized by a critical exponent that takes a different value on the two sides. As a result, when the network units have an odd saturating nonlinear response function, the falloff in gains and integration times is much slower on the chaotic side of the transition. This means that, under appropriate conditions, good performance can be achieved with less fine tuning beyond the edge, within the chaotic regime.

Figure 1

The self-consistent solution for the order parameters, $q_0$ and $q$ (which is 0), in the stationary state (a), and the Lyapunov exponent (b) plotted as a function of the synaptic variability, $g$. The dotted line at $g=1$ indicates the edge of chaos.

Figure 2

The factor $\sqrt{\gamma }$ plotted for $\phi \left(x\right)=\tanh \left(x\right)$ as a function of the synaptic variability, $g$. $\sqrt{\gamma }$ takes the maximum value of 1 at the edge of chaos ($g=1$; dotted line) and falls off more slowly in the chaotic regime ($g>1$) than for $g<1$.

Figure 3

The factor $K/R$ plotted for $\phi \left(x\right)=\tanh \left(x\right)$ as a function of the synaptic variability, $g$. In the presence of observation noise, $R$ is maximized at the edge of chaos and falls off more slowly in the chaotic regime ($g>1$) than for $g<1$ reflecting $\gamma$ shown in Fig. 2.

Figure 4

Numerical calculation of Eq. (13) (circles) with ${\sigma }_{\mathrm{obs}}=0.1$. compared with the analytic result of Eq. (22) (solid line). The numerical result was obtained by linearly decoding a simulated network with $N=3000$ and $K=20$. The small circles describe performances of each network and the large circles describe the average performance across different networks. The analytic results matched well with the numerical results.

Figure 5

The critical behavior of $R$ does not depend on details of the nonlinearity or on the noise level. For any saturating odd nonlinear response function, $R$ diverges linearly on the nonchaotic and quadratically on the chaotic side of the edge or transition. The solid line describes the asymptotic behavior; the dash-dotted line describes $\phi \left(x\right)=\tanh \left(x\right)$ and ${\sigma }_{\text{obs}}=0.1$; and the dashed line describes $\phi \left(x\right)=\text{erf}(\sqrt{\pi }x/2)$ and ${\sigma }_{\text{obs}}=0.3$.