Mean-field theory of echo state networks

Dynamical systems driven by strong external signals are ubiquitous in nature and engineering. Here we study “echo state networks,” networks of a large number of randomly connected nodes, which represent a simple model of a neural network, and have important applications in machine learning. We develop a mean-field theory of echo state networks. The dynamics of the network is captured by the evolution law, similar to a logistic map, for a single collective variable. When the network is driven by many independent external signals, this collective variable reaches a steady state. But when the network is driven by a single external signal, the collective variable is non stationary but can be characterized by its time averaged distribution. The predictions of the mean-field theory, including the value of the largest Lyapunov exponent, are compared with the numerical integration of the equations of motion.

Figure 1

Distribution of the activation potential $a_i$. A reservoir with size $N=1000$ and normalized gain of $g=2$ and no source was run for 200 time steps, and then the histogram of the activation potential was plotted in light orange. For comparison, a Gaussian with the variance predicted by the mean-field theory was plotted (solid blue line).

Figure 2

Histogram of the variance ${\Sigma }^2$ of the activation potential in the presence of source. A reservoir of size $N=500$ with normalized gain $g=2$ and a single independent and identically distributed normal source with variance ${\xi }^2=1$ was run for 200 time steps to remove the influence of the initial state. The reservoir then was run for 30 000 time steps, the variance of the activation potential ${\Sigma }^2\left(t\right)$ was collected, and its histogram is plotted in dark gray (red). The same procedure was done following the mean-field theory prediction of Eq. (2) and plotted in light gray (orange).

Figure 3

Variance of the activation potential ${\Sigma }^2$ as a function of normalized gain $g$. For each activation potential, a reservoir of size $N=500$ with a single independent and identically distributed Gaussian source with variance ${\xi }^2=0.2$ was run according to Eqs. (1) and (2) for 200 time steps to eliminate transitory effects, and then the variance ${\Sigma }^2\left(t\right)$ of the activation potential was recorded for 2000 time steps. For each gain, the mean and standard deviation (over time) of ${\Sigma }^2\left(t\right)$ was computed. The mean is plotted with a solid gray (red) line, and the mean $\pm$ one standard deviation are plotted with a dotted gray (red) line. The light gray (orange) dashed curve [superimposed on the solid gray (red) curve in the figure] corresponds to the predictions of the mean-field theory computed according to Eq. (6). The asymptotic behaviors discussed in Sec. 4c, ${\lim }_{g^2\to 0}{\Sigma }^2={\xi }^2=0.2$ and ${\lim }_{g^2\to \infty }{\Sigma }^2=g^2$, are clearly visible on the graph.

Figure 4

Standard deviation of ${\Sigma }^2\left(t\right)$ as a function of source volatility $\xi$. The normalized gain was chosen to be $g=0.9$. Reservoirs of size $N=500$ with $K=1,5,25,500$ independent and identically distributed Gaussian source with total variance ${\xi }^2$ were simulated according to Eqs. (1) and (3). The reservoirs were run for 200 time steps to eliminate transitory effects, and then the variance ${\Sigma }^2\left(t\right)$ of the activation potential was recorded for 2000 time steps. The standard deviation (over time) of ${\Sigma }^2\left(t\right)$ was computed and is plotted in the figure using dots. This is compared the predictions of the mean-field theory [Eq. (8)], plotted using triangles. Note that the mean-field predictions go to zero as $K\to \infty$, but we cannot have more than 500 sources for a reservoir of this size.

Figure 5

Convergence of Lyapunov exponent estimate. A single reservoir of size $N=500$, normalized gain $g=2$, and source variance ${\xi }^2=0.2$ was run for 200 steps. Then the states were perturbed with independent and identically distributed Gaussians with standard deviation ${10}^{-10}$ and the size of the perturbation was recorded for 100 time steps. The Lyapunov exponent $\Lambda$ was computed for the different lags: $\sqrt[t]{{\delta }^2\left(t\right)/{\delta }^2\left(0\right)}$. The solid dark gray (red) line is the result of a simulation done according to Eqs. (1) and (2), the dotted light gray (orange) line is the result of a simulation done in the annealed approximation, and the straight dark gray (blue) line is the mean-field theory prediction computed from (10).

Figure 6

Lyapunov exponent as a function of gain. An $N=500$ reservoir with source variance ${\xi }^2=0.2$ was run for 200 time steps and then perturbed by independent and identically distributed Gaussians with standard deviation ${10}^{-12}$ and run for 20 time steps. The Lyapunov exponent was computed based on how much the perturbation changed and is plotted in dark gray (red) dots. The mean-field theory prediction is shown in a solid gray (blue) line.

Figure 7

Lyapunov exponent as a function of source volatility. An $N=500$, $g=2$ reservoir was run for 200 time steps and then perturbed by independent and identically distributed Gaussians with standard deviation ${10}^{-12}$ and run for 20 time steps. The Lyapunov exponent was computed based on how much the perturbation changed and is plotted in dark gray (red) dots. The mean-field theory prediction is shown in a solid gray (blue) line.