Path integral approach to universal dynamics of reservoir computers

In this work, we give a characterization of the reservoir computer (RC) by the network structure, especially the probability distribution of random coupling constants. First, based on the path integral method, we clarify the universal behavior of the random network dynamics in the thermodynamic limit, which depends only on the asymptotic behavior of the second cumulant generating functions of the network coupling constants. This result enables us to classify the random networks into several universality classes, according to the distribution function of coupling constants chosen for the networks. Interestingly, it is revealed that such a classification has a close relationship with the distribution of eigenvalues of the random coupling matrix. We also comment on the relation between our theory and some practical choices of random connectivity in the RC. Subsequently, we investigate the relationship between the RC's computational power and the network parameters for several universality classes. We perform several numerical simulations to evaluate the phase diagrams of the steady reservoir states, common-signal-induced synchronization, and the computational power in the chaotic time series inference tasks. As a result, we clarify the close relationship between these quantities, especially a remarkable computational performance near the phase transitions, which is realized even near a nonchaotic transition boundary. These results may provide us with a new perspective on the designing principle for the RC.

Figure 1

Eigenvalue spectrum density of the random coupling matrix on the complex plane. Each random matrix is sampled from (a) the Gaussian distribution, (b) the Gamma distribution, (c) the symmetrized-Gamma distribution with $J_0/J=1.5,1/J=1,N=500$, and the mean of the symmetrized-Gamma distribution is set to be zero. Each of the figures in the center of (a)–(c) is a color map of the eigenvalue spectrum density $\rho \left(\omega \right)$. The upper and right figures are integrated density in the directions of imaginary and real axis, respectively.

Figure 2

Phase diagram of the Gaussian network with the network size $N=500$ and the mean $E\left[J_{ij}\right]=J_0/N$ and the variance $V\left[J_{ij}\right]=J^2/N$. (a) Site mean and (b) site variance of $\varphi \left(r_i\right)$, which are averaged over 100 trials. (c) Phase diagram of the steady state of the Gaussian network. We show the site mean (blue) and the site variance (orange) of $\varphi \left(r_i\right)$ in the parameter space. We set the RGB color values by (R, G, B) $=\{255(1-{\tilde{\sigma }}^2),255[1-({\tilde{\sigma }}^2+\tilde{\mu })/2],255(1-\tilde{\mu })\}$, where $\tilde{\mu }$ and ${\tilde{\sigma }}^2$ are the ensemble normalized mean and variance of MSE. The maximum and minimum values of the normalized quantities are set to be unity and zero, respectively. The contours with the dotted and dash-dotted lines indicate ${\tilde{\sigma }}^2=0.1$ and ${\tilde{\mu }}^2=0.1$, respectively. (d) Color map of the spectral radius of a random matrix for $N=500$ sampled from the Gaussian distribution. The contour lines are indicated by the common logarithm scale and their levels are displayed as labels of the colorbar. The solid lines are empirical contours and the dotted ones are loci of $(1/J,J_0/J)$ satisfying $1/J={10}^{-0.2k}\times max\{J_0,J\}(k=-2,-1,...,5)$. (e) Color map of the ensemble median of the Lyapunov exponent for Eq. (27) for the Gaussian Network with $N=500$. The black line shows the zero contour and the gray area indicates that the exponent is in the range $[0,0.001]$. The white dotted line indicates the theoretical 0 contour of the spectral radius of $J_{ij}$. (f) Median of the Lyapunov exponent for 100 ensembles. Here, we set $J_0=0$ and plot the results for $N=100$, 500 and 1000 by green, orange and blue dots, respectively. The error bars indicate the interquartile range.

Figure 3

Phase diagram of the Gamma-distributed random neural network with the network size $N=500$ and the mean $E\left[J_{ij}\right]=J_0/N$ and the variance $V\left[J_{ij}\right]=J^2/N$. (a) Site mean and (b) site variance of $\varphi \left(r_i\right)$, which are averaged over 100 trials. (c) Phase diagram of the steady state of the Gamma-distributed random neural network. We show the site mean (orange) and the site variance (blue) of $\varphi \left(r_i\right)$ in the parameter space. The RGB colors are the same as Fig. 2. The contours with the dotted and dash-dotted lines indicate ${\tilde{\sigma }}^2=0.1$ and ${\tilde{\mu }}^2=0.1$, respectively. (d) Color map of the ensemble mean of the spectral radius of random matrix sampled from the Gamma distribution for $N=500$. The solid lines are empirical contours and the dotted ones are loci of $(1/J,J_0/J)$ satisfying $1/J={10}^{-0.2k+0.05}\times {(J_0/J)}^{0.8}(k=-2,-1,...,5)$. (e) The color map of the ensemble median of the Lyapunov exponent for Eq. (27) for the Gamma network with $N=500$. The white dotted line is same as the contour of ${\tilde{\mu }}^2=0.1$ showed in panel (c).

Figure 4

Phase diagram of the symmetrized Gamma-distributed random neural network with the network size $N=500$ and the mean $E\left[J_{ij}\right]=J_0/N$ and the variance $V\left[J_{ij}\right]=J^2/N$. (a) Phase diagram of the steady state of the symmetrized Gamma-distributed random neural network. We plot site-variance of $\varphi \left(r_i\right)$, which is averaged over 100 trials. (b) color map of the spectral radius of random matrix sampled from the symmetrized Gamma distribution for $N=500$. (c) The color map of the ensemble median of the Lyapunov exponent for Eq. (27) for the symmetrized Gamma network with $N=500$. The gray area indicates that the exponent is in the range $[0,0.001]$.

Figure 5

Common signal-induced synchronization for input signal of the Lorenz system in Gaussian network ($N=500$). The color map shows the site averege and $J$-average of the variance of $tanh\left(r_i\left(t_f\right)\right)$ over all the ensemble of initial state $r_i\left(0\right)$. Common signal-induced synchronization is realized only in the regime where the values become zero. The white dotted line represents the zero contour of the Lyapunov exponent of the Gaussian network without input shown in Fig. 2 for comparison.

Figure 6

Ensemble median (upper) and interquartile range (lower) of the MSE of the time series inference task for the test data using the Gaussian (a), (d), the Gamma (b), (e), and the symmetrized Gamma networks (c), (f) with $N=500$. The white point and the white dashed line in each figure indicate the minimum point of the MSE and the contour of 95% level of the minimum, respectively. The MSEs plotted here are used in a common logarithm scale.

Figure 7

(a), (d) Ensemble median of the Lyapunov exponent of the closed loop dynamics, (b), (e) Ensemble median, and (c), (f) interquartile range of the MSE of test data of the closed loop system constructed by (upper) Gaussian and (lower) Gamma networks with $N=500$. The values showed in panels (b), (c), (e), (f) are given in the common logarithmic scale.