Dynamics of random recurrent networks with correlated low-rank structure

A given neural network in the brain is involved in many different tasks. This implies that, when considering a specific task, the network's connectivity contains a component which is related to the task and another component which can be considered random. Understanding the interplay between the structured and random components and their effect on network dynamics and functionality is an important open question. Recent studies addressed the coexistence of random and structured connectivity but considered the two parts to be uncorrelated. This constraint limits the dynamics and leaves the random connectivity nonfunctional. Algorithms that train networks to perform specific tasks typically generate correlations between structure and random connectivity. Here we study nonlinear networks with correlated structured and random components, assuming the structure to have a low rank. We develop an analytic framework to establish the precise effect of the correlations on the eigenvalue spectrum of the joint connectivity. We find that the spectrum consists of a bulk and multiple outliers, whose location is predicted by our theory. Using mean-field theory, we show that these outliers directly determine both the fixed points of the system and their stability. Taken together, our analysis elucidates how correlations allow structured and random connectivity to synergistically extend the range of computations available to networks.

Figure 1

Spectral outliers via low-rank perturbations. Spectrum of $J+{\mathbf{mn}}^T$ with (a) no correlations, (b) exponential overlaps, and (c) truncated overlaps, Eq. (11). See Eq. (B1) for details on the construction of $\mathbf{n}$. The values of nonzero ${\widehat{\theta }}_k$ are displayed in each plot. Orange circles and stars indicate the theoretical prediction, dots refer to the spectra of the finite-size connectivity matrices, computed numerically. (d) Overlaps ${\theta }_k={\mathbf{n}}^T{J}^k\mathbf{m}$ for the cases above. The dashed line are the target overlaps ${\widehat{\theta }}_k$ for the exponential correlation. Parameters: $N=2000$ and $g=0.8$.

Figure 2

Scaling of the norm of the rank-one perturbation with number of induced outliers. The vector $\mathbf{n}$ is the least square solution to implementing a set of outliers $\mathrm{\Lambda }=\{{\lambda }_1,\cdots ,{\lambda }_K\}$, with ${\lambda }_k=1.25+0.25k$, see Appendix pp4. (a) Log-linear plot of the Frobenius norm of $J$ and ${\mathbf{mn}}^T$ as a function of the number of outliers. The dashed line is the theoretical prediction. (b) Spectrum of $J+{\mathbf{mn}}^T$ for $K=9$ outliers. Parameters: $N=1000$ and $g=0.8$.

Figure 3

Manifold $\mathcal{M}$, Eq. (16) constraining fixed points, Eq. (15). (a) Projection of $\mathcal{M}$ for three networks with different strength of randomness $g$ (see main text for the three-dimensional basis). The negative side for $\widehat{\kappa }<0$ is symmetric and not shown. The squares on the manifolds indicate the inputs $\widehat{\kappa }=(1,2)$. The straight lines in the plane $y_{\mathbf{a}}=0$ are the asymptotic directions ${\widehat{\mathbf{x}}}_{\sim }$ for the manifolds. [(b)–(d)] Correlation ${\rho }_{12}$ between two points $\widehat{\mathbf{x}}\left({\widehat{\kappa }}_i\right)$ on $\mathcal{M}$ for two different inputs ${\widehat{\kappa }}_1,{\widehat{\kappa }}_2\in [0,3]$. (b), (c), and (d) correspond to the random connectivity strengths $g=0.1,0.5,\text{and}0.9$, respectively. Note the different scales on the color bars.

Figure 4

Two fixed points induced by rank-one perturbation correlated to the random connectivity $J$. (a) Spectrum of $J+{\mathbf{mn}}^T$. The squares indicate the corresponding averaged slopes at the fixed points, as predicted by Eq. (26). [(b) and (c)] Eigenvalues of the Jacobian at the two fixed points ${\mathbf{x}}^{\left(1\right)}$ (b) and ${\mathbf{x}}^{\left(2\right)}$ (c). Stars indicate the theoretical predictions for exceptional stability eigenvalues (only meaningful outside the bulk). (d) Fixed points and manifold $\mathcal{M}$. The colored lines indicate trajectories starting from the two fixed points (blue and orange), a point on the manifold $\mathcal{M}$ (green) and the origin. All trajectories converge on ${\mathbf{x}}^{\left(1\right)}$ or its negative counterpart. At each point, 50 different initial conditions are obtained by adding Gaussian noise (SD = 0.5). The fixed point correlation is indicated by ${\rho }_{12}$. Parameters: $g=0.8$ and $N=1000$. The rank-one perturbation is obtained by the least-squared $\mathbf{n}$, see Appendix pp4.

Figure 5

Limit cycle induced by oscillatory instability. (a) Spectrum of the connectivity matrix. The outlier ${\lambda }_3$ is real-valued and larger than one, so there is a corresponding fixed point ${\mathbf{x}}_3$. (b) The spectrum at the fixed point. The predicted stability eigenvalue ${\gamma }_{-}$ lies inside the bulk and is not labeled. (c) Fixed point and dynamics as in Fig. 4. Trajectories start at the fixed point (blue), a point on the manifold $\mathcal{M}$ (orange) and the origin (green). (d) Scalar feedback $\kappa \left(t\right)$ for the different initial conditions. Parameters: $g=0.6$ and $N=1000$. The rank-one perturbation is obtained by the least-squared $\mathbf{n}$, see Appendix pp4.

Figure 6

Fixed points and dynamics for a rank-two perturbation with structures ${\mathbf{mn}}^T$ and ${\mathbf{uv}}^T$ drawn independently of each other as well as of $J$. [(a)–(c)] Spectra of the Jacobian at the origin (a) and at the two fixed points ${\mathbf{x}}^{\left(1\right)}$ (b) and ${\mathbf{x}}^{\left(2\right)}$ (c). Stars denote the predictions for infinite size networks. (d) Projection of fixed points on vectors $\mathbf{m}$ and $\mathbf{u}$, and trajectories starting around ${\mathbf{x}}^{\left(1\right)}$ (blue), ${\mathbf{x}}^{\left(2\right)}$ (orange), ${\mathbf{x}}^{\left(1\right)}+{\mathbf{x}}^{\left(2\right)}$ (green) and $\mathbf{0}$ (red). The correlation between the two fixed points is indicated by ${\rho }_{12}$. Other parameters as in Fig. 4.

Figure 7

Stability eigenvalues ${\gamma }_{\pm }$ depend on the outlier $\lambda$ corresponding to a fixed point. The ${\gamma }_{\pm }$ are solutions to the mean-field equations, fully determined by the eigenvalue $\lambda$ and random strength $g$. (a)–(c) correspond to $g=(0.4,0.8,1.2)$, respectively. Orange lines: radius of the bulk; any eigenvalues with smaller magnitude are not observable, and hence not numerically testable for finite size networks. Where the absolute values of ${\gamma }_{\pm }$ coincide, the two form a pair of complex conjugates. Note that for $g>1$, the minimal $\lambda$ to stabilize the chaotic activity is larger than one.