Macroscopic Fluctuations Emerge in Balanced Networks with Incomplete Recurrent Alignment

Networks of strongly-coupled neurons with random connectivity exhibit chaotic, asynchronous fluctuations. In previous work, we showed that when endowed with an additional low-rank connectivity consisting of the outer product of orthogonal vectors, these networks generate large-scale coherent fluctuations. Although a striking phenomenon, that result depended on a fine-tuned choice of low-rank structure. Here we extend that work by generalizing the theory of excitation-inhibition balance to networks with arbitrary low-rank structure and show that low-dimensional variability emerges intrinsically through what we call incomplete recurrent alignment. We say that a low-rank connectivity structure exhibits incomplete alignment if its row-space is not contained in its column-space. In the setting of incomplete alignment, recurrent connectivity can be decomposed into a subspace-recurrent component and an effective-feedforward component. We show that high-dimensional, microscopic fluctuations are propagated via the effective-feedforward component to a low-dimensional subspace where they are dynamically balanced by macroscopic fluctuations. We present biologically plausible examples from excitation-inhibition networks and networks with heterogeneous degree distributions. Finally, we define the alignment matrix as the overlap between left and right-singular vectors of the structured connectivity, and show that the singular values of the alignment matrix determine the amplitude of macroscopic variability, while its singular vectors determine the structure. Our work shows how macroscopic fluctuations can emerge generically in strongly-coupled networks with low-rank structure. Furthermore, by generalizing excitation-inhibition balance to arbitrary low-rank structure our work may find relevance in any setting with strongly interacting units, whether in biological, social, or technological networks.


Introduction
The dynamic balance of excitation (E) and inhibition (I) is a paradigmatic theory for describing the activity of neocortical networks [1][2][3][4]. The theory describes how recurrent interactions generate asynchronous irregular firing activity which is typically observed in the cortex across many species, especially in awake, active states of behavior [5][6][7][8][9]. The E-I balance network model is driven by strong feed-forward excitation, and strong recurrent connectivity, dominated by inhibition, is necessary in order to balance that input. The result is that the average activity of E and I populations dynamically balance the mean synaptic input, enabling fluctuations to propagate asynchronously. The resulting networks can account for various empirical observations of cortical activity, including not only irregular firing but also low pairwise correlations [4] and broad firing-rate distributions [10]. Furthermore, balanced networks perform fast-tracking of external input which can underly predictive coding [11], and they are capable of amplifying input feature selectivity [12,13] and generating stable patterns of activity for associative memory [14][15][16].
Many qualitative aspects of the dynamics of excitation-inhibition balance can be understood by studying simpler firing-rate models [17]. Firing rate models of randomly connected excitatory and inhibitory populations with strong interactions and strong feed-forward input exhibit a dynamic cancelation of the mean input at the population level [18,19], similarly to the spiking models. The resulting state is chaotic and asynchronous, due to asymmetric random connections [20].
It had been a longstanding question whether such randomly connected networks could intrinsically generate largescale coherent fluctuations. We previously showed that when randomly connected networks are endowed with additional low-rank connectivity of a particular structure, fluctuations emerge that are shared coherently across the entire network [21]. Specifically, connectivity structure consisting of outer-products of orthogonal pairs embed a purely feedforward structure into the recurrent network such that fluctuations along the row-space are propagated to the column-space, yielding shared variability in the column-space, without generating feedback that would either supress fluctuations or drive saturation. The same qualitative phenomenon was studied also by Darshan et al [22] and Hayakawa and Fukai Hayakawa and Fukai [23].
In the current work we study a broader framework of what we refer to as "incomplete recurrent alignment," of which an orthogonal outer-product is one limiting case. In order to develop our theory of incomplete recurrent alignment, we generalize the theory of excitation-inhibition balance to networks with arbitrary low-rank structure. A number of recent studies have explored the dynamics of networks with low-rank structured connectivity in addition to a random component of connectivity, yet these have all focused on weakly-coupled low-rank structure [24,25]. We show that embuing random networks with strong low-rank connectivity yields a natural generalization of excitation-inhibition balance: a regime of dynamic balance in which strong input to a low-dimensional subspace is canceled without fine-tuning, and fluctuations can emerge in the orthogonal complement.
Using our generalized dynamic balance formalism, we show that incomplete recurrent alignment genereates macroscopic fluctuations. We say that a strongly-coupled recurrent network has incomplete alignment if the row-space of its structured connectivity is not entirely contained in its column-space. We decompose the structrued connectivity into a "subspace recurrent" component, through which macroscopic activity in the column-space is able to dynamically balance its input, and an "effective-feedforward" component, through which microscopic fluctuations in the orthogonal subspace serve as a source of fluctuating input to the macroscopic dynamics in the column-space.
In the general case of incomplete alignment, this fluctuating source from the orthogonal subspace is dynamically balanced by macroscopic fluctuations in the column-space, which we refer to as the "balance subspace". This balancing of fluctuations is analogous to the way a balanced network cancels shared fluctuations received from external sources [26], except that here the source of the fluctuating input is recurrent. The larger the extent of misalignment, the larger the macroscopic fluctuations that arrise in order to achieve balance. Our theory yields a second-order balance equation for intrinsically generated macroscopic fluctuations, and we show how the macroscopic correlation structure is fully determined by the overlap matrix between left-and right-singular vectors of the structured connectivity, while the time-course of fluctuations is inherited from the time-course of microscopic fluctuations.
In Section I we introduce the model. In Section II we present the decomposition into macroscopic order parameters that reside in the low-dimensional "balance subspace" on the one hand, and microscopic degrees of freedom in the orthogonal subpsace on the other. We show that strong low-rank connectivity yields dynamic balance -a linear equation for the macroscopic firing rates in the balance subspace, and microscopic chaotic fluctuations in the orthogonal subspace. In Section III we show that incomplete alignment of the low-rank connectivity projects the microscopic fluctuations into the balance subspace yielding amplified shared variability. We derive expressions for the amplitude, spatial structure and timescale of this variability. Finally, in Section IV we study two concrete examples of biologically relevant incomplete alignment, excitatory-inhibitory networks with degenerate synaptic weight parameters (as previously studied in [27]), and networks with heterogeneous out-degrees.

I. MODEL
We study a network of N firing-rate neurons with a connectivity matrix consisting of structured and random components. The structured component, M, is given by a rank D matrix, where D is finite in the large N limit, and the random component J has i.i.d. components assumed for simplicity to be Gaussian. Individual elements of both components of the connectivity scale as 1 √ N . Explicitly, we write the structured connectivity matrix in reduced singular value decomposition (SVD) form: where both U and V are N -by-D matrices with O (1) components and orthogonal columns of norm √ N , i.e. U T U = V T V = N I DxD (notice that we adopt here a scaling of U and V that differs from the standard SVD convention). Σ is a diagonal D-by-D matrix with O (1) positive elements, σ k .
We define the random component, J, whose elements are sampled i.i.d. from J ij ∼ N 0, g 2 N with a "gain" parameter g = O(1). For simplicity, we assume that the elements of J are drawn independently of the structure of U and V, i.e., we assume that 1 N V T J k U ≈ 0 for all k ≥ 1Schuessler et al. [28].
The dynamics of the inputs (Fig 1A) are driven by strong external drive, , and given by: where the firing rates of individual neurons are given by r i = φ (h i ), an instantaneous, sigmoidal non-linearity of the inputs. Unless otherwise mentioned, simulations and numerical calculations in this paper use φ (h)) = tanh (h)), which can be thought of as the change in firing rate relative to some baseline.
As displayed in Figure 1A, the network structure can be understood schematically: v k , i.e. the kth right-singular vector of M, performs a read-out of the network activity; this readout scaled by σ k and fed back into the network along the corresponding left-singular vector u k (Fig 1A). √ N input along the columns of U. Therefore, the subspace spanned by the columns of U can receive strong recurrent inputs that will drive saturation unless they are balanced, and it will define the macroscopic order parameters of our system.

II. BALANCE SUBSPACE DECOMPOSITION AND THE ALIGNMENT MATRIX
We call the column-space of the structured connectivity the "balance subspace". We define the decomposition into the balance subspace and its orthogonal complement: for any N -dimensional vector or matrix with N -dimensional column-space, X, we write X = X + X ⊥ , where X = 1 N UU T X is the projection in the balance subspace, and X ⊥ = I − 1 N UU T X is the projection in the orthogonal subspace.
We study the components in the balance subspace relative to the columns of U, that is we write X = UX witĥ Figure 1C displays the geometry of this decomposition for the population firing rate vector, r = Ur + r ⊥ . We will similarly decompose the input currents, h, into Uĥ + h ⊥ .
Additionally, as will be motivated in the following section, we will decompose the matrix of right-singular vectors, V = UV + V ⊥ , introducing the "alignment matrix",V, between the column-space and row-space of the structured connectivity,V The alignment matrix is the D-by-D matrix consisting of the overlap of each right-singular vector of the structured connectivity along each left-singular vector. Figure 1D displays the geometry of the two overlapping hyperplanes, given by the span of left-and right-singular vectors Importantly, if the right-singular vectors are all contained in the balance subspace, i.e. if the row-space is entirely contained in the column-space, thenthe orthogonal complement is null (V ⊥ = 0). We refer to this condition as "full recurrent alignment". Full recurrent alignment is obtained if an only ifV TV = I DxD , i.e. if the alignment matrix A B C D Figure 1. Model Dynamics and Schematic. (A) Schematic representation of network structure with external drive vector, f , random recurrent connectivity, J, and structured recurrent connectivity represented as a read-out along each right-singular vector, v k , scaled by σ k , and fedback to the network along left-singular vector, u k . (B) Balance subspace decomposition. Schematic representing the decomposition into balance subspace and the orthogonal complement. The subspace recurrence within the balance subspace is given by ΣV T . The effective-feedforward connectivity from the orthogonal complement to the balance subspace is given by ΣV T ⊥ , which is 0 in the fully aligned case. (C) Balance subspace decomposition for population vector of firing rates, r. Top: Equations definingr and r ⊥ . Bottom: Geometric visualization of the balance subspace decomposition.The projection into the balance subspace is given by the compontents,r k , along each left-singular vector, u k . The projection onto the orthgonal complement is r ⊥ . The same definitions apply for the dynamical variables, h, and the external drive, f . (D) Balance subspace decomposition of the connectivity structure, defining the alignment matrix, V. Top: Equations definingV and V ⊥ . The same definitions as in (C) are applied to a matrix of D column vectors, each of length N . Bottom: Geometrix visualization of incomplete alignment. The span of the columns of V may generally not be constrained to the balance subspace. The right-singular vector, v0, is projected onto the balance subspace where its component along each left-singular vector, u0 and u1, define the matrix elementsV00 andV10, respectively. The projection of v0 onto the orthogonal complement defines, v ⊥ 0 , the first column of V ⊥ . Similarly for all right-singular vectors, not shown. If the connectivty structure is "fully aligned", then the span of V is identical to the span of U and the right-singular vectors are obtained from the left-singular vectors by rotation and reflection, such that the alignment matrix,V, is orthonormal. is orthonormal.
We will assume the external drive is aligned with the balance subpsace so that f = Uf .

A. Balance Subspace Dynamics
We decompose the dynamics 2, and first study the the dynamics in the balance subspace, which are the macroscopic order parameters of the network. To do so we apply the above decomposition (Eqn3) on the structured connectivity, M, Eq. 1, in order to writeM = 1 √ N ΣV T . We note that by construction, M ⊥ = 0, since M projects entirely into the balance subspace.
We use the alignment matrix,V, and the orthogonal complement ,V ⊥ , to further decompose M, givingM = The first term drive the "subspace-recurrent" component of the input, which is due to activity within the balance subspace, while the second term drives the "effective-feedforward" input, which is due to activity in the orthogonal complement.
Given population firing rates, r = Ur + r ⊥ , the subspace-recurrent input is √ N ΣV Tr and the effective-feedforward input is 1 Projecting the dynamics (Eqn 2) onto the balance subspace yields the macroscopic dynamcis: term is the contribution from the random connectivity, J, onto the balance subspace, which we have ignored. Note that the effective-feedforward contribution via V ⊥ is of O (1).
These macroscopic dynamics admit a balanced fixed point governed by D linear equations: Note that the effective-feedforward input does not contribute to the balance fixed point up to leading order. It will nevertheless have an impact on the macroscopic fluctuations around the fixed point as we shall see below.
The linear balance equations and their solution generalize the 2-dimensional E-I balance equations to arbitrary low-rank structure and emphasize that they have a natural basis in the columns of U, i.e. the left singular vectors of the structured component of connectivity. Moreover these balance equations are independent of the structure of U, depending only on the singular values and the alignment matrixV.
Finally, we note that in general the balance equations may or or may not have an obtainable solution (e.g., consistent with non-saturating local rates). In the following we will assume that the local firing activations and external inputs are normalized such that the balance equations yield a feasible solution, see Appendix V and V.

B. Microscopic dynamics
We now consider the dynamics in the orthogonal complement, by projecting the full dynamics (Eqn 2) via h ⊥ = I − 1 N UU T h (recall that we assume that the external drive is contained in the balance subspace): Due to the random connectivity, J ⊥ , These microscopic dynamics can be described by a Gaussian dynamic meanfield theory which we detail in the Appendix following Kadmon and Sompolinsky [19]. The mean-field theory predicts that for sufficiently strong random connectivity these equations will generate asynchronous chaotic dynamics. The order parameter of the chaotic state is the mean single-neuron autocorrelation function C (τ ) ≡ δr i (t) δr i (t + τ ) . The implicit differential equation governing C(τ ) is derived from the dynamic mean-field theory and given in Appendix V. Note that these microscopic dynamics and the resulting mean-field theory depend on the macroscopic dynamics in the balance subspace,ĥ. Both the fixed point values ofĥ and the autocorrelation, C (τ ), must be consistent with the firing rates determined by the balance equations. We solve the dynamic mean-field theory and find a good match between simulation and theory (SI Fig 1).
As mentioned above, we assume that the balance firing rates given by Eqn 6 yield a non-saturating solution such that sufficiently strong random connectivity will yield chaotic microscopic dynamics (See Appendix V).

III. INCOMPLETE ALIGNMENT AMPLIFY FLUCTUATIONS
In this section we assume a network in the chaotic balanced state described above. In this state, the macroscopic balance subspace firing rates,r (defined via 3) satisfy the balance equations (Eqn 6), and are thus constant to leading order.
In this section we show that incomplete alignment amplifies macroscopic fluctuations in the balance subspace, δr (t) ≡r (t) − r . These are quantified by the D-by-D matrix of covariance functions: We will study the dynamics of the macroscopic, balance subspace fluctuations, δĥ ≡ĥ − ĥ , by subtracting the time-average from Eqn 5. We first consider the case of full alignment, in which V ⊥ = 0, and there is no effective-feedforward input from the orthogonal subspace to the balance subspace. The dynamics of δĥ are then given by contribution is from the random connectivity, and we remind the reader that the alignment matrix,V = 1 N U T V, is orthonormal in the case of full alignment.
In this case, we must have √ N δr 1, otherwise the input fluctuations driving δĥ will be O (1), and this will destabilize the balance state. We verify this numerically and find that in fully algined networks the total variance of temporal fluctuations, given by the trace ofĈ (0), is O 1 N 2 (as also discussed recently Kadmon et al. [11]).
To probe the impact of misalignment, we simulate networks with varying levels of alignment (see Appendix V for details) and varying network size. We find that in misaligned networks the macroscopic fluctuations are larger by an order of magnitude: their total variance is O 1 N ( Fig 3A) . Furthermore, we find that incomplete alignment yields non-trivial covariance structure, whereas in fully aligned networksĈis essentially structureless ( Fig 3B).
To explain the emergence of macroscopic fluctuations, we return to the dynamics of δĥ, in the general, V ⊥ = 0, case: where,η = 1 √ N V T ⊥ δr ⊥ , are the fluctuations in the effective-feedforward input from the orthogonal subspace to the balance subspace due to the misaligned connectvity. If the typical elements of V ⊥ are O (1) then microscopic fluctuations in the orthogonal subspace are projected to the balance subspace yielding input-fluctuations,η, that are O (1). Thus we expect incomplete alignment to drive significant macroscopic input correlations. Indeed, as we show for an example network in Figure 3C, the net input-fluctuations from the orthogonal subspace onto the balance subspace is O (1).
A stable balanced state requires that these O (1) effective feed-forward input-fluctuations fromη be canceled to leading order by recurrent balance subspace fluctuations in δr, maintaining O 1 √ N fluctuations in δĥ. This argument yields a fluctuation balance equation for the macroscopic activity order parameters: to leading order.
The fluctuation blance equation (Eqn 10), allows us to derive an analytical expression for the amplitude, structure and temporal profile of the fluctuations in δr. As we show in Appendix V: is the average single-neuron autocorrelation, which can be calculated by dynamic mean-field theory given only the balance fixed point,r * (see Appendix V and SI Fig 1).
The total temporal variance in the balance subspace is where s k are the singular values of the alignment matrix,V , showing that decreased alignment, as measured by decreased singular values of the alignment matrix, increases the macroscopic temporal fluctuations. Furthermore, we can see that as a network approaches full alignment, s k → 1, the O 1 N leading contribution to the net macroscopic fluctuations vanishes, consistent with our observation (Fig 3A) of O 1 N 2 scaling in fully aligned networks .
Our analytical expression (Eqn 11) reveals how misalignment, viaV, imprints a non-trivial spatial structure on the fluctuations in the balance subspace. From Eq. 11 (see also Appendix V), one sees that the eigenvectors of the cross-covariance,Ĉ, i.e. the principal components of the balance subspace activity, are given by the left singular vectors of the alignment matrix,V. The corresponding eigenvalues are determined by the corresponding singular values ofV, and given by C(0) Thus, the spatial structure of macroscopic fluctuations is entirely determined by the singular value decomposition of the alignment matrix. Interestingly, it is independent of Σ, the singular values of the full structured connectivity, because Σ multiplies (Eq. 9) both the effective-feedforward input,η, and the balance subspace recurrent input,V T δr, and therefore does not enter the fluctuation balance equation (Eq. 10).
Aonther noted feature of Eqn 11 is thatĈ (τ ) is a product of a D-by-D matrix and a scalar temporal profile, thus the time course of macroscopic fluctuations are identical across the D modes of the balance subspace, and are given by the average single-neuron autocorrelation function, C (τ ).
In order to verify our predictions, we construct networks with Gaussian balance subspace and a heterogeneous set of {s k } ∈ [0, 1] ( Figure 3D-F, see Appendix V for details). For our choice of parameterization we derive an expression for the total temporal variance as a function of detV = k s k , and verify it over an order of magnitude for different instantiations ofV with randomly chosen singular vectors (Fig 3F). For a given choice ofV, we show that both our closed-form expressions for the spatial structure ( Fig 3D) and our theoretical prediction of the time-course yield very good predictions (Fig 3E).
For a fixed network size, the macroscopic fluctuations grow with decreased alignment until at least one singular value, s j , is on the order of magnitude of 1 √ N . As we detail in Appendix V, at that scale, the activity of the corresponding mode is unconstrained by the leading-order balance equations (Eq. 6). At the same time, the recurrent subspace fluctuations (Eqn 9), δr, are not sufficient to fully cancel the effective-feedforward fluctuations, η, and the fluctuation balance equation (Eqn 10) cannot be satisfied. In that situation, the theory derived here breaks down, and macroscopic synchronous fluctuations with δĥ, δr ∼ O (1) can emerge. Similar scenarios were studied in [22,23]).
The self-consistent solution to the fluctuations describing the fluctuations inĥ are beyond the scope of this work.

A. Degenerate Excitation-Inhibition Balance
We first provide a sketch of the application of our generalized balance framework in the well-known setting of excitation-inhibition balance. Typically such a network is constructed by randomly and independently assigning connections to a fraction, p, of all pairs of neurons, with the synaptic weight (and sign) from neuron j to neuron i depending on each of their identities as either excitatory or inhibitory: where α i = I for N I inhibitory neurons, and α i = E for the remaining N E excitatory neurons, and the sign of the weight is corresponding to the pre-synaptic neuron, so that if α j = E (I) the jth column of W is positive (negative). Such a random binary connectivity matrix can be approximated by a low-rank, deterministic component with a 2-by-2 block structure, and an additional random component [19]. The low-rank component, though it is not symmetric because of the excitation-inhibition structure, is in general fully aligned: The balance subspace is the two-dimensional subspace spanned by two block-vectors, one with matching signs and one with opposing signs: That is, the balance subspace consists of a "sum mode" and a "difference mode" of the excitatory and inhibitory populations. The read-out performed by the structured connectivity is from the same subspace, that is, the span of {v 0 , v 1 } is identical and therefore the alignment matrix is orthonormal.
However, consider the situation in which the synaptic weight is independent of the identity of the post-synaptic   neuron: J EI = J II = J I and J EE = J IE = J E . Because the average synaptic strengths in this network do not depend on the identity of the post-synaptic neuron, the structured component of connectivity is rank one: it is the outer product of a sum-mode and a difference mode. As Helias et al Helias et al. [27] have also shown, this parameterization yields amplified fluctuations in both the E and I populations. We now show that this these fluctuations can be characterized as a specific case of our theory of incomplete alignment.
For simplicity, we assume N I = N E = N 2 , (see Appendix V for general case). Explicitly we write the structured component of connectivity as σ √ N u 0 v T 0 , according to our generalized balance formalism, with the following definitions: where 1 is the uniform column-vector of length N 2 .
We observe that the alignment matrix is a scalar in this case, and is given by: Inhibition-dominance and network stability will require that J I ≥ J E . We assume the external drive is uniform, f = 1r 0 , and then the balance equation yields population average firinĝ As the relative strength of inhibition decreases toward parity with excitation, the alignment between v 0 and u 0 shrinks: the row-vector, v T 0 , reads out the difference between net E and I activity, while the column-vector, u 0 , drives E and I equally. To avoid saturation while changing the I-E ratio, the external drive must shrink to compensate for diminished alignment, by scaling r 0 ∝vσ = J I − J E .
Applying our theory (Eqn 11) we find that reduced I-E ratio will lead to larger fluctuations of the population firing rate, By fixing J E and varying the ratio J I J E while scaling r 0 to maintain the balance firing rate, we confirm this prediction via simulations in Figure 4F.
Note that in the limit of J I → J E , the expression for the size of these fluctuations diverges. In that limit, external drive must be zero in order to avoid saturation, and the size of the fluctutaions along u 0 , i.e. coherent fluctuations shared by the entire population, is O (1). Our theory breaks down once the fluctuations inr are sufficiently large relative to 1 √ N , as they begin to significantly impact the fluctuations in the orthogonal complement [21,23].

B. Heterogeneous Out-Degrees
We now employ our theory of misalignment to study the dynamics of networks with heterogeneous out-degrees (see Appendix V for the case of both heterogeneous out-and in-degrees). Consider a single inhibitory population in which each neuron i has K i randomly chosen outgoing connections, with each non-zero synapse having weight − J √ N , where K is the average number of connections per neuron. Just as in the E-I setting, such a random binary connectivity structure can be approximated by a deterministic low-rank component and a random component of connectivity. The deterministic component in this case can be written as: where we have defined the relative out-degrees, This deterministic M is a rank-one matrix given by M = σu 0 v T 0 , with the following definitions: where p ≡ K N , k 2 is the mean-square of the relative out-degrees, and 1 is the column vector of all ones.
The alignment matrix is scalar in this case as well and is given bŷ Thus we see that the extent of alignment, and therefore the strength of "subspace reccurence" decreases with increasing breadth of the out-degree distributions The balance equation givesr  such that the balance firing rates in the population are unaffected by broadening of the out-degree. This is intuitive because the mean recurrent input is not expected to depend on the breadth of the out-degree.
Nevertheless, broad a out-degree distribution increases the extent of coherent fluctuations, as decreased alignment means an increasing of "effective-feedforward" connectivity from the orthogonal complement to the balance subspace. Concretely, we have:Ĉ Note that k 2 − 1 is exactly the variance of the relative out-degrees. The increase of correlations with broader out-degree was observed in Roxin [29].
We find that broader out-degree distribution leads to larger coherent fluctuations and confirm this prediction via simulations ( Fig 4F). It is worth noting that the structure of V ⊥ does not enter the mean-field theory, beyond the assumption that η = 1 √ N V T ⊥ δr is an independent mean-zero Gaussian process. In practice, we generate out-degrees from a log-normal distribution and find that the simulations fit the theory well within a broad range of out-degree variability, although the fit worsens as the variability increases, as presumably the Gaussian assumption is violated.

V. DISCUSSION
We have generalized the theory of dynamic excitation-inhibition balance to networks with arbitrary low-rank structured connectivity, M = 1 √ N UΣV T , together with an additive random component of connectivity. We decompose the network dynamics into the "balance subspace", given by span (U), which receives strong structured recurrent input, and its orthogonal complement within which the recurrent input is driven by random connectivity. The core insight of our theory is that the macroscopic, low-dimensional dynamics in the balance subspace are determined by the alignment matrix,V = 1 N U T V, between left-and right-singular vectors. We derive linear balance equations for the activity in the balance subspace which are a generalization of the two balance equation in excitation-inhibition networks. These balance equations are independent of the underlying statistics of the balance subspace itself, U, but rather depend only on the alignment,V, and the singular values, Σ, which scale the recurrent connectivity in each of the balance subspace modes.
We observe thatVquantifies the alignment between row-space and column-space of the structured connectivity, that is, between the the read-out and feed-back subspaces. If the read-out subspace is identical to the feed-back subspace, thenV is orthonormal. IfV is not an orthonormal matrix then the structured connectivity is not fully aligned. In this case the structured connectivity can be decomposed to a subspace-recurrent component that reads-out network activity strictly from within the balance subspace, and an effective-feedforward component that reads-out network activity from the orthogonal complement and projects it to the balance subspace.
We show that increased misalignment increases coherent, low-dimensional fluctuations in the balance subspace. This is due to the strengthening of the effective-feedforward component of the structured connectivity, which reads-out microscopic fluctuations from the orthogonal complement and projects them to the balance subspace. It is important to emphasize that the fluctuating input currents projected from the orthogonal complement to the balance subspace are of O (1), but that balance leads to a dynamic canceling of these fluctuating currents to leading order. This cancelation is manifested by a set of fluctuation-balance equations that enable us to determine the size, structure, and temporal profile of firing rate fluctuations in the balance subspace,Ĉ (τ ), as a function ofV. Despite the suppression of fluctuations via dynamic balance, the magnitude of coherent fluctuations due to misalignment is an order of magnitude larger than under fully aligned connectivity. These fluctuations increase with misalignment as a function of the singular values ofV, as the subspace-recurrent feedback is weakened requiring larger macroscopic fluctuations in order to cancel the input fluctuations from the orthogonal complement. Because they are driven by the microscopic fluctuations in the orthogonal complement, the temporal profile of these shared macroscopic fluctuations is identical to the average temporal profile of single-neuron fluctuations. The spatial structure of the fluctuations within the D-dimensional balance subspace is given by the left-singular vectors of the alignment matrix.
Note that in the limit where at least one of the singular values ofV is zero, fluctuation balance cannot be achieved along the corresponding modes. Any external drive into these modes will drive saturation, but if there is no external drive to these modes, then the fluctuations in the orthogonal complement will be propagated to this mode via the structured connectivity in a fully feed-forward manner, without being suppressed by dynamic balance. This can lead to O (1) shared fluctuations as studied in [21].
We have studied continuous rate-neuron models, for their analytical tractability. However, we expect that our theory will hold qualitatively for spiking-neuron network models as well. This is because we find that despite the elevated shared fluctuations, networks with incomplete alignment exhibit asynchronous dynamics as long as the singular values ofVare non-zero. Thus we expect that spiking networks with strong low-rank connectivity structure will operate in an asynchronous, irregular regime that is well-described by a mean-field theory [3] which can be etended to include the impact of misaligned structure as we have done here.
We have studied the structure of temporal fluctuations but our approach can be readily extended to study the structure of quenched variability over multiple realizations of the random connectivity. We expect that incomplete alignment will amplify quenched fluctuations in a similar manner.
Note that throughout this work we assume that the external drive vector,f , is such that enables a balanced fixed point. This requires that the external drive be almost entirely contained in the balance subspace (up to O 1 √ N projections). The impact of misaligned external drive is to effectively weaken the input to the balance subspace, thus shifting the balance fixed point and eventually suppressing chaos and yielding saturation, as we showed in [30]. In addition, a balanced fixed points imposes further constrains onf . For example, in standard E-I balance networks, external drive onto the E population must be sufficiently stronger that that onto the I population in order to maintain non-negative firing rates [2,31]. Describing these constraints in the general low-rank setting is beyond the scope of this work.
We note that the random component of connectivity may have statistical structure, for example different celltypes may have different overall variability of their synaptic strengths yielding a block-structure on the variances [19,28,32]. Additionally, the low-rank structure may be correlated with the random component of connectivity [28]. These types of correlations in the connectivity may have non-trivial impact in the regime of dynamic balance which may be of interest for future study.

Relationship to Previous Models of Dynamic Balance
Our framework of generalized dynamic balance unifies much previous work. For example the typical E-I balance network has rank-2 deterministic structure, in which both the column-space and row-space consist of a "sum" mode and a "difference" mode, and therefore the structure is fully aligned. Some E-I network models (e.g. [3,33]), however, use a degenerate block-structure by constraining the average weights to be independent of post-synaptic population. As we have shown here, in this case, the structured connectivity has rank 1 and is only partially aligned. The balance subspace is a sum mode, u 0 ∝ 1, while the read-out mode, v 0 , is a difference mode. As the relative strength of inhibition decreases, the recurrent connectivity becomes less alignedand yields larger and larger coherent fluctuations. These increased fluctuations in the case of degenerate E-I structure have been previously studied in [27]. As suggested in the discussion in [27], even stronger correlations arrise in the case of a "doubly-degenerate" E-I structure, in which in addition to being independent of post-synaptic population, the average weights are set to zero. Such a case was studied in [23] and corresponds to the limit whereV = 0 in our formalism.
E-I networks with distance dependent connectivity have been the focus of a number of past studies [14,22,34,35].
In the typically studied setting, each of the E and I connectivity profiles share the same periodic boundary conditions and are constrained to finite spatial frequency modes, and therefore both the column-space and row-space are spanned by a concatenated Fourier basis of E and I cells, and thereforeV is orthonormal. [22] present cases in which E-I networks with distance-dependent connectivity have a singularV. For example, they consider the case in which the I-to-E connectivity has spatial dependence while all other connectivity profiles are spatially uniform. In our formalism, the column-space (columns of U) no longer includes Fourier modes of the I population, but rather consists of Fourier modes of the E population concatenated with the zero vector over the I population. On the other hand the row-space (columns of V) does not include Fourier modes from the E population, but rather, consists of I-population Fourier modes concatenated with the zero vector over the E population, and these modes are entirely orthogonal to U so that they are fully feed-forward. Thus spatially correlated fluctuations are propagated from the I population to the E population, where they are not canceled. As further discussed in [22], the situation is different if the I population has distant-dependent connectivity internally as well. In our formalism this would mean that while columns of V would still not include Fourier modes of E, the columns of U would once again consist of the concatenated E and I Fourier modes. Thus, this is an example of partial alignment. Therefore, the spatially correlated fluctuations in the presence of I-to-I distance dependent are an order of magnitude smaller than the purely feed-forward case, they are still an order of magnitude larger than in the fully aligned case in which the E population also has distance-dependent projections (whether to itself or to the I population).
Networks with heterogeneous in-degrees have been previously shown to exhibit broken balance [36,37]. That result can be undestood in a straightforward manner in our framework of generalized balance: if the in-degrees from the external drive are not a linear combination of E and I in-degrees, then the external drive is not fully aligned with the balance subspace and there will be strong external drive to the orthogonal complement which will drive saturation [30]. In addition to heterogeneous in-degrees, here we have studied the impact of heterogeneous out-degrees on dynamic balance. We show that broad out-degree distributions in balanced networks are a form of incomplete alignment and result in increased coherent fluctuations. A similar phenomenon was observed numerically in [29]. We provide an analytical expression for the size of fluctuations as a function of the breadth of the out-degrees, and verify it numerically. We furthermore show that negative correlations between in-and out-degrees will further amplify the shared fluctuations (AppendixV) Other work has shown that balance networks can exhibit shared variability if they have shared fluctuations in their external input [38,39]. Our work assumes a constant external input.

Conclusion
Previous studies have explored particular examples of low-rank deterministic structure in balanced networks, most often via distance-dependent connectivity [14,22,34,39] or sub-population structure [19,38]. In such low-rank structures, the column-space (the span of the left-singular vectors) is typically identical to the row-space (the span of the right-singular vectors). We call such networks "fully aligned", and study the more general situation of partial alignment in which the row-space is not entirely contained in the column-space of the low-rank matrix.
We show that such incomplete alignment can have qualitative impact on network dynamics. The key feature of the structured connectivity in our analysis is the alignment matrix, comprised of the overlaps between left-and right-singular vectors.
Low-rank structured connectivity may reflect different cell-types, distance-dependence, functional connectivity, as well as heterogeneity between neurons [14,22,34,37]. Such a generalization may be important for incorporating biological realism into balance-network models. From another perspective, low-rank structure has been of recent interest in designing networks that perform specific computations [25,40]. Our work studies a new regime where low-rank connectivity is strong, and suggests a bridge between networks designed for computation and biological networks exhibiting dynamic balance.
We have developed a generalized theory of dynamic balance, which both unifies previous studes and reveals new results. Our generalization expands the study of dynamic balance to a broad class of low-rank structures -those with only partial alignment between column-space and row-space. These structures have previously only appeared in particular cases, but in our framework they appear as the general case of low-rank connectivity. We show that incomplete alignment generates coherent fluctuations via effective-feed-forward propagation from a high-dimensional subspace with microscopic chaos to a low-dimensional, balance subspace. We derive a set of fluctuation balance equations that provides an analytical solution for the structure of coherent fluctuations in the balance subspace.
This theory may find relevance well beyond neuroscience. Recent studies of complex systems attempt to explore the relationship between structure and dynamics in a variety of real-world networks [41,42]. Many of these studies limit the strength of interaction between units (∼ 1 N ) in order to facilitate mean-field approaches. The theory of excitation-inhibition balance studies a regime of strong interactions (∼ 1 √ N ) but until now its application has remained limited to neuroscience because of the excitation-inhibition structure (Dale's Law). Our generalized framework of dynamic balance may be relevant for any setting with strongly interacting units, whether biological, social, or technological networks.

Appendix
Appendix A -Self-Consistency of the Balance Solution As discussed in the bain text, the macroscopic dynamics in the balanced subspace (Eqn 5) admit a balanced fixed point governed by the D linear balance equations: ΣV Tr * +f ≈ 0 (Eqn 6 in the main text). Balance is achieved when the firing rates in the balance subspace satisfy this equation up to a finite-size correction of 1 √ N . These firing rates,r, are given by the components of the full population firing rate vector, r = φ (h), along U: As we shall see, these D equations constrain both the residual fields in the balance subspace,ĥ * , as well as the statistics of the microscopic, local dynamics in the orthogonal complement, h ⊥ .
The dynamical state in the orthogonal complement can be either a stable fixed point (FP) or chaos. Given a fixed h * , those dynamics are described by a mean-field theory that predicts that the microscopic degrees of freedom, h ⊥i , are be described as independent, identical, mean-zero Gaussians, either fixed in time or fluctuating with a monotonically decaying autocorrelation. The mean-field theory follows previous work (e.g. [19,20]), and we detail it for our setting below in Appendix V. The result of the mean-field theory is that if balance is achieved, avoiding saturation, then there is a FP with Gaussian statistics that transitions to chaos for sufficiently strong random component of connectivity. The expression for the variance in the orthogonal subspace, ∆ 0 = h 2 ⊥i , depends only onĥ * and g. In the FP regime, ∆ 0 is the spatial disorder and is given by a single implicit equation, while in the chaotic regime, ∆ 0 averages both spatial and temporal disorder and is constrained by a pair of equations together with the spatial variance of the single-neuron long-time averages.
Given, ∆ 0 ,r is given by first averaging over the Gaussian component in the orthogonal subspace and then projecting the result onto balance subspace: where Dz is an integral over the standard normal measure, Dz = dz 2π e − z 2 2 . Combining the balance balance requirementr * ≈r bal (Eqn 6) gives us a set of D implicit equations, which given ∆ 0 , determineĥ * : We stress that this Gaussian mean-field equation for self-consistency in the balance subspace holds regardless of whether the dynamics in the orthogonal subspace are in the FP or chaotic regime, but the total variance in the orthogonal subspace, ∆ 0 , must also be found in a manner self-consistent withĥ * ,as we show below.
We now detail the mean-field theory describing the dynamics in the orthogonal subspace for both fixed point and chaos. We will assume an approximate fixed point in the balance subspace,ĥ * , that enables the corresponding firing rates,r * , to satisfy the D balance equations (Eqn 6).
We further assume full external alignment, as we have throughout the main text, so that the external drive, f , does not project into the orthogonal subspace. The dynamics are given by: where, we remind the reader, X ⊥ = I − 1 N UU T X is the orthogonal compliment of the vector or matrix of column-vectors, X, and the vector of firing rates is given by r = φ (h). Through the non-linearity, φ, the dynamics in the orthogonal complement depend on the dynamics in the balance subspace,ĥ * .

B1 Fixed Point and its Stability
The fixed point equation in the orthogonal subspace is Given a fixedĥ * , we follow the mean-field theory presented, for example in [19], which treats the recurrent drive due to the random connectivity, as independent, identical, mean-zero Gaussians. This theory assumes that due to the random connectivity, N j J ⊥ij r j 2 ≈ j J 2 ⊥ij r 2 j ≈ N J 2 ij r 2 j , which therefore determines the spatial variance at the fixed point, ∆ 0 ≡ h 2 ⊥i , to be The mean-square firing rates, r 2 j = φ 2 (h j ) , must be found by averaging over the network, where h j has a component in the balance subspace and an additional Gaussian component with variance ∆ 0 . Because the two components are independent, we can perform the average over the Gaussian randomness before averaging over the population, and thus we arrive at an implicit mean-field fixed-point equation for ∆ 0 : where Dz means averaging over the standard normal measure Dz = e − z 2 2 dz.
Given a fixedĥ * , the Jacobian matrix for the stability of the fixed point in the orthogonal complement is given by J ⊥ij φ (h j ). As we have done previously, we assume that h j is independent of J ⊥ij , and this Jacobian matrix is a random matrix with column-wise variance. The support of the eigenvalues of such a matrix is identical that of a random matrix with uniform variance that is given by the average of the column-wise variances [43]. Thus, the effective gain at the fixed point, which is given by the maximal real part of the eigenvalues of the Jacobian, is given by g 2 ef f = J 2 ⊥ij φ (h j ) 2 , which can be calculated as: The fixed point in the orthogonal complement, h * ⊥ , will be stable for g ef f < 1, and for g ef f > 1 the microscopic dynamics in the orthogonal compliment will be chaotic. Those dynamcis are described by a dynamic mean-field theory (DMFT), detailed in the next section.
The stability calculation here is equivalent to considering perturbations within the orthogonal complement, with the balance subspace held fixe. A complete treatment of stability should consider arbitrary perturbations, in both the orthogonal complement and the balance subspace, following [19,25].

B2 -Dynamic Mean-Field Theory of Chaos in the Orthogonal Subspace
In the chaotic regime, the input to the orthogonal subspace can still be considered Gaussian but its temporal statistics must be derived, that is we seek to find the autocorrelation: The dynamics in the orthogonal subspace can be represented by a single stochastic differential equaiton: where η i (t) = j J ⊥ ij r j (t). Again, due to the randomness of the connectivity, η i has mean-zero over neurons and time, and the average autocorrelation of η i is a scaled version of the average autocorrelation of r i : where we have written for the average autocorrelation of the firing rates. Note that the notation now denotes averaging over both time and neurons. Additionally, note that we have inserted the notation C T for the total autocorrelation in order to differentiate from the mean single-neuron temporal autocorrelation ( 2 ) introduced in the main text in Section III, Equation 11.
where u i is the ith row of the left-singular vector matrix, and gives the projection of neuron i in the balance subspace. Next we rewrite the two correlated Gaussians, h ⊥ i (t) and h ⊥ i (t + τ ), via three independent Gaussians, one of which contributes the correlated component: where we have introduced ∆ 0 ≡ ∆ (0) as the total variance. These three Gaussians need to be integrated over, and then the balance subspace structure averaged to yield Thus given the autocorrelation, ∆ (τ ), in the orthogonal complement, we have an expression for the average single neuron firing rate autocorrelation, C T (τ ). Next, following the standard DMFT approach ( [19,20]) we write an implicit differential equation that determines the autocorrelation self-consistently: The total variance, ∆ 0 , is the initial condition of Eqn 44 and must be found self-consistently along with the conditions ∂ ∂τ ∆ (0) = 0 and ∂ 2 ∂τ 2 ∆ (∞) = 0. As detailed in [19,20], the differential equation (Eqn 44) can be re-expressed as a one-dimensional dynamics under a potential energy. Givenĥ, the initial condition, ∆ 0 , can be found by using the requirement that the potential energy at the initial condition equals its value at τ → ∞, together with the fact that ∆ (∞) = g 2 C T (∞). Therefore, in practice, given the fixed point in the balance subspace,ĥ * , the total variance in the orthogonal subspace, ∆ 0 = h ⊥ In sum, the mean-field characterization of the system determinesĥ * and ∆ 0 via either D + 1 equations (in the FP regime) or D + 2 equations (in the chaotic regime). When these equations are satisfied the network generalizes the dynamic balance of excitation-inhibition. This balance is dynamic in the sense that without fine-tuning, the macroscopic firing rates in the balance subspace self-adjust to cancel the strong external drive. Depending on the single-neuron transfer function and the strength of the random component of connectivity, balance can take the form of either a stable fixed-point, or the more familiar balanced state of chaotic dynamics with local fluctuations propagating in the orthogonal complement.

Appendix C -The Case of Gaussian Structured Connectivity
Here we study the mean-field theory for the case in which the structured connectivity is Gaussian. In particular, the elements of U are taken to be i.i.d. by N (0, 1). The Gaussianity construction greatly simplifies the mean-field expression forr k (Eqn 30). In the general case, calculatingr k requires first averaging r i = φ u T iĥ + h ⊥ i over the variability in the orthogonal subspace by a Gaussian integral for each neuron i (where u i is the ith row of U as above), and then computing a weighted average over the structure of the kth column of U. That iŝ In the Gaussian case, the other modes l = k contribute Gaussian variablilty which can simply be added to the variability from the orthogonal subspace, and then u ik can be averaged over as an additional Gaussian. Thereforê r k reduces to a double Gaussian integral: one integral for the weighted average over the structure of the kth mode, which is coupled toĥ k , and a second Gaussian that combines the remaining D − 1 structured modes together with the orthogonal complement:r The Gaussian integral over u can be performed via integration by parts. Explicitly, one writes w ≡ e − u 2 2 u and dv ≡ φ du, then´dv w = −´dw v =´Du ∂φ ∂u . Now the two Gaussians, u and z, combine to a single Gaussian integral:r We find that due to the Gaussianity of U,ĥ is proportional tor. Therefore the D equations forĥ k reduce to one implicit scalar equation for ĥ : where, of course,r is determined to leading order by the balance equations.
The Gaussianity of U also simplifies the mean-field calculation of the variance in the orthogonal complement, ∆ 0 . In the FP equation (Eqn 35) the sum over neurons can be replaced by a Gaussian integral over the balance subspace and combined with the Gaussian integral over the orthogonal complement to yield Similarly, the effective gain at the FP in the orthogonal complement is given by Furthermore, we can simplify the dynamic mean-field expression for the total autocorrelation , , as well. In Eqn 43: Thus we find that in the Gaussian setting, the DMFT calculation of the autocorrelation in the orthogonal subspace, ∆ (τ ), depends only on the norm in the balance subspace, ĥ . We exploit this in order to calculateĈ (τ ) (Eqn 11) as shown in Figure 4(B-D), and we verify the calculation of C T (τ ) and ∆ (τ ) directly in Supplementary Figure 1.
Appendix D -Constructing Connectivity Matrices with Incomplete Alignment D1 -Constructing Arbitrary Low-Rank Structure with Uniform Misalignment As a concrete example we study a specific form of misalignment in which the alignment matrix is scaled down uniformly across the balance subspace modes. Concretely, for this section we fix the rank D and then in order to construct the structured connectivity, M = 1 √ N UΣV T , we first fix the diagonal of Σ to be non-negative numbers (usually we set them to be all ones for simplicity), and then sample the elements of U independelty from a standard Gaussian distribution. In order to define V, we first construct an arbitrary orthonormal D-by-D matrix,Â, which will be the alignment matrix in the case of full alignment. To constructÂ we generate random pairs of orthonormal vectors of dimension D which each serve as the real and imaginary part of an eigenvector ofÂ. We associate to each pair a uniform random phase constrained to have negative real part. For odd D we add a real eigenvector with eigenvalue −1. Explicitly, we generate orthonormal vectors q k re and q k im for k = 1... D /2 , and θ k ∼ U π 2 , 3π 2 , and then define And for odd D we set z (D+1) /2 to be a random real orthonormal vector and θ(D+1) /2 = −π. Next we define Z to be the matrix of column vectors consisting of z k and Θ to be the diagonal matrix with θ k along the diagonal, and finally we haveÂ where the T here means conjugate-transpose. ThusÂ is a random orthonormal D-by-D matrix with eigenvalues constrained to the left half of the complex plane. This matrix will define the structure of true recurrence within the balance subspace, which will be additionally scaled by a scalar parameter to adjust the extent of alignment as follows.
We construct an N -by-D matrix A ⊥ , with A T ⊥ A ⊥ = N I DxD such that U T A ⊥ = 0. And finally we define V as: Thus the parameter a scales down the alignment matrix uniformly: With this parameterization of the alignment matrix we haver * = − 1 a Σ −1f for the solution to the balance equations. Plugging this into the mean-field equation for ĥ * in the case of Gaussian structure (Eqn 48, we have: This equation is coupled with the mean-field equation for ∆ 0 , Eqn 49, and we solve this pair of equations and then additionally calculate g ef f (Eqn 50) and compare to simulations in Fig S1.

D2 -Constructing Heterogeneous Misalignment
Recall that our dynamic mean-field theory prediction for the total temporal variance in the balance subspace is as derived in Section III of the main text. Note that in the paramaterization of Appendix C1, all D singular values ofV, are given by a. In order to verify our theory in a more generic framework, we fix the left-and right-singular vectors ofV and then construct the D singular values. In order to nevertheless restrict simulations to a single parameter, we adjust the absolute value of the determinant ofV, and then set the sequence of D singular values to decay exponentially from 1 while constraining their product.
Explicitly, given a fixed detV = D−1 k=0 s k , we write We then define s k = exp 2k ls For this parameterization we can write the total temporal variance as This is the theory curve displayed alongside simulation results in Fig 4 (B-D) and Fig S2 (B-E).

Appendix E -Fluctuation Balance under Recurrent Misalignment
We return to the balance subspace dynamics in the general setting of incomplete misalignment (Eqn 5) Recall that this expression for the dynamics ignores the projection of the random connectivity onto the balance subspace, which is of order of magnitude O 1 √ N .
The dynamics of the fluctuations, δĥ ≡ĥ − ĥ , are given by  Figure 3C in the main text, our simulations confirm that O (1) input fluctuations from the orthogonal complement are canceled to leading order by recurrent balance subspace input, yielding small net input to the balance subspace.
This arguement yields a fluctuation balance equation: to leading order.
We employ the SVD ofV in order to further simplify, writingV = LSR T = D k=1 L k s k R T k , where L k and R k are the kth left-and right-singular vectors of the alignment matrix, respectively. We writeη R k ≡ R T kη , for the effective-feedforward input along the kth right-singular vector of the alignment matrix, and δr L k ≡ L T k δr, for the corresponding rate fluctuations along the kth left-singular vector. Then we can re-express te the fluctuation balance requirement as Therefore, for moderate misalignment, i.e. s k ∼ O (1), we expect the fluctuations in the balance subspace, δr, to To findĈ (τ ) we observe that by definition V ⊥ is orthogonal to U, and it is independent of J by assumption. Therefore the fluctuations in the effective-feedforward drive,η, can be approximated as a D-dimensional Gaussian with matrix of cross-correlation functions: where C T (τ ) = r i (t) r i (t + τ ) is the average total autocorrelation funciton of the firing activity. For notational purposes we write C (τ ) ≡ C T (τ ) − C T (∞) = δr i (t) δr i (t + τ ) in the main text (Eqn 11), for δr i = r i − r i . Note that C (τ ) captures the average single-neuron temporal variability, while the long-time autocorrelation, C T (∞), captures the spatial variabity over single-neuron average firing rates.
To simplify the expression for the covariance ofη and derive an expression forĈ (τ ), we recall that V ⊥ = V − UV.
We note that the particular structure of the columns of V ⊥ is unconstrained other than the requirement of being orthogonal to every column of U, but that the D-by-D Gram matrix is fully determined byV: Interestingly, this implies that the effective-feedforward drive is uncorrelated in the basis given by the right-singular vectors ofV, i.e.
We thus find, by applying Eqn 64, that the corresponding balance subspace rate fluctuations projected along the left-singular vectors, L k , are uncorrelated: In other words, the left-singular vectors of the alignment matrix, L k , are the eigenvectors of the covariance,Ĉ, or equivalently, the principal components of the macroscopic fluctuations in the balance subspace. The corresponding eigenvalues, i.e. the variances, are C(τ ) The total variance in the balance subspace , TrĈ (0) = δr T δr , is We can change bases to return to the standard basis of the balance subspace (the columns of U): which is equivalent to the result presented in Eqn 11 in the main text.

Appendix F -The Case of Non-Alignment
In this section we study the case of a complete non-alignment, in which at least one of the singular values of the alignment matrix, As discussed in the previous section, the Gram matrix of V ⊥ = V − UV is determined by byV and given by Therefore we can find some N -by-D matrix U ⊥ whose columns have norm √ N and are all orthogonal to each column of U, and write Thus, we see that in this scenario, with s j ∼ O 1 √ N , there is one macroscopic subspace, u L j ≡ UL j , which does not send any recurrent feedback to the balance subspace, and therefore as we will show, the activity in this subspace is unconstrained by the balance equations.
To see this, we first rescale the balance subspace dynamics (Eqn 60) by Σ −1 and write: x ≡ Σ −1ĥ and y ≡ Σ −1f , yielding dynamics to leading order, where we have momentarily ignored the effective-feedforward input, which we will return to below.
Next, we rotate the rescaled dynamics to the basis of right-singular vectors ofV, x R k ≡ R T k x, while projecting the balance subspace activity to the basis of left-singular vectors,r L k ≡ L T kr , yielding D dynamical equations: Modes with non-zero alignment, s k ∼ O (1), yield a linear balance equation: which is a restatement of the D-dimensional balance equations in the main text, using the identity y R k = R T k Σ −1f (Eqn 6).
For the unaligned mode, however, balance requires small external drive,ŷ R j ∼ O 1 √ N . That is, we require that the strong external drive in the balance subspace have no projection on the jth right-singular veector of the alignment matrix. We note that this requirement is an extension of our assumption throughout thatf is chosen in order to allow a set of balance equations with obtainable firing rates. Therefore, we can write the balance subspace firing rates aŝ where r L j (t) is unconstrained by the balance equations.
We now turn to the fluctuation dynamics of jth mode. We write s j = s √ N and y R j = y √ N , and we have , which in the largne N limit is a Gaussian process with autocorrelation η R j (t) η R j (t + τ ) = C T (τ ) following Eqn 67. This equation predicts O (1) fluctuations in both x R j andr L j , but the self-consistent solution involves all the other modes and is beyond the scope of this work.
In the limit of zero alignment, s = y = 0, the x R j are expected to be simply a Gaussian process with autocorrelation given by Note, however, that the fluctuations inĥ = Σx will have non-trivial impact on the microscopic fluctuations in the orthogonal subspace, and therefore ∆(τ ) can no longer be derived from the DMFT theory above (Eqn 44), except in the limit of σ j 1. As shown in Landau and Sompolinsky [21], in that limit the fluctuations inĥ R j do not impact ∆ (τ ) to leading order, and therefore this regime exhibits a passive coherent chaos with macroscopic fluctuations that are driven in a purely feedforward-like manner by the microscopic chaos in the orthogonal complement via the non-aligned mode of the structured connectivity.

G1 -Degenerate Excitation-Inhibition Balance Example for Unequal Population Size
As in the main text, we consider an E-I network in which the synaptic weight depends only on the pre-synaptic neuron: J EI = J II = J I and J EE = J IE = J E , such that the structured connectivity becomes rank one: σ We consider a network with N I = γN inhibitory neurons, and the remaining N E = (1 − γ) N are excitatory. The structured connectivity is where 1 Nx is the uniform column-vector of length N X .
The alignment between u 0 and v 0 is given by: Inhibition-dominance and network stability will require that J I ≥ 1−γ γ J E . At the critical boundary,v = 0.
External allignment requires that the external drive be uniform, f = 1, and the balance equation yields population average firingr The external drive can be made to compensate for diminished alignment by scaling r 0 ∝vσ = γJ I − (1 − γ) J E . In this case even as inhibition is weakened and alignment decreases, the balanced fixed point remains unchanged to leading-order. In this situation,Ĉ (t) = r (t)r (t + τ ) is given by:

G2 -Heterogeneous In-and Out-Degrees
Here we consider the case of heterogeneity in both out-and in-degrees, with possible correlations between them. We have a single inhibitory population in which each neuron i is randomly connected via K in i incoming connections, and has K out i randomly chosen outgoing connections, with each non-zero synapse having weight − J √ N , where K is the average number of connections per neuron. Such a connectivity structure can be approximated by a deterministic rank-one structure given by We define the relative in/out degrees as k α i ≡ K α i K , and the mean-square of the relative in/out degrees is k 2 α , for α ∈ {in, out}. Then we can write where p ≡ K N .
The scalar alignment in this case isv where c = k in i − 1 (k out i − 1) is the covariance of the relative in-and out-degrees. We find that if the in-degree and out-degrees are uncorrelated, thenv = − 1 k 2 in k 2 out and the extent of alignment decreases with increasing breadth of the degree distributions. Correlations between inand out-degrees increase the alignment and in the extreme case of fully correlated degrees, the absolute alignment remains large, and depends only on the relative breadth of the two distributions, for examplev = −1 for fully correlated degree distributions with identical variances.
External alignment requires that f = u = k in k 2 in , similar to [37]. The balance equation giveŝ We find that if in-and out-degrees are anticorrelated the balance-rates will be driven up.
Note that in this setting the balance subspace is defined by the in-degrees:r = 1 For correlated in-and out-degrees we find: So that positive correlations between in-and out-degrees decrease shared fluctuations while negative correlations amplify them. Simulations of the uniform misalignment parameterization show that misalignment of the structured connectivity increases fluctuations in the balance subspace by an order of magnitude. The logarithm of the standard deviation ofr (t) is plotted against the logarithm of network size. In the case of full alignment (a = 1) the slope is almost exactly −1, indicating that the varianceĈ (0) ∼ 1 N 2 . On the other hand, even mild misalignment (a = 0.8) yields a slope of approximately −0.5, which is consistent with our theory predictingĈ (0) ∼ 1 N for misaligned structural connectivity. Data points are averages over 10 realizations, black dotted lines are linear fits. (B)-(D) In the case of Gaussian structured connectivity, the dynamic mean-field theory for the fluctuations in the orthogonal subspace depend only on the norm of the firing rates in the balance subspace. (B) As described in Appendix V, we vary the determinant of the alignment matrixV while keeping the predicted norm of the balance firing rates fixed. This figure confirms that indeed, in our simulations both the firing rates and the dynamical variables in the balance subspace (r in blue andĥ in red, respectively) remain constant and very near the dynamic mean-field theory prediction throughout the parameter range. These simulations correspond to Figure 4B. Shaded regions display the standard deviation over 20 random realizations ofV with fixed singular values, s k (C) We vary the predicted norm of the balance firing rate while fixing the singular values ofV. Bothr in blue andĥ in red are well-predicted by DMFT. (D) The autocorrelation of the microscopic degrees of freedom in the orthogonal complement, ∆ (τ ) ≡ h ⊥ i (t) h ⊥ i (t + τ ) , displayed for the same four values of r as in (C), together with the corresponding DMFT prediciton. (E) Same as (D) but for the autocorrelation of the singlue neuron firing rates, C (τ ) ≡ ri (t) ri (t + τ ) . N = 10000 accept where otherwise mentioned.