Linear Classification of Neural Manifolds with Correlated Variability

Understanding how the statistical and geometric properties of neural activity relate to performance is a key problem in theoretical neuroscience and deep learning. Here, we calculate how correlations between object representations affect the capacity, a measure of linear separability. We show that for spherical object manifolds, introducing correlations between centroids effectively pushes the spheres closer together, while introducing correlations between the axes effectively shrinks their radii, revealing a duality between correlations and geometry with respect to the problem of classification. We then apply our results to accurately estimate the capacity of deep network data.

Figure 1

(a) Neural manifolds arising from different instances of $P=3$ object classes (bird, vase, and cloud [26, 27, 28, 29]), with $N=3$ neurons. We parametrize the manifolds in terms of a centroid $u_0^{\mu }$, axes $u_{i>0}^{\mu }$, and shape vectors ${\mathcal{S}}^{\mu }$, determining which linear combinations of the axes lie within the manifold. (b) Neural manifolds with correlations in their centroids. (c) Neural manifolds with fully correlated axes. In all three images, different colors correspond to different object class manifolds.

Figure 2

The effect of correlations on the optimization landscape for $V_0$. First column: Two manifolds with (a) uncorrelated and (c) correlated centroids arising from the activations of two neurons, $n_1$ and $n_2$. Second column: Level curves for $\parallel V-T{\parallel }_{y,C}^2$, given fixed $y$ and $V_{i>0}$ for the (b) uncorrelated and (d) correlated manifolds. Shaded regions correspond to areas where the constraint is satisfied—i.e., sections of the set $\mathcal{A}$ in Eq. (3). Clearly, correlations warp the optimization landscape along the eigenvectors $P_1$, $P_2$ of the centroid covariance matrix with off-diagonal sign flips $y^{\mu }{y}^{\nu }{C}_{\nu ,0}^{\mu ,0}$.

Figure 3

Comparison of three different capacity estimators, including the low-rank approximation of Ref. [16], to the numerically estimated ground truth simulation capacity (blue triangles) described in Ref. [40]. The correlation intensity denotes the magnitude of the off-diagonal correlations—see the Supplemental Material for more details [37].

Figure 4

The capacity for correlated spheres. (a) Visual demonstration of spherical manifolds with low-rank axis and centroid correlations. (b) The zero-margin capacity as a function of only the input ratio $r\sqrt{1-\lambda }/(r_0\sqrt{1-\psi })$. Points represent averages over five random sphere samplings, and the solid line represents the theoretical prediction. For each experiment, we fix three of the four parameters and vary the remaining one to obtain a fixed value of the ratio.

Figure 5

Comparison of the low-rank approximation (yellow dashed line) [16], ${\alpha }_M$ (red dotted line) [15], and our ${\alpha }_{cor}$ calculation (green solid line) to the ground truth simulation capacity (blue triangles) [17] on data manifolds arising from the ResNet50 artificial neural network architecture trained using SimCLR on the ImageNet dataset [22, 43].