Classification and Geometry of General Perceptual Manifolds

Perceptual manifolds arise when a neural population responds to an ensemble of sensory signals associated with different physical features (e.g., orientation, pose, scale, location, and intensity) of the same perceptual object. Object recognition and discrimination require classifying the manifolds in a manner that is insensitive to variability within a manifold. How neuronal systems give rise to invariant object classification and recognition is a fundamental problem in brain theory as well as in machine learning. Here, we study the ability of a readout network to classify objects from their perceptual manifold representations. We develop a statistical mechanical theory for the linear classification of manifolds with arbitrary geometry, revealing a remarkable relation to the mathematics of conic decomposition. We show how special anchor points on the manifolds can be used to define novel geometrical measures of radius and dimension, which can explain the classification capacity for manifolds of various geometries. The general theory is demonstrated on a number of representative manifolds, including ${\ell }_2$ ellipsoids prototypical of strictly convex manifolds, ${\ell }_1$ balls representing polytopes with finite samples, and ring manifolds exhibiting nonconvex continuous structures that arise from modulating a continuous degree of freedom. The effects of label sparsity on the classification capacity of general manifolds are elucidated, displaying a universal scaling relation between label sparsity and the manifold radius. Theoretical predictions are corroborated by numerical simulations using recently developed algorithms to compute maximum margin solutions for manifold dichotomies. Our theory and its extensions provide a powerful and rich framework for applying statistical mechanics of linear classification to data arising from perceptual neuronal responses as well as to artificial deep networks trained for object recognition tasks.

Popular Summary

Our brains have the ability to accurately classify objects that we see despite huge variability in parameters such as illuminations, pose, and background features. Recent advances in machine learning have resulted in neural networks with similar abilities. However, researchers lack a mathematical understanding of how biological and artificial systems achieve such remarkable recognition accuracy. Here, we show how statistical mechanical theory can be used to explain fundamental principles underlying the ability of a neural circuit to discriminate objects in the face of enormous physical variability.

We geometrically model the variability in the neural representation of a particular object as a manifold. The number of manifolds that can be classified at a particular stage in the network grows in proportion to the dimensionality of the neural representation, but the proportionality depends on the shape of the manifolds. Our theory can be used to analyze the structure of the manifold representations as they transform and propagate through the network, leading ultimately to successful classification.

Our theory describes the shape of the neural manifolds using geometrical measures that are able to predict when a randomly labeled set of manifolds can be separated. These measures lead to quantities that characterize manifolds with arbitrary geometry and can be computed efficiently; we use them to analyze prototypical manifold models of neural responses.

Our work provides a new theoretical framework to understand and analyze the representations formed by hierarchical neural networks, and it may lead to novel insights into how perceptual systems efficiently code and process sensory information.

Figure 1

Perceptual manifolds in neural state space.(a) Firing rates of neurons responding to images of a dog shown at various orientations $\theta$ and scales $s$. The response to a particular orientation and scale can be characterized by an $N$-dimensional population response. (b) The population responses to the images of the dog form a continuous manifold representing the complete set of invariances in the ${\mathbb{R}}^N$ neural activity space. Other object images, such as those corresponding to a cat in various poses, are represented by other manifolds in this vector space.

Figure 2

Model of manifolds in affine subspaces. $D=2$ manifold embedded in ${\mathbb{R}}^N$. $\mathbf{c}$ is the orthogonal translation vector for the affine space, and ${\mathbf{x}}_{\mathbf{0}}$ is the center of the manifold. As the scale $r$ is varied, the manifold shrinks to the point ${\mathbf{x}}_{\mathbf{0}}$ or expands to fill the entire affine space.

Figure 3

Conic decomposition of $-\overrightarrow{T}=-\overrightarrow{V}+\lambda \tilde{S}$ at (a) zero margin $\kappa =0$ and (b) nonzero margin $\kappa >0$. Given a random Gaussian $\overrightarrow{T}$, $\overrightarrow{V}$ is found such that $\parallel \overrightarrow{V}-\overrightarrow{T}\parallel$ is minimized, while $-\overrightarrow{V}$ is on the polar cone ${\mathcal{S}}_{\kappa }^{\circ }.\lambda \tilde{S}$ is in $\mathrm{cone}(\mathcal{S})$ and $\tilde{S}$ is the anchor point, a projection on the convex hull of $\mathcal{S}$.

Figure 4

Determining the anchor points of the Gaussian distributed vector $(\overrightarrow{t},t_0)$ onto the convex hull of the manifold, denoted as $\tilde{s}(\overrightarrow{t},t_0)$. Here, we show, for the same vector $\overrightarrow{t}$, the change in $\tilde{s}(\overrightarrow{t},t_0)$ as $t_0$ decreases from $+\infty$ to $-\infty$. (a) $D=2$ strictly convex manifold. For sufficiently positive $t_0$, the vector $-\overrightarrow{t}$ obeys the constraint $g(\overrightarrow{t})+t_0>\kappa$; hence, $-\overrightarrow{t}=-\overrightarrow{v}$, and the configuration corresponds to an interior manifold (support dimension $k=0$). For intermediate values where $(t_{\text{touch}}>t_0-\kappa >t_{\mathrm{fs}})$, $(-\overrightarrow{t},-t_0)$ violates the above constraints, and $\tilde{s}$ is a point on the boundary of the manifold that maximizes the projection on the manifold of a vector $(-\overrightarrow{v},-v_0)$ that is the closest to $(-\overrightarrow{t},-t_0)$ and obeys $g(\overrightarrow{v})+t_0+\lambda =\kappa$. Finally, for larger values of $-t_0$, $\overrightarrow{v}=0$, and $\tilde{s}$ is a point in the interior of the manifold in the direction of $-\overrightarrow{t}$ (fully supporting with $k=3$). (b) $D=2$ square manifold. Here, in both the interior and touching regimes, $\tilde{s}$ is a vertex of the square. In the fully supporting regime, the anchor point is in the interior and collinear to $-\overrightarrow{t}$. There is also a partially supporting regime when $-t_0$ is slightly below $t_{\mathrm{fs}}$. In this regime, $-\overrightarrow{v}$ is perpendicular to one of the edges and $\tilde{s}$ resides on an edge, corresponding to manifolds whose intersections with the margin planes are edges ($k=2$).

Figure 5

Gaussian anchor points, with mapping from $\overrightarrow{t}$ to points ${\tilde{s}}_g(\overrightarrow{t})$, showing the relation between $-\overrightarrow{t}$ and the point on the manifold that touches a hyperplane orthogonal to $\overrightarrow{t}$. $D=2$ manifolds shown are (a) circle, (b) ${\ell }_2$ ellipsoid, (c) polytope manifold. In (c), only for values of $\overrightarrow{t}$ of measure zero (when $\overrightarrow{t}$ is exactly perpendicular to an edge) will ${\tilde{s}}_g(\overrightarrow{t})$ lie along the edge; otherwise it coincides with a vertex of the polytope. In all cases, ${\tilde{s}}_g(\overrightarrow{t})$ will be in the interior of the convex hulls only for $\overrightarrow{t}=0$. Otherwise, it is restricted to their boundary.

Figure 6

Distribution of $\parallel \tilde{s}\parallel$ (norm of manifold anchor vectors) and $\theta =\angle (-\overrightarrow{t},\tilde{s})$ for $D=2$ ellipsoids. (a) Distribution of $\tilde{s}$ for an ${\ell }_2$ ball with $R=5$. The Gaussian geometry is peaked at $\parallel \tilde{s}\parallel =r$ (blue), while $\parallel \tilde{s}\parallel <r$ has nonzero probability for the manifold anchor geometry (red). (b),(c) $D=2$ ellipsoids, with radii $R_1=5$ and $R_2=\frac{1}{2}{R}_1$. (b) Distribution of $\parallel \tilde{s}\parallel$ for Gaussian geometry (blue) and anchor geometry (red). (c) Corresponding distribution of $\theta =\angle (-\overrightarrow{t},\tilde{s})$.

Figure 7

Bimodal ${\ell }_2$ ellipsoids. (a) Ellipsoidal radii $R_i$ with $r=1$. (b) Classification capacity as a function of scaling factor $r$: full mean-field theory capacity (blue lines); approximation of the capacity given by equivalent ball with $R_M$ and $D_M$ (black dashed lines); simulation capacity (circles), averaged over five repetitions, measured with 50 dichotomies per repetition. (c) Manifold dimension $D_M$ as a function of $r$. (d) Manifold radius $R_M$ relative to $r$. (e) Fraction of manifolds with support dimension $k$ for different values of $r$: $k=0$ (interior), $k=1$ (touching), $k=D+1$ (fully supporting). (e1) Small $r=0.01$, where most manifolds are interior or touching. (e2) $r=1$, where most manifolds are in the touching regime. (e3) $r=100$, where the fraction of fully supporting manifolds is 0.085, predicted by $H(-t_{\mathrm{fs}})=H(\sqrt{\sum _i{R}_i^{-2}})$ [Eq. (39)].

Figure 8

Ellipsoids with radii computed from realistic image data. (a) SVD spectrum $R_i$, taken from the readout layer of GoogLeNet from a class of ImageNet images with $N=1024$ and $D=1023$. The radii are scaled by a factor $r$: $R_i^{\prime }=r{R}_i$ for (b) Classification capacity as a function of $r$: full mean-field theory capacity (blue lines); approximation of the capacity as that of a ball with $R_M$ and $D_M$ from the theory for ellipsoids (black dashed lines); simulation capacity (circles), averaged over 5 repetitions, measured with 50 random dichotomies per each repetition. (c) Manifold dimension as a function of $r$. (d) Manifold radius relative to the scaling factor $r$, $R_M/r$ as a function of $r$.

Figure 9

Separability of ${\ell }_1$ balls. (a) Linear classification capacity of ${\ell }_1$ balls as a function of radius $r$ with $D=100$ and $N=200$: MFT solution (blue lines), spherical approximation (black dashed lines), full numerical simulations (circles). Inset: Illustration of ${\ell }_1$ ball. (b) Manifold radius $R_M$ relative to the actual radius $r$. (c) Manifold dimension $D_M$ as a function of $r$. In the small $r$ limit, $D_M$ is approximately $2\log D$, while, in large $r$, $D_M$ is close to $D$, showing how the solution is orthogonal to the manifolds when their sizes are large. (d1)–(d3) Distribution of support dimensions: (d1) $r=0.001$, where most manifolds are either interior or touching; (d2) $r=2.5$, where the support dimension has a peaked distribution; (d3) $r=100$, where most manifolds are close to being fully supporting.

Figure 10

Linear classification of $D$-dimensional ring manifolds, with uniform $R_n=\sqrt{(2/D)}r$. (a) Classification capacity for $D=20$ as a function of $r$ with $m=200$ test samples (for details of numerical simulations, see SM [16], Sec. S9): mean-field theory (blue lines), spherical approximation (black dashed lines), numerical simulations (black circles). Inset: Illustration of a ring manifold. (b) Manifold dimension $D_M$, which shows $D_M\sim D$ in the large $r$ limit, showing the orthogonality of the solution. (c) Manifold radius relative to the scaling factor $r$, $(R_M/r)$, as a function of $r$. The fact that $(R_M/r)$ becomes small implies that the manifolds are “fully supporting” to the hyperplane, showing the small radius structure. (d) Manifold dimension grows with affine dimension, $D$, as $2\log D$ for small $r=(1/2\sqrt{2\log D})$ in the scaling regime. (e1)–(e3) Distribution of support dimensions. (d1) $r=0.01$, where most manifolds are either interior or touching; (d2) $r=20$, where the support dimension has a peaked distribution, truncated at $(D/2)$; (d3) $r=200$, where most support dimensions are $(D/2)$ or fully supporting at $D+1$.

Figure 11

(a),(b) Classification of ${\ell }_2$ balls with sparse labels. (a) Capacity of ${\ell }_2$ balls as a function of $\overline{f}=f(1+r^2)$, for $D=100$ (red) and $D=10$ (blue) mean-field theory results. Overlaid black dotted lines represent an approximation interpolating between Eqs. (55) and (57) (details in SM [16], Sec. S9). (b),(c) Classification of general manifolds with sparse labels. (b) Capacity of ${\ell }_1$ ellipsoids with $D=100$, where the first 10 components are equal to $r$, and the remaining 90 components are $\frac{1}{2}r$, as a function of $\overline{f}=f(1+R_g^2)$. $r$ is varied from ${10}^{-3}$ to $10^3$: numerical simulations (circles), mean-field theory (lines), spherical approximation (dotted lines). (c) Capacity of ring manifolds with a Gaussian falloff spectrum, with $\sigma =0.1$ and $D=100$ (details in SM [16]). The Fourier components in these manifolds have a Gaussian falloff, i.e., $R_n=A\exp (-\frac{1}{2}{[2\pi (n-1)\sigma ]}^2)$.

Figure 12

Manifold configurations and geometries for classification of ${\ell }_1$ ellipsoids with sparse labels, analyzed separately in terms of the majority and minority classes. The radii for ${\ell }_1$ ellipsoids are $R_i=r$ for $1\le i\le 10$ and $R_i=\frac{1}{2}r$ for $11\le i\le 100$, with $r=10$. (a) Histogram of support dimensions for moderate sparsity $f=0.1$: minority (blue) and majority (red) manifolds. (b) Histogram of support dimensions for high sparsity $f={10}^{-3}$: minority (blue) and majority (red) manifolds. (c) Manifold dimension as a function of $\overline{f}=f(1+R_g^2)$ as $f$ is varied: minority (blue) and majority (red) manifolds. (d) Manifold radius relative to the scaling factor $r$ as a function of $\overline{f}$: minority (blue) and majority (red) manifolds.