Modeling the Influence of Data Structure on Learning in Neural Networks: The Hidden Manifold Model

Understanding the reasons for the success of deep neural networks trained using stochastic gradient-based methods is a key open problem for the nascent theory of deep learning. The types of data where these networks are most successful, such as images or sequences of speech, are characterized by intricate correlations. Yet, most theoretical work on neural networks does not explicitly model training data or assumes that elements of each data sample are drawn independently from some factorized probability distribution. These approaches are, thus, by construction blind to the correlation structure of real-world datasets and their impact on learning in neural networks. Here, we introduce a generative model for structured datasets that we call the hidden manifold model. The idea is to construct high-dimensional inputs that lie on a lower-dimensional manifold, with labels that depend only on their position within this manifold, akin to a single-layer decoder or generator in a generative adversarial network. We demonstrate that learning of the hidden manifold model is amenable to an analytical treatment by proving a “Gaussian equivalence property” (GEP), and we use the GEP to show how the dynamics of two-layer neural networks trained using one-pass stochastic gradient descent is captured by a set of integro-differential equations that track the performance of the network at all times. This approach permits us to analyze in detail how a neural network learns functions of increasing complexity during training, how its performance depends on its size, and how it is impacted by parameters such as the learning rate or the dimension of the hidden manifold.

Popular Summary

Deep neural networks—machine-learning algorithms that mimic the human brain—have reached impressive levels of performance on tasks ranging from image classification to natural language processing. Yet from a theoretical point of view, the reasons for their success remain unclear. Closing this gap between theory and practice might help address open problems in the practice of machine learning. A key challenge is understanding how the structure of real-world data impacts learning in neural networks. Here, we address this question by introducing a model for structured data sets that we call the “hidden manifold model.” This model provides a more realistic framework in which the dynamics and performance of machine learning can be analyzed.

The hidden manifold model is a type of statistical model for generating high-dimensional data from low-dimensional inputs. We show experimentally that training two-layer neural networks on such data reproduces some effects seen when training neural networks on practical data sets for image classification. We also derive a “Gaussian equivalence principle,” which allows us to study the dynamics and the performance of two-layer neural networks analytically and in detail.

Our work provides a model and new analytical techniques to analyze learning when inputs have nontrivial correlations, thus paving the way for a systematic study of how learning is shaped by data structure.

Figure 1

We illustrate the notion of a hidden manifold in input space using CIFAR10 example images. Each black point indicates a possible input in a high-dimensional input space ${\mathbb{R}}^N$. Most points in this space cannot be interpreted as images at all; however, those points that can be interpreted as real images tend to concentrate on a lower-dimensional manifold, here sketched as a two-dimensional curved surface in a three-dimensional space. The intrinsic dimension $D$ of these lower-dimensional manifolds has been measured numerically [12, 13, 14, 15].

Figure 2

The hidden manifold model proposed here is a generative model for structured datasets, where inputs $x$ [Eq. (4)] (blue and green balls) concentrate on a lower-dimensional manifold in input space (yellow surface). Their label $y^{*}$ is a function of their position on the manifold; here, we show the setup of a classification task with two classes $y^{*}=\pm 1$. In our analysis, the labels are generated according to Eq. (5).

Figure 3

The analytical description of the hidden manifold generalization dynamics matches experiments even at moderate system size. We plot the time evolution of the generalization error ${\epsilon }_g(\alpha )$ (a) and the order parameters $R^{km}$ (b), $Q^{k\ell }$ (c) and second-layer weights $v^k$ (d) obtained by integration of the ODEs (solid lines) and from a single run of SGD (crosses). Parameters: $g(x)=\mathrm{erf}(x/\sqrt{2})$, $N=10000$, $D=100$, $M=2$, $K=2$, $\eta =0.2$, and ${\tilde{v}}^m=1$.

Figure 4

The ODE analysis is asymptotically correct for nonrandom feature matrices $F$. We plot the time evolution of the generalization error ${\epsilon }_g$ obtained by integration of the ODEs (solid lines) and from a single run of SGD (2) (crosses) for two different matrices $F$: (i) Elements $F_{ir}$ are drawn IID from the standard normal distribution (blue); (ii) $F$ is a Hadamard matrix [70]. $f(x)=\mathrm{sgn}(x)$, $g(x)=\tilde{g}(x)=\mathrm{erf}(x/\sqrt{2})$, $N=1023$, $D=1023$, $M=2$, $K=2$, $\eta =0.2$, and ${\tilde{v}}^m=1$.

Figure 5

The impact of the latent dimension $\delta \equiv D/N$. We plot the final test error ${\epsilon }_g^{*}$ of sigmoidal students trained on the hidden manifold model with three different intrinsic dimensions $D$ as a function of $\delta =D/N$, where $N$ is the input dimension. The average is taken over five runs. The solid lines are the asymptomatic theoretical predictions derived in Sec. 4c1. The shaded bars indicate experimental estimates [12, 13, 14, 15] for $\delta$ for the CIFAR10 dataset (left) and the MNIST dataset (right). $f(x)=\mathrm{sgn}(x)$, $g(x)=\tilde{g}(x)=\mathrm{erf}(x/\sqrt{2})$, $M=K=2$, $\eta =0.2$, and ${\tilde{v}}^m=1$.

Figure 6

The impact of the learning rate $\eta$. We plot the final test error ${\epsilon }_g^{*}$ of sigmoidal students trained on the hidden manifold model for a range of learning rates $\eta$ for sigmoidal networks with $K=M=2$ (blue lines) and $K=M=6$ (green lines). We repeat the experiments for two values of $D$, choosing $N$ such that $\delta =D/N=0.01$. Inset: Generalization dynamics during training ($K=M=2$). Parameters: $f(x)=\mathrm{sgn}(x)$, $g(x)=\tilde{g}(x)=\mathrm{erf}(x/\sqrt{2})$, $\delta =0.01$, ${\tilde{v}}^m=1$, and $K=M$.

Figure 7

(a) Asymptotic generalization for online learning of a student with $K=ZM$ hidden nodes learning from a teacher with $M=1$ (blue lines), $M=2$ (violet lines), and $M=4$ (green lines) hidden nodes, respectively. The dotted, dash-dotted, and dashed lines correspond to $D=50$, 100, and 200, respectively. Error bars indicate two standard deviations over five runs. The solid line is the theoretical prediction obtained for $\tilde{t}=0$. (b),(c) Teacher-student overlap $R^{km}$ [Eq. (17)], second-layer weights $v^k$, and the normalized overlap $v^k{R}^{km}$ obtained in two simulations used in the left plot with $M=2$ and $K=6$, starting from different initial conditions, all other things being equal. Parameters: In all plots, $f(x)=\mathrm{sgn}(x)$, $g(x)=\mathrm{erf}(x/\sqrt{2})$, $\eta =0.2$, ${\tilde{v}}^m=1$, $\delta =0.01$, and $\eta =0.2$.

Figure 8

Two-layer neural networks learn functions of increasing complexity. We plot the generalization error of sigmoidal two-layer networks with an increasing number of hidden nodes $K$ during a single run of online learning with the hidden manifold model with $\delta =0.05$, $D=25$, $M=10$, and ${\tilde{v}}^m=1/M$ on the HMM (a) and when trained on odd-versus-even digit classification on MNIST, averaged over ten runs (b). Error bars indicate two standard deviations. For details, see Secs. 5a and pp4. $g(x)=\mathrm{erf}(x/\sqrt{2})$, $\eta =0.2$, and $N=784$. (b) Batch size 32.

Figure 9

Neural networks have different memorization patterns for random and structured datasets. We plot the memorability of training images, i.e., the frequency with which an image from the training set is correctly classified by a neural network after training for only a single epoch. In (a) and (b), we reproduce the result of Arpit et al. [86] for CIFAR10 (full blue line). This curve demonstrates the existence of hard and easy examples which are never, or always, classified correctly. (a) shows that this property disappears in all models when the labels are reshuffled (dashed lines). The insets indicate the training accuracy after training, using circles for randomized datasets and squares for unmodified data sets. (b) shows that these hard and easy examples also exist in the structured data models, TeacherS (green line) and the HMM (red line), but not in the unstructured Gaussian one (orange line).

Figure 10

Two-layer sigmoidal neural networks learn functions of increasing complexity on different datasets. We plot the mean-squared error as a function of training time for sigmoidal networks with an increasing hidden layer when trained on three different datasets. The curves are obtained by averaging ten different runs, starting from different initial weights. Error bars indicate one standard deviation. For all plots, $g(x)=\mathrm{erf}(x/\sqrt{2})$, Gaussian initial weights with standard deviation ${10}^{-3}$, and batch size 32.

Figure 11

Two-layer ReLU neural networks also learn functions of increasing complexity and show more run-to-run variance. We plot the mean-squared error as a function of training time for sigmoidal networks with an increasing number of nodes $K$ when trained on three different datasets. For all plots, $g(x)=\max (0,x)$, Gaussian initial weights with standard deviation ${10}^{-3}$, and batch size 32. For online learning, we choose a teacher with $\tilde{g}(x)=\max (x,0)$, $M=10$, and ${\tilde{v}}^m=1/M$.