Spatially heterogeneous learning by a deep student machine

Despite spectacular successes, deep neural networks (DNNs) with a huge number of adjustable parameters remain largely black boxes. To shed light on the hidden layers of DNNs, we study supervised learning by a DNN of width $N$ and depth $L$ consisting of $NL$ perceptrons with $c$ inputs by a statistical mechanics approach called the teacher-student setting. We consider an ensemble of student machines that exactly reproduce $M$ sets of $N$-dimensional input/output relations provided by a teacher machine. We show that the statistical mechanics problem becomes exactly solvable in a high-dimensional limit which we call a “dense limit”: $N\gg c\gg 1$ and $M\gg 1$ with fixed $\alpha =M/c$ using the replica method developed by Yoshino [SciPost Phys. Core 2, 005 (2020)] In conjunction with the theoretical study, we also study the model numerically performing simple greedy Monte Carlo simulations. Simulations reveal that learning by the DNN is quite heterogeneous in the network space: configurations of the teacher and the student machines are more correlated within the layers closer to the input/output boundaries, while the central region remains much less correlated due to the overparametrization in qualitative agreement with the theoretical prediction. We evaluate the generalization error of the DNN with various depths $L$ both theoretically and numerically. Remarkably, both the theory and the simulation suggest that the generalization ability of the student machines, which are only weakly correlated with the teacher in the center, does not vanish even in the deep limit $L\gg 1$, where the system becomes heavily overparametrized. We also consider the impact of the effective dimension $D(\le N)$ of data by incorporating the hidden manifold model [Goldt, Mézard, Krzakala, and Zdevorová, Phys. Rev. X 10, 041044 (2020)] into our model. Replica theory implies that the loop corrections to the dense limit, which reflect correlations between different nodes in the network, become enhanced by either decreasing the width $N$ or decreasing the effective dimension $D$ of the data. Simulation suggests that both lead to significant improvements in generalization ability.

Figure 1

Schematic picture of the multilayer perceptron network of depth $L$ and width $N$. In this example, the depth is $L=4$. Each arrow represents an $M$-component vector spin ${\mathbf{S}}_i=(S_i^1,S_i^2,...,S_i^M)$ with its component $S_i^{\mu }=\pm 1$ representing the state of a “neuron” in the $\mu \mathrm{th}$ pattern.

Figure 2

Schematic picture of the phase space of machines: the gray box represents the set of all machines which can be generated varying the parameters (e.g., synaptic weights) given a network structure. The yellow region represents the subspace in which machines agree with the teacher's machine for a given $M$ set of training data. Liquid phase: if the number of training data $M$ is small, the subspace is so large that the machines are typically widely separated and their mutual overlap $Q$ is typically zero. Crystalline phase: $M$ is large enough that machines have a finite overlap $Q$ with respect to each other.

Figure 3

A loop of interactions in a DNN extended over three layers, through three perceptrons and four bonds.

Figure 4

Schematic pictures of the teacher-student setting.

Figure 5

Variables (green) associated with a perceptron $\blacksquare$ which changes sign by the flip of the gauge variable ${\sigma }_{\blacksquare }\to -{\sigma }_{\blacksquare }$.

Figure 6

Schematic picture of the hidden manifold model.

Figure 7

Spatial profile of the order parameter obtained by solving the replica-symmetric saddle-point equations. Top: the overlap of spins (neurons), and bottom: the overlap of bonds (synaptic weights). Here $L=20$. Different lines corresponds to $\alpha =16-{10}^3$ with equal spacing in $ln\alpha =0.23\cdots$ .

Figure 8

Order parameters and generalization error obtained by solving the replica-symmetric saddle-point equations. In the panels on the first and second rows, overlaps of spins $q\left(l\right)$ (filled symbols) and $Q\left(l\right)$ bonds (open symbols) are shown. In the bottom row, the generalization error $\epsilon$ is shown.

Figure 9

Learning curves of DNN with various depths $L$ obtained by solving the replica-symmetric saddle-point equations.

Figure 10

Schematic picture of the closed and unclosed loop at the boundary.

Figure 11

Relaxation of the loss function in learning [annealing with $t_{\max }={10}^4$ (MCS)] observed by the MC simulation. In all cases, $N=10$. (a) Various $\alpha =M/c$ with $L=10$ and $c=5$. (b) Various $L$ with $\alpha =M/c=4$ and $c=5$. (c) Various $N$ with $L=10,\alpha =4$, and $c=5$. (d) The same as (c) but with $c=10$. The unit of time $t$ is 1 (MCS).

Figure 12

Time evolution of simple student-student overlap $q\left(l\right)$ (filled symbols) and teacher-student overlap $r\left(l\right)$ (open symbols) observed in MC simulations of the unlearning protocol. Here $N=10,\alpha =4,c=5$. Panel (a) shows data at $t=1,2,4,8,10,20,40,80$. Panels (b) and (c) show the simple student-student overlap $q\left(l\right)$ and simple teacher-student $r\left(l\right)$ overlap, respectively.

Figure 13

Spatial profile of the normalized squared teacher-student overlaps $r_2\left(l\right)$ and student-student overlaps $q_2\left(l\right)$ [see Eq. (36)] for training (a),(b) and test (c),(d), and time evolution of the simple teacher-student overlap $r\left(L\right)$ in the output layer $\left(l=L\right)$ for test (e),(f) [See Eq. (42)]. All data are obtained by the MC simulations. In panels (a), (c), and (e), data of $q_2\left(l\right)$ (open symbols) and $r_2\left(l\right)$ (filled symbols) of learning/unlearning are represented by red/blue points $\left(\alpha =4\right)$. Panels (a) and (c) show the normalized squared overlaps at various times $t=1,10,100,1000,10000$ (increasing along the arrows) at each layer. Panel (e) shows the time evolution of the simple teacher-student overlap $r\left(L\right)$ at the output layer $\left(l=L\right)$ for the test data. Panels (b) and (d) show the normalized squared teacher-student overlap for unlearning (open symbols)/learning (filled symbols) at $\alpha =2,4,8$ at $t={10}^4$ (MCS). Panel (f) show the time evolution of the simple teacher-student overlap $r\left(L\right)$ at the output layer $\left(l=L\right)$ for the test data, obtained by unlearning (open symbols)/learning (filled symbols) protocols with $\alpha =2,4,8,16,32$. Here $N=10,L=10$, and $c=5$ for all data.

Figure 14

Finite $N$ effect and finite $c$ effect: the asymmetry becomes smaller as $N$ increases for unlearning (open symbols)/learning (filled symbols). $N=10,20,30,40,80$ with $\alpha =1.0$ and (left) $c=5$, (right) $c=10$. All data are obtained by MC simulations of $t={10}^4$ (MCS).

Figure 15

Spatial profile of the normalized squared overlaps and the generalization error in systems with various widths $N$ (top panels) and hidden dimension $D(\le N)$ (bottom panels) obtained by MC simulations. In the top panels, data obtained by both learning (filled symbols) and unlearning (open symbols) are shown. Panels (a) and (b) show the normalized squared overlaps for training, while (c) and (d) show those for the test. Panels (e) and (f) show the generalization error $\epsilon$ [see Eq. (43)]. In panel (e) we also show the generalization error $\epsilon$ obtained by the theory in the dense limit: $N\to \infty$ followed by $N\to \infty$ [see Eq. (12) and Fig. 9]. In panels (a)–(d) data with $L=5,10,20$ are shown. In panels (a), (c), and (e) data with $N=8,10,20$ are shown. In panels (b), (d), and (f) data with $D=4,6,8,10$ are shown. In all cases, $\alpha =4,c=5$, and $t={10}^4$.

Figure 16

Some examples of the activation function $f\left(h\right)$ and the associated confining potential $U\left(S\right)$. Here $u=1$ for the piecewise linear function.

Figure 17

Graphical representation of a contribution to ${\tilde{G}}_1$ (and ${\tilde{F}}_1)$.

Figure 18

A contribution to $G_2$ which is one-line reducible.

Figure 19

A loop of interactions in a DNN extended over three layers, through three perceptrons and four bonds.

Figure 20

More extended loop.