Soft mode in the dynamics of over-realizable online learning for soft committee machines

Overparametrized deep neural networks trained by stochastic gradient descent are successful in performing many tasks of practical relevance. One aspect of overparametrization is the possibility that the student network has a larger expressivity than the data generating process. In the context of a student-teacher scenario, this corresponds to the so-called over-realizable case, where the student network has a larger number of hidden units than the teacher. For online learning of a two-layer soft committee machine in the over-realizable case, we present evidence that the approach to perfect learning occurs in a power-law fashion rather than exponentially as in the realizable case. All student nodes learn and replicate one of the teacher nodes if teacher and student outputs are suitably rescaled and if the numbers of student and teacher hidden units are commensurate.

Figure 1

Dependence of the generalization error in the realizable scenario, $K=M$, on the normalized number of examples $\tilde{\alpha }=\alpha \eta$ (“time”), computed to $O\left(\eta \right)$. Different colors indicate different widths $M$ of the teacher. The height of the plateau converges monotonously with $M$. The initialization of the ordinary differential equations is given by ${\chi }^2={10}^{-4}$, where ${\chi }^2$ is the variance of initial student vector elements.

Figure 2

Evolution of the generalization error in realizable and over-realizable scenarios, $K\ge M$, as a function of $\tilde{\alpha }=\alpha \eta$, computed to $O\left(\eta \right)$ for $M=2$. The plateau height is independent of $K$, while the plateau length is proportional to $K$. The inset shows that a rescaling of the learning rate with $K$ leads to a collapse of the curves with different $K$. The initialization of the ordinary differential equations is given by ${\chi }^2={10}^{-4}$.

Figure 3

Evolution of the student-teacher overlap $R_{in}$ as a function of $\tilde{\alpha }=\alpha \eta$, computed to $O\left(\eta \right)$ for $M=2$, $K=4$. Students 1 and 2 imitate teacher 1 after the specialization transition, while students 3 and 4 imitate teacher 2, indicated by the $R_{in}$ being close to one. The remaining overlaps tend to zero. The initialization of the ordinary differential equations is given by ${\chi }^2={10}^{-4}$. The inset shows the asymptotic course of the upper of student-teacher overlaps displayed by a logarithmic $x$ axis.

Figure 4

Double logarithmic plot of the asymptotic generalization error as a function of $\tilde{\alpha }=\alpha \eta$, computed for $M=2$ to $O\left(\eta \right)$, in both the realizable $K=M$ scenario and the over-realizable scenario $K\ge M$. The inset shows the evolution of the generalization error on a single logarithmic scale during the early stage of the specialization transition. While the generalization error approaches perfect learning with an exponential decay in the realizable case, one finds an exponentially decaying transient followed by a power law ${\epsilon }_g\propto {\tilde{\alpha }}^{-2}$ for $K>2$. The initialization of the ordinary differential equations is given by ${\chi }^2={10}^{-4}$.

Figure 5

Evolution of the generalization error in the case of a noisy teacher $M=2$ and noise variance ${\sigma }_{\gamma }^2=0.01$ for $K\in \{2,4,6\}$ and $\eta =1$. The asymptotic generalization error decreases with increasing $K$. The inset shows that the rescaled asymptotic generalization error $K{\epsilon }_g$ is only weakly dependent on $K$. The initialization of the ODEs is given by ${\chi }^2={10}^{-4}$.