Jamming in Multilayer Supervised Learning Models

Critical jamming transitions are characterized by an astonishing degree of universality. Analytic and numerical evidence points to the existence of a large universality class that encompasses finite and infinite dimensional spheres and continuous constraint satisfaction problems (CCSP) such as the nonconvex perceptron and related models. In this Letter we investigate multilayer neural networks (MLNN) learning random associations as models for CCSP that could potentially define different jamming universality classes. As opposed to simple perceptrons and infinite dimensional spheres, which are described by a single effective field in terms of which the constraints appear to be one dimensional, the description of MLNN involves multiple fields, and the constraints acquire a multidimensional character. We first study the models numerically and show that similarly to the perceptron, whenever jamming is isostatic, the sphere universality class is recovered, we then write the exact mean-field equations for the models and identify a dimensional reduction mechanism that leads to a scaling regime identical to the one of spheres.

Figure 1

Contact number as a function of the pressure in the Parity machine for $K=2$ and $\alpha =3$ with two different system sizes (10 realizations for each size). Increasing the system size, the fluctuations decrease. Close to unjamming, we find $z-1\sim p^{1/2}$ as is shown in the inset where the same plot is shown in log scale.

Figure 2

Cumulative distributions of forces and gaps near a jamming point at $\alpha =3$. The simple perceptron corresponds to the model studied in [13, 20, 21]. In agreement with the replica analysis, both distributions follow power-law with exponents $C_f\sim f^{1+\theta }$ and $C_h\sim h^{1-\gamma }$, respectively. The force and gap variables are rescaled such that their power-law regions are collapsed into a dashed line, indicating the theoretical slope of power-law exponents.