Data-driven effective model shows a liquid-like deep learning

The geometric structure of an optimization landscape is argued to be fundamentally important to support the success of deep neural network learning. A direct computation of the landscape beyond two layers is hard. Therefore, to capture the global view of the landscape, an interpretable model of the network-parameter (or weight) space must be established. However, the model is lacking so far. Furthermore, it remains unknown what the landscape looks like for deep networks of binary synapses, which plays a key role in robust and energy efficient neuromorphic computation. Here, we propose a statistical mechanics framework by directly building a least structured model of the high-dimensional weight space, considering realistic structured data, stochastic gradient descent training, and the computational depth of neural networks. We also consider whether the number of network parameters outnumbers the number of supplied training data, namely, over- or under-parametrization. Our least structured model reveals that the weight spaces of the under-parametrization and over-parameterization cases belong to the same class, in the sense that these weight spaces are well connected without any hierarchical clustering structure. In contrast, the shallow-network has a broken weight space, characterized by a discontinuous phase transition, thereby clarifying the benefit of depth in deep learning from the angle of high-dimensional geometry. Our effective model also reveals that inside a deep network, there exists a liquid-like central part of the architecture in the sense that the weights in this part behave as randomly as possible, providing algorithmic implications. Our data-driven model thus provides a statistical mechanics insight about why deep learning is unreasonably effective in terms of the high-dimensional weight space, and how deep networks are different from shallow ones.

Figure 1

Schematic illustration of how to construct an effective physics model analyzing the deep-learning weight space. We consider a four-layer neural network of binary weights to perform a classification task, where the input data pass through the weight regions of the left boundary (LB), the interior (ITR), and the right boundary (RB) in order. After the neural network is trained from different initializations, the weight statistics including magnetizations (first-order moments) $m_i=\langle {\sigma }_i\rangle$ and pairwise correlations (second-order moments) $C_{ij}=\langle {\sigma }_i{\sigma }_j\rangle$ will be computed directly from the collected weight configurations. Based on the weight statistics, we establish an effective data-driven Ising model that is able to reflect the weight space characteristics, and then apply the entropy landscape analysis by introducing a distance-dependent extra term into the Boltzmann distribution of the effective Ising model. The entropy landscape analysis can help to explore the internal structure of the weight parameter space by counting how dense weight parameters $\mathit{\sigma }$ are at a typical distance $d$ away from the reference one ${\mathit{\sigma }}^{*}$.

Figure 2

Neural network training and properties of weight configurations sampled from independent runs of training ($P=60000$, $N=675$). (a) Training trajectories of four-layer networks. The cross-entropy (the orange line) and test accuracy (the blue line) can reach 0.61 and $83.5\%$, respectively at the 200th epoch, where the fluctuation indicated by the shadow regime is computed from ten independent training runs. (b) Two-dimensional representations of $20000$ random initializations (green dots) and $20000$ learned weight configurations (blue dots). The low dimensional projection is carried out by the t-SNE technique [35]. The random initialization denotes the weight configuration (or initial state) sampled by the Gaussian random initial fields ${\mathit{\theta }}_0$. The collected weight configurations (or solution states) are obtained at the end of the training starting from the initial fields ${\mathit{\theta }}_0$. For the solution states, the deeper their colors are, the higher the corresponding test accuracies are, while the random initial states are all of the same green color of chance-level accuracy.

Figure 3

Properties of the data-driven Ising model ($P=60000$, $N=675$). The Ising model is constructed by the maximum entropy method. The total area of the histogram is normalized to one. (a) Histograms of full-model (all) parameters. Couplings $J_{ij}$ and biases $h_i$ (inset) are shown. (b) Reconstructed correlation $C_{ij}$ and magnetization $m_i$ (inset) from the effective Ising model vs the measured ones for the full model. Full lines indicate the equality. (c) Ising energy density ${\epsilon }_{\mathrm{Ising}}$ vs test accuracy of the full model. Each accuracy corresponds to a large number of weight configurations, where the blue-solid line indicates expectation values and the light-blue shadow denotes the fluctuation. The blue-dashed line indicates the typical energy of the Ising model. (d) Histograms of model parameters in subparts (see Fig. 1) of the deep hierarchy. Couplings $J_{ij}$ and biases $h_i$ (inset) are shown.

Figure 4

Properties of the data-driven Ising model ($P=60000$, $N=675$). The Ising model is constructed by the pseudolikelihood maximization method ($\lambda =0.05$). The ${\ell }_1$ regularization training leads to a sparse model where about $73.5\%$ of weights are zero. (a) Histograms of full-model (all) parameters. Couplings $J_{ij}$ and biases $h_i$ (inset) are shown. The total area of the histogram is normalized to one. (b) Reconstructed correlation $C_{ij}$ and magnetization $m_i$ (inset) from the effective Ising model vs the measured ones for the full model. Full lines indicate the equality.

Figure 5

Entropy landscape analyses in three different scenarios of deep learning. (a) Under-parametrization case ($\alpha \approx 0.01$). The curves are the mean results over 20 distinct reference weight configurations ${\mathit{\sigma }}^{*}$ selected from the low energy region (for references coming from high- and medium-energy regions, see Fig. 13 and Fig. 14 in Appendix pp4). Entropy curves are plotted along the direction of the decreasing coupling field $x$ from 3 to $-3$, while the Hamming distance per weight $d$ vs $x$ is plotted by considering two directions of increasing and decreasing $x$ (inset). No hysteresis loops and entropy gaps are observed. (b) Over-parametrization case ($\alpha \approx 1.35$). Other conditions are the same as (a). (c) Shallow network case. The inset shows a hysteresis loop for distance $d$ vs coupling field $x$, and the entropy curves correspond to the lower-branch of the hysteresis loop. A first-order phase transition arises in this case showing a complex landscape in the high-dimensional weight space.

Figure 6

Entropy landscape analyses of the sparse Ising model (constructed by the pseudolikelihood maximization method, and $\lambda =0.05$). Other model settings are the same as Fig. 4. (a) Under-parametrization case ($\alpha \approx 0.01$). The curves are the mean results over 20 distinct reference weight configurations ${\mathit{\sigma }}^{*}$ selected from the low energy region (for references coming from high- and medium-energy regions, the profile holds qualitatively). Entropy curves are plotted along the direction of the decreasing coupling field $x$ from 3 to $-3$, while the Hamming distance per weight $d$ vs $x$ is plotted by considering two directions of increasing and decreasing $x$ (inset). No hysteresis loops and entropy gaps are observed. (b) Over-parametrization case ($\alpha \approx 1.35$). Other conditions are the same as (a).

Figure 7

Iso-energy entropy profile for the under-parameterization case. (a) The curves are mean results estimated from ten independent trials where the reference configurations ${\mathit{\sigma }}^{*}$ are selected from the low energy region of ${\epsilon }_{\mathrm{Ising}}$. Both the iso-energy case and the normal case (no fixed-energy constraint) share the same reference configurations. (b) The iso-energy constraint is achieved by fixing the energy to the Ising energy density ${\epsilon }_{\mathrm{Ising}}$ of the reference configuration, for which the secant method is applied to search for a compatible inverse temperature $\beta$, while the coupling field $x$ still varies from 3 to $-3$. In the normal case, the inverse temperature $\beta$ is set to 1. $d=0.5$ (or $x=0$) corresponds to the original unperturbed model.

Figure 8

Distribution of TAP free energies of the Ising model. The model is constructed by the maximum entropy method. The histogram is obtained by iterating the TAP equation (see Appendix pp8) from $2000$ random initializations of the magnetizations. (a) Under-parametrization case. (b) Over-parametrization case. (c) Shallow network case.

Figure 9

Monte Carlo dynamics on the landscape of the Ising model. The dynamics starts at a reference weight configuration, and is measured in a time unit of one proposed spin flip (one step in the abscissa). The test accuracy and Ising energy are both tracked during the dynamics. The fluctuation is computed from 100 independent trials. (a) Deep network case. (b) Shallow network case.

Figure 10

Effects of permutation symmetry on the entropy landscape ($P=60000$, $N=675$). The Ising model is constructed by the maximum entropy method. The procedure of generating permutation symmetric weight configurations is detailed in Appendix pp9. The fraction of permutation symmetry ($r_{\mathrm{ps}}$) is also specified there, and indicates how large the number of the permutation-dependent weight configurations (used for learning the Ising model) is. The entropy is the mean value over 20 independent references. (a) Full network. (b) Interior part of the network. (c) Left boundary of the network. (d) Right boundary of the network.

Figure 11

Algorithmic implication of the effective model. Training with randomly-frozen ITR works, consistent with our theory. The training is carried out on the network architecture 784-100-100-10 with the full dataset. The performance of training with trained ITR is compared.

Figure 12

Three-spin correlations of the effective Ising model, in comparison with those estimated from the collected weight ensemble. (a) Predicted three-spin correlations vs the measured ones (data). 200 spins are randomly selected from the effective model to do the comparison, due to a huge number of three-spin correlations. The full line shows equality, and dashed lines correspond to the deviation criterion $\epsilon$, which is the absolute value of the deviation between the predicted three-spin correlation and the true one. The predicted three-spin correlation is estimated by running a Monte Carlo simulation of the effective Ising model. (b) Predicted percentage vs deviation criterion. The predicted percentage in all considered three-spin correlations is the percentage of those three-spin correlations whose absolute deviation from the true one is within the given deviation criterion $\epsilon$.

Figure 13

Entropy landscape analyses in the high-energy references case. The reference weight configurations are randomly selected from the high-energy region, which displays no significant difference from the low-energy reference case (Fig. 5). (a) Under-parametrization case. (b) Over-parametrization case.

Figure 14

Entropy landscape analyses in the medium-energy reference case. The reference weight configurations are randomly selected from the medium-energy region, which displays no significant difference from the low-energy reference case (Fig. 5). (a) Under-parametrization case. (b) Over-parametrization case.

Figure 15

Histograms of model parameters in the Ising model of the shallow-network case. (a) Histogram of inferred couplings. The couplings are distributed relatively broader compared with non-shallow networks. (b) Histogram of the inferred biases. The biases are also distributed broadly.

Figure 16

Entropy landscape analysis on the fashion-MNIST dataset. The network structure remains 20-15-15-10 and the training data size is 500 corresponding to the over-parameterization case. One million weight configurations after learning are collected to calculate the weight statistics, which should reach the test accuracy threshold equal to $70\%$. The root-mean squared errors $\mathrm{\Delta }$ are less than 0.085 to obtain an effective Ising model. To get the entropy curves, 20 weight configurations are selected from the low energy region as the reference configurations ${\mathit{\sigma }}^{*}$. Other conditions are the same as in the MNIST case.

Figure 17

Detection of double descent phenomenon in deep learning of continuous weights or binary weights. The network structure is specified by 20-h-h-10, and the training data size is $2000$. The ratio between the network complexity and the training data size is defined by $\alpha$. We choose the cross entropy as the cost function. [(a),(b)] Deep learning of continuous weights. The point where the training error vanishes defines the interpolation point, indicating a peak in test error as well. [(c),(d)] Deep learning of binary weights. The double descent phenomenon is absent. We actually train the network of binary weights for a total number of ${10}^5$ epochs, therefore the abscissa should be multiplied by a factor of 100.

Figure 18

Performance comparison between deep and shallow networks. Ten independent trainings are considered for the fluctuation.

Figure 19

Schematic illustration of permutation symmetry transformation. The transformation is carried out by a permutation of hidden neurons according to descending levels of pre-activation (denoted by ${\mathbf{a}}^l$), i.e., ${\mathbf{a}}_{\mathrm{new}}^l={\mathbb{S}}_h{\mathbf{a}}_{\mathrm{old}}^l$, which further leads to a new weight configuration ${\mathit{\sigma }}_{\mathrm{gr}}$, related to the original one $\mathit{\sigma }$ via ${\mathit{\sigma }}_{\mathrm{gr}}=\mathcal{T}\mathit{\sigma }$, where the matrix entry takes either zero or one. The thick edges represent the involved weights for the transformation, i.e., the incoming and outgoing synapses for the permuted neurons.