Mechanisms of dimensionality reduction and decorrelation in deep neural networks

Deep neural networks are widely used in various domains. However, the nature of computations at each layer of the deep networks is far from being well understood. Increasing the interpretability of deep neural networks is thus important. Here, we construct a mean-field framework to understand how compact representations are developed across layers, not only in deterministic deep networks with random weights but also in generative deep networks where an unsupervised learning is carried out. Our theory shows that the deep computation implements a dimensionality reduction while maintaining a finite level of weak correlations between neurons for possible feature extraction. Mechanisms of dimensionality reduction and decorrelation are unified in the same framework. This work may pave the way for understanding how a sensory hierarchy works.

Figure 1

Schematic illustration of a deep neural network. The deep neural network performs a layer-by-layer nonlinear transformation of the original input data (a high-dimensional vector $\mathbf{v}$). During the transformation, a cascade of internal representations (${\mathbf{h}}^1,\cdots ,{\mathbf{h}}^d$) are created. Here, $d=3$ denotes the depth of the deep network.

Figure 2

Representation dimensionality versus depth in deterministic deep networks with random weights. Ten network realizations are considered for each network width. $\rho /\sqrt{N}=0.05, g=0.8$, and ${\sigma }_b=0.1$. The right inset shows how the overall strength of covariance changes with depth and connection strength ($g$), and the left inset is a mechanism illustration (${\mathrm{\Sigma }}^{0,1,*}$ has been scaled by $N$). The crosses show simulation results ($g=0.8$) obtained from ${10}^5$ sampled configurations at each layer, compared with the theoretical predictions.

Figure 3

(a) Representation behavior as a function of depth in generative deep networks. Ten network realizations are considered for each network width. The top inset shows an example ($N=150$) of reconstruction errors ($\varepsilon \equiv \parallel {\mathbf{h}}^{\prime }{-\mathbf{h}\parallel }_2^2$) between input $\mathbf{h}$ and reconstructed one ${\mathbf{h}}^{\prime }$ for each layer during learning. The bottom inset shows the overall strength of covariance as a function of depth. (b) Numerically estimated off-diagonal correlation versus its theoretical prediction ($N=150$). In these plots, we generate $M=60000$ training examples (each example is an $N$-dimensional vector) from the random RBM whose parameters follow the normal distribution $\mathcal{N}(0,g/N)$ for weights and $\mathcal{N}(0,{\sigma }_b)$ for biases. $g=0.8$ and ${\sigma }_b=0.1$. Then these examples are learned by the DBN (see simulation details in Appendix pp3). (c) One typical learning trial shows how the estimated dimensionality evolves.

Figure 4

The behavior of the additive term ${\left(K_1^l\right)}^2\mathrm{{\rm Y}}$ as a function of the network depth.

Figure 5

The eigenvalue distribution of the covariance matrix estimated from the deep computation. (a) The distribution for the deep networks with random weights ($N=100$). One hundred instances are used. In the inset, the distribution at deeper layers is compared with the Marchenko-Pastur law of a random-sample covariance matrix. The tail part is enlarged for comparison. (b) The distribution obtained from an unsupervised learning system of deep belief networks is strongly different from the Marchenko-Pastur law. Ten instances are used.