Phase transitions in restricted Boltzmann machines with generic priors

We study generalized restricted Boltzmann machines with generic priors for units and weights, interpolating between Boolean and Gaussian variables. We present a complete analysis of the replica symmetric phase diagram of these systems, which can be regarded as generalized Hopfield models. We underline the role of the retrieval phase for both inference and learning processes and we show that retrieval is robust for a large class of weight and unit priors, beyond the standard Hopfield scenario. Furthermore, we show how the paramagnetic phase boundary is directly related to the optimal size of the training set necessary for good generalization in a teacher-student scenario of unsupervised learning.

Figure 1

($\alpha ,{\beta }^{-1}$) RS phase diagram of a random RBM for varying pattern, hidden, and visible unit priors (or $\delta$, ${\mathrm{\Omega }}_h$, ${\mathrm{\Omega }}_{\sigma }$). (a) Boolean visible units and ${\mathrm{\Omega }}_h=1$; the retrieval region approaches the line $\alpha =0$, ${\beta }^{-1}\in [0,{\mathrm{\Omega }}_h]$ as $\delta \to 0$. (b) Boolean visible units and $\delta =1$; the retrieval region approaches the line ${\beta }^{-1}=0$, $\alpha \in [0,{\alpha }_c\left(\delta \right)]$ as ${\mathrm{\Omega }}_h\to 0$. (c) $\delta =1$, ${\mathrm{\Omega }}_h=1$ and soft visible units regularized with a spherical constraint. The retrieval region shrinks to low load as ${\mathrm{\Omega }}_{\sigma }\to 1$. In the three plots the blue line is for the standard Hopfield model (${\mathrm{\Omega }}_{\sigma }=0$, ${\mathrm{\Omega }}_h=\delta =1$), which turns out to have the largest retrieval region.

Figure 2

Learning strong patterns ($\beta >{\mathrm{\Omega }}_h=1$) by training an RBM with $N=1000$ and $P=2$ from a training set of $M=100$ samples ${\left\{{\overline{\mathbf{\sigma }}}^b\right\}}_{b=1}^M$ supplied by a teacher GHM. (a) Overlap between planted (teacher) and inferred (student) patterns. (b) Effective pattern temperatures tend to $\beta$ during learning. (c) RBM training was performed by gradient descent on the likelihood $\mathcal{L}=-lnP\left(\overline{\mathbf{\sigma }}\right|\mathbf{\xi })$, using contrastive divergence [4] to compute the gradient at each learning period $t$.

Figure 3

Overlap between planted (${\mathbf{\xi }}^{★}$) and inferred ($\mathbf{\xi }$) patterns in a teacher-student experiment for weak patterns ($\beta <1$) as a function of the training set size $M=\gamma N$. Vertical lines indicate the theoretical detectability transitions. (a) $P=1$, ${\mathrm{\Omega }}_h=1$, ${\gamma }_c$ grows with decreasing $\beta$. (b) $P=1$, $\beta =0.8$, ${\gamma }_c$ grows with decreasing ${\mathrm{\Omega }}_h$ (low signaling hidden units). (c) $\beta =0.7$, ${\mathrm{\Omega }}_h=1$, ${\gamma }_c$ is effectively independent of $P$ as $P\ll N$. Inferred $\mathbf{\xi }$ are obtained from a MC dynamics that samples from the posterior (15) with a system of $N=2000$ visible units.