Exact Training of Restricted Boltzmann Machines on Intrinsically Low Dimensional Data

The restricted Boltzmann machine is a basic machine learning tool able, in principle, to model the distribution of some arbitrary dataset. Its standard training procedure appears, however, delicate and obscure in many respects. We bring some new insights to it by considering the situation where the data have low intrinsic dimension, offering the possibility of an exact treatment and revealing a fundamental failure of the standard training procedure. The reasons for this failure—like the occurrence of first-order phase transitions during training—are clarified thanks to a Coulomb interactions reformulation of the model. In addition, a convex relaxation of the original optimization problem is formulated, thereby resulting in a unique solution, obtained in precise numerical form on $d=1$, 2 study cases, while a constrained linear regression solution can be conjectured on the basis of an information theory argument.

Figure 1

Bipartite structure of the RBM (left). Hyperplanes defined by the weight vectors and bias associated with each hidden variable can delimit fixed density regions in input space (right).

Figure 2

1D intrinsic data ($N_v={10}^3$) with five clusters solved with $N_h=20$ predefined features thanks to a natural gradient ascent of the LL. Dotted lines indicate location of features with nonvanishing weights $q_j$. The feature contributions $\mathcal{F}(m)-h(m)$ to the free energy are seen to regress $h(m)$ on the data. The resulting distribution is shown (red) on the inset with the empirical training distribution (blue) and the failed result of a standard RBM training (green).

Figure 3

2D intrinsic dataset ($N_v={10}^3$) with six pointlike clusters and a circular one (upper left) and corresponding RBM density (upper right) found with $N_h=900$ predefined features, along with its free energy landscape (bottom left) and Coulomb charges distribution (bottom right).