Disentangling Representations in Restricted Boltzmann Machines without Adversaries

A goal of unsupervised machine learning is to build representations of complex high-dimensional data, with simple relations to their properties. Such disentangled representations make it easier to interpret the significant latent factors of variation in the data, as well as to generate new data with desirable features. The methods for disentangling representations often rely on an adversarial scheme, in which representations are tuned to avoid discriminators from being able to reconstruct information about the data properties (labels). Unfortunately, adversarial training is generally difficult to implement in practice. Here we propose a simple, effective way of disentangling representations without any need to train adversarial discriminators and apply our approach to Restricted Boltzmann Machines, one of the simplest representation-based generative models. Our approach relies on the introduction of adequate constraints on the weights during training, which allows us to concentrate information about labels on a small subset of latent variables. The effectiveness of the approach is illustrated with four examples: the CelebA dataset of facial images, the two-dimensional Ising model, the MNIST dataset of handwritten digits, and the taxonomy of protein families. In addition, we show how our framework allows for analytically computing the cost, in terms of the log-likelihood of the data, associated with the disentanglement of their representations.

Popular Summary

Finding meaningful representations of complex data in automatic ways is a long-standing goal of machine learning. Representations should capture enough information about the data to guarantee high-fidelity reconstruction, discard irrelevant details, and have simple relations with important features underlying the data distribution. Methods for disentangling representations often rely on an adversarial game, where two neural networks compete while mutually improving their performances. However, this approach generally suffers from numerical instabilities and is difficult to implement in practice. We show how disentanglement can be achieved with a single model, in which parameters are adequately constrained.

We illustrate our approach on restricted Boltzmann machines (RBMs), a simple kind of network with one layer for data configurations and another for representations. RBMs are widely applicable in many contexts, where limitations on the quantity of available data preclude the usage of more complex, deeper architectures. The simplicity of RBMs makes them amenable to analytical calculations, which elucidates the role of the constraints.

Our approach allows us to better understand the cost (in terms of quality of the generated data) associated with disentanglement. We hope it will also make controlled generation of data and feature discovery easier in future applications. Besides the applications to RBMs we present here, our constraint-based framework could in principle be applied to other unsupervised architectures.

Figure 1

Entangled vs disentangled representations. A set of high-dimensional data points (bottom) is mapped through unsupervised learning onto a latent representation (top). Data are colored in purple and orange according to a binary-valued attribute, e.g., being an odd or even number for MNIST images of handwritten digits. Left: When representations are entangled, the separation of data classes is not aligned with a single latent direction. Right: When representations are disentangled, one or few directions in latent space (blue) separate the labeled classes, while other directions are not correlated with the label (red).

Figure 2

Datasets considered in the paper and entanglement of representations. (a) CelebA dataset of face images [23]; two-dimensional Ising model; $\mathrm{MNIST}0/1$ database of handwritten digits [24]; multiple sequence alignments from the Pfam PF00013 family of the KH domain. (b) Samples generated by different RBMs trained on each dataset. See Supplemental Material [26] Appendix A 6 for the architectures of the RBMs used in each case. (c) Histogram of the absolute value of the Pearson correlations between hidden-unit inputs and the chosen label; see Eq. (5). Smiling or not smiling for CelebA, sign of the magnetization for the Ising model, whether the digit is a 0 or 1 for $\mathrm{MNIST}0/1$, and whether the KH sequence is from bacterial or eukaryotic origin.

Figure 3

Model schema. (a) Constraints imposed on all hidden units promote overlapping hidden input distributions of the two classes. (b) Constraints imposed on a subset of hidden units (red) promote class separation on the remaining hidden units (blue).

Figure 4

First- and second-order constraints. (a) The first-order constraint (6) ensures that the classes have the same means in input space by imposing orthogonality of the weights to the vector separating their centers of mass in data space (red). (b) Second-order constraints (9) ensure that the two classes have the same covariance in input space.

Figure 5

Application to the CelebA dataset. Left: label corresponds to the presence or absence of eyeglasses. Right: label corresponds to smiling or not smiling. (a) Selected images from the data arranged by the value of their projection along the vector $\mathbf{q}$ defined in Eq. (7). Below, the histogram of these projections is computed for all images in the data. The inset shows a heat map of the vector $\mathbf{q}$. (b) Samples generated by an unconstrained RBM and histogram of their projections on vector $\mathbf{q}$. (c) Samples generated by a RBM, all the hidden units of which are subject to the constraint in Eq. (6) (dashed red). The histogram (red) of projections on $\mathbf{q}$ concentrates on intermediate values. (d) Samples generated by a RBM trained under constraint (6) acting on all but one released hidden unit (dashed blue) and histogram of projections along $\mathbf{q}$ (blue). Details about the RBMs’ architecture and training can be found in the Supplemental Material [26] Appendix A 6.

Figure 6

Transitions between labeled classes in the CelebA dataset. RBMs are trained subject to the linear constraint acting on all but the first hidden unit denoted $h^{\star }$. Samples are generated conditioned on a frozen value of $h^{\star }$, which is flipped in the center of the Markov chain (indicated by the dashed blue lines). (a) Label corresponds to the “eyeglasses” attribute of CelebA. Samples are collected every three Gibbs iterations. (b) Label corresponds to the “smiling” attribute of CelebA. Samples are collected every five Gibbs iterations.

Figure 7

Learning RBMs on two-dimensional Ising model data. (a) Magnetization and heat capacity as functions of the temperature for the samples generated by the Ising model (13). (b) Magnetization and heat capacity of samples generated by the RBM trained on the Ising data. (c) Magnetization and heat capacity of samples generated by the RBM with constraint (6) acting on all hidden units. (d) Magnetization and heat capacity of samples generated by the RBM with quadratic constraint (9) acting on all hidden units. (c) Magnetization and heat capacity of samples generated by the RBM with linear constraint (6) acting on all but one hidden unit. (f) Maximum AUC of classifiers trained to predict the sign of the sample magnetization from the RBM inputs. (g),(h) Typical weights learned by the RBM at selected temperatures ($1/T=0.35$, 0.4, 0.46, 0.5) for the unconstrained RBM and for the RBM with the first-order constraint. (i) Free weights attached to the released hidden unit compared to $4\beta$ times the magnetization of the Ising model.

Figure 8

Manipulating representations of the RBM trained on $\mathrm{MNIST}0/1$. (a) Samples generated by the RBM initialized with a data image (0 or 1). Top two rows show a standard (unconstrained) RBM. Bottom two rows show samples from the RBM trained with linear (red dashed) and quadratic (green dashed) constraints. In both cases, a Markov chain is generated by Gibbs sampling (starting from a 0 or 1 data digit), and images are saved every 64 steps, until reaching a total of 16 samples as shown. (b) Lower bound ${\mathcal{S}}_{\text{label}}+{\mathcal{L}}_{\text{class}}$ to the mutual information between inputs and labels [see Eq. (25)] vs classifier width. The bounds to MI are measured in bits and shown in discontinuous lines. Colors correspond to the different RBM models. Black: standard and unconstrained. Red: fully constrained with linear constraints; see Eq. (6). Green: fully constrained with quadratic constraints; see Eq. (9). (c) Samples from the RBM trained with first-order constraint acting on all but one hidden unit, which is flipped at the middle of the MC chain (blue arrow). Starting from a 0 data digit, samples are saved every 64 Gibbs steps. Top panel shows an enlarged view of the transition, with images every three steps instead. The lower panels show the logarithm of the unnormalized probability $\ln \tilde{P}(\mathbf{v})=\ln ({\sum }_{\mathbf{h}}{e}^{-E(\mathbf{v},\mathbf{h})})$ of generated digits by constrained RBMs, evaluated on RBMs trained only on 0s (RBM0) or 1s (RBM1). Purple and orange dashed lines correspond to the average $\ln \tilde{P}(\mathbf{v})$ of data digits 0 and 1.

Figure 9

Manipulating representations of a RBM trained on $\mathrm{MNIST}0/1/2/3$. (a) Sketch of the constraints applied to hidden-unit weights in the case of multiple classes, here, $D=4$. (b) Vectors ${\mathbf{q}}_d^{(1)}$ for digit classes 0, 1, 2, and 3; see Eq. (26). (c) Inputs received by the three released hidden units [in blue on panel (a)], when the 6000 digit images in classes 0, 1, 2, and 3 are presented ($x$ axis). In the fourth, bottom panel, the inputs received by a random hidden unit from the constrained group (black) are shown. (d) Weights $w_{I\mu }$ learned by the released hidden units $\mu =1$, 2, 3. (e) Samples generated from this machine by Gibbs sampling (images shown are taken every 64 steps). The first (top row) released unit 1 is active, while the other two are inactive. Then, we activate unit 2 (second row) while inactivating unit 1 (blue arrow), and similarly for 3 (third row). In the last row, all three units are inactive.

Figure 10

Taxonomy of protein families. (a) Sequence logos of eukaryotic (purple, above) and bacterial (orange, below) sequences from the PF00013 protein family. We use the following color code: green for polar residues, blue for basic, red for acidic, and orange for hydrophobic. Gaps are shown in black. (b) Ribbon structure of KH domain showing locations of the Gly–Gly loop and flanking helices. Image prepared with Mol* Viewer [47]. (c) Sequence logos of 100 000 generated sequences when the released hidden unit is set to 1 (top) or 0 (below). To ensure that sampling is equilibrated, we track the average and standard deviation of the energy of the samples in time and see that these statistics are essentially constant after approximately 200 steps, suggesting that samples can be collected every 5000 steps. (d) Weights of the released hidden unit. (e) Inputs received by the released hidden unit when presented with sequences from the two classes. (f) Markov chain started from bacterial (orange) or eukaryotic (purple) sequences from the data. The panel shows the probability of being a eukaryotic vs bacterial sequence in a perceptron classifier. Discontinuous lines are the average value for the data sequences of each class. A total of 1024 Gibbs sampling steps are taken, and the flip of $h^{*}$ occurs at step 512 (blue arrow). (g) Enlarged view near the transition, showing also the log of the unnormalized marginal [$\log {\tilde{P}}_{\mathrm{RBM}}(\mathbf{v})$] of sampled sequences (right axis) evaluated on a RBM trained on the full family.

Figure 11

Semisupervised training with a subsampled labeled dataset. (a) Overlap (27) between ${\mathbf{q}}_{\mathrm{sub}}^{(1)}$ (computed on a subsampled-labeled dataset) and ${\mathbf{q}}_{\mathrm{full}}^{(1)}$ (computed on the full dataset) plotted as a function of the number of labeled examples in the subsampled dataset divided by the dimension ($28\times 28=784$ for MNIST). An average of over 100 random realizations of the subsampled dataset is taken. The black solid curve shows the empirical result, while the dashed green curve is the theoretical estimate (30). Inset shows a diagram of how class separation relates to the overlap in connection to (30). (b) For the pink and cyan dots of (a), we plot an example of the obtained vectors ${\mathbf{q}}_{\mathrm{sub}}^{(1)}$ in comparison to ${\mathbf{q}}_{\mathrm{full}}^{(1)}$. (c) Label manipulation using the subsampled ${\mathbf{q}}_{\mathrm{sub}}^{(1)}$ in the two cases. (d) Histogram of log-likelihoods of (subsampled) training and withheld dataset for a RBM trained on a subset of 0 (top) or 1 (bottom) digits, corresponding to the labeled datasets used in the cyan dot in the previous panels. The black and green vertical lines indicate the average values. (e) Histogram of log-likelihoods of training and withheld dataset of the partially constrained RBM in the cyan setting of the previous panels.

Figure 12

Likelihood calculations. First row shows numerical estimates of the log-likelihood using RBMs with binary hidden units, along with the costs of applying Eq. (6) partially or on the full hidden layer. Bottom row shows analytical results obtained in a RBM with one hidden spin unit and the remaining Gaussian hidden units (Fig. 13). First column shows the legend: black for the unconstrained model, red for models with all hidden units constrained, and blue for models with the constraint acting on all but one hidden unit. Subsequent columns show the results for the three datasets considered: $\mathrm{MNIST}0/1$, two-dimensional Ising model ($L=64$), and the KH protein domain. The discontinuous arrows in the first panel highlight the likelihood costs of partial label erasure (red) and disentanglement (blue).

Figure 13

Gaussian-spin RBM. (a) The Gaussian-spin RBM has one spinlike hidden unit $h^{*}=h_1=\pm 1$, whereas all other hidden units are Gaussian. (b) The spin hidden unit (blue) separates the two labeled classes. Gaussian hidden units (red) model intraclass variability. (c) Illustration of Poincaré theorem.