Barriers and dynamical paths in alternating Gibbs sampling of restricted Boltzmann machines

Clément Roussel, Simona Cocco, and Rémi Monasson
Phys. Rev. E 104, 034109 – Published 7 September 2021

Abstract

Restricted Boltzmann machines (RBM) are bilayer neural networks used for the unsupervised learning of model distributions from data. The bipartite architecture of RBM naturally defines an elegant sampling procedure, called alternating Gibbs sampling (AGS), where the configurations of the latent-variable layer are sampled conditional to the data-variable layer and vice versa. We study here the performance of AGS on several analytically tractable models borrowed from statistical mechanics. We show that standard AGS is not more efficient than classical Metropolis-Hastings (MH) sampling of the effective energy landscape defined on the data layer. However, RBM can identify meaningful representations of training data in their latent space. Furthermore, using these representations and combining Gibbs sampling with the MH algorithm in the latent space can enhance the sampling performance of the RBM when the hidden units encode weakly dependent features of the data. We illustrate our findings on three datasets: Bars and Stripes and MNIST, well known in machine learning, and the so-called lattice proteins dataset, introduced in theoretical biology to study the sequence-to-structure mapping in proteins.

  • Figure
  • Figure
  • Figure
  • Figure
  • Figure
  • Figure
  • Figure
10 More
  • Received 12 July 2021
  • Accepted 18 August 2021

DOI:https://doi.org/10.1103/PhysRevE.104.034109

©2021 American Physical Society

Physics Subject Headings (PhySH)

Statistical Physics & ThermodynamicsNetworks

Authors & Affiliations

Clément Roussel, Simona Cocco, and Rémi Monasson*

  • Laboratory of Physics of the École Normale Supérieure, CNRS UMR 8023 & PSL Research, Sorbonne Université, 75005 Paris, France

  • *Corresponding author: remi.monasson@phys.ens.fr

Article Text (Subscription Required)

Click to Expand

References (Subscription Required)

Click to Expand
Issue

Vol. 104, Iss. 3 — September 2021

Reuse & Permissions
Access Options
Author publication services for translation and copyediting assistance advertisement

Authorization Required


×
×

Images

×

Sign up to receive regular email alerts from Physical Review E

Log In

Cancel
×

Search


Article Lookup

Paste a citation or DOI

Enter a citation
×