• Open Access

Deterministic and Generalized Framework for Unsupervised Learning with Restricted Boltzmann Machines

Eric W. Tramel, Marylou Gabrié, Andre Manoel, Francesco Caltagirone, and Florent Krzakala
Phys. Rev. X 8, 041006 – Published 8 October 2018

Abstract

Restricted Boltzmann machines (RBMs) are energy-based neural networks which are commonly used as the building blocks for deep-architecture neural architectures. In this work, we derive a deterministic framework for the training, evaluation, and use of RBMs based upon the Thouless-Anderson-Palmer (TAP) mean-field approximation of widely connected systems with weak interactions coming from spin-glass theory. While the TAP approach has been extensively studied for fully visible binary spin systems, our construction is generalized to latent-variable models, as well as to arbitrarily distributed real-valued spin systems with bounded support. In our numerical experiments, we demonstrate the effective deterministic training of our proposed models and are able to show interesting features of unsupervised learning which could not be directly observed with sampling. Additionally, we demonstrate how to utilize our TAP-based framework for leveraging trained RBMs as joint priors in denoising problems.

  • Figure
  • Figure
  • Figure
  • Figure
  • Figure
  • Figure
  • Figure
5 More
  • Received 17 February 2017
  • Revised 24 July 2018

DOI:https://doi.org/10.1103/PhysRevX.8.041006

Published by the American Physical Society under the terms of the Creative Commons Attribution 4.0 International license. Further distribution of this work must maintain attribution to the author(s) and the published article’s title, journal citation, and DOI.

Published by the American Physical Society

Physics Subject Headings (PhySH)

Networks

Authors & Affiliations

Eric W. Tramel1, Marylou Gabrié2, Andre Manoel1, Francesco Caltagirone3, and Florent Krzakala2,4

  • 1OWKIN, Inc., New York, New York 10022, USA
  • 2Laboratoire de Physique Statistique, École Normale Supérieure, PSL University, Paris 75005, France
  • 3Snips, Paris 75002, France
  • 4Université Pierre et Marie Curie, Sorbonne Universités, Paris 75005, France and LightOn, Paris 75005, France

Popular Summary

Is it possible to build a machine that teaches itself? How can we grade its proficiency in a learned task? Is it possible to observe what the machine has learned? These are just a few of the open questions in the field of unsupervised machine learning. To help address these questions, we have developed a framework for training, comparing, and analyzing restricted Boltzmann machines (RBMs), an important practical and theoretical building block for deep neural networks.

An RBM “learns” by employing a joint statistical neural model trained to maximize the correlation between data, external observables, and a set of parameters from which it builds internal representations of those observables. Our framework relies on statistical physics methods as a basis for investigating statistical inference over many interacting variables, a common feature of machine learning models. Specifically, we use the Thouless-Anderson-Palmer formalism from spin-glass theory to approximate the macroscopic behavior of the many simple, widely interacting neurons that comprise an RBM.

In our numerical experiments, we demonstrate the effective deterministic training of our proposed models and show interesting features of unsupervised learning that could not be directly observed with sampling. We also show how to use our framework to put RBMs to work on more practical tasks, such as cleaning up noisy signals.

The framework we propose is not only useful for the analysis and inspection of restricted Boltzmann machines, but it also leads to novel practical training techniques and new applications for these unsupervised models.

Key Image

Article Text

Click to Expand

References

Click to Expand
Issue

Vol. 8, Iss. 4 — October - December 2018

Subject Areas
Reuse & Permissions
Author publication services for translation and copyediting assistance advertisement

Authorization Required


×
×

Images

×

Sign up to receive regular email alerts from Physical Review X

Reuse & Permissions

It is not necessary to obtain permission to reuse this article or its components as it is available under the terms of the Creative Commons Attribution 4.0 International license. This license permits unrestricted use, distribution, and reproduction in any medium, provided attribution to the author(s) and the published article's title, journal citation, and DOI are maintained. Please note that some figures may have been included with permission from other third parties. It is your responsibility to obtain the proper permission from the rights holder directly for these figures.

×

Log In

Cancel
×

Search


Article Lookup

Paste a citation or DOI

Enter a citation
×