Mean-field message-passing equations in the Hopfield model and its generalizations

Marc Mézard
Phys. Rev. E 95, 022117 – Published 14 February 2017

Abstract

Motivated by recent progress in using restricted Boltzmann machines as preprocessing algorithms for deep neural network, we revisit the mean-field equations [belief-propagation and Thouless-Anderson Palmer (TAP) equations] in the best understood of such machines, namely the Hopfield model of neural networks, and we explicit how they can be used as iterative message-passing algorithms, providing a fast method to compute the local polarizations of neurons. In the “retrieval phase”, where neurons polarize in the direction of one memorized pattern, we point out a major difference between the belief propagation and TAP equations: The set of belief propagation equations depends on the pattern which is retrieved, while one can use a unique set of TAP equations. This makes the latter method much better suited for applications in the learning process of restricted Boltzmann machines. In the case where the patterns memorized in the Hopfield model are not independent, but are correlated through a combinatorial structure, we show that the TAP equations have to be modified. This modification can be seen either as an alteration of the reaction term in TAP equations or, more interestingly, as the consequence of message passing on a graphical model with several hidden layers, where the number of hidden layers depends on the depth of the correlations in the memorized patterns. This layered structure is actually necessary when one deals with more general restricted Boltzmann machines.

  • Figure
  • Figure
  • Figure
  • Figure
  • Received 26 August 2016

DOI:https://doi.org/10.1103/PhysRevE.95.022117

©2017 American Physical Society

Physics Subject Headings (PhySH)

Interdisciplinary PhysicsStatistical Physics & ThermodynamicsPhysics of Living SystemsNetworks

Authors & Affiliations

Marc Mézard

  • Physics Department, Ecole Normale Supérieure, PSL Research University, Paris

Article Text (Subscription Required)

Click to Expand

References (Subscription Required)

Click to Expand
Issue

Vol. 95, Iss. 2 — February 2017

Reuse & Permissions
Access Options
Author publication services for translation and copyediting assistance advertisement

Authorization Required


×
×

Images

×

Sign up to receive regular email alerts from Physical Review E

Log In

Cancel
×

Search


Article Lookup

Paste a citation or DOI

Enter a citation
×