Advanced mean-field theory of the restricted Boltzmann machine

Learning in restricted Boltzmann machine is typically hard due to the computation of gradients of log-likelihood function. To describe the network state statistics of the restricted Boltzmann machine, we develop an advanced mean-field theory based on the Bethe approximation. Our theory provides an efficient message-passing-based method that evaluates not only the partition function (free energy) but also its gradients without requiring statistical sampling. The results are compared with those obtained by the computationally expensive sampling-based method.

Figure 1

A RBM is composed of one hidden layer and one visible layer. No lateral connections exist within both hidden and visible layers. Each hidden node is connected to all visible nodes with symmetric coupling weights and is responsible for capturing high-order dependence. The original RBM is shown in the left panel as an example of three hidden nodes (solid circles) and five visible nodes (empty circles). The right panel shows a transformed factor graph after marginalization of hidden states for theoretical analysis. Each factor node (square node) represents the probabilistic normalization of a hidden node given the state of all visible nodes (see the main text).

Figure 2

Free-energy density ($f=F/N$) of single instances of RBMs. Iterations of the message passing equations are always converged to produce the data points. The error bars give statistical errors across ten random instances. (a) Free-energy density as a function of $\alpha$ (density of hidden nodes); $g=1.0$ and $v=0.05$. The inset shows the absolute difference, $\left|\mathrm{\Delta }f\right|$, of the free-energy density estimated by the exact enumeration and the Bethe approximation (BA) for $N=20$ (see a comparison for ten instances with $\alpha =0.6$ in the inset, where the line indicates equality). (b) Free-energy density as a function of weight strength $g$ and field strength $v$ (shown in the inset). $\alpha =0.5$.

Figure 3

Stability parameter $\lambda$ as a function of model parameters. $N=1000, \alpha =0.5$, and $v=0.05$. The error bars give statistical errors across ten random instances. The left inset gives stability versus $\alpha$ with $g=1.0$ and $v=0.05$. The right inset gives two examples (black and red) taken from the main figure at $g=2.1$. $S\left(0\right)$ is the initial total variance.

Figure 4

Evaluation performance of mean-field theory in comparison with the Gibbs sampling. Iterations of the recursive equations always converged to produce the data points. The error bars give statistical errors across ten random instances. (a) RMS error as a function of $g$; $N=100$ and $v=0.02$. (b) Scatter plot for a typical example of $N=100, \alpha =0.5, g=0.1$, and $v=0.02$. The inset shows an example of $g=0.55$ (other parameters do not change). The line indicates equality. (c) RMS error ${\delta }_{\mathit{m}}$ reached by CD-$k$ as a function of $k$ in comparison to the Bethe approximation; $N=100, \alpha =0.2, g=0.55$, and $v=0.02$. The result is averaged over five random instances.