Variational mean-field theory for training restricted Boltzmann machines with binary synapses

Unsupervised learning requiring only raw data is not only a fundamental function of the cerebral cortex, but also a foundation for a next generation of artificial neural networks. However, a unified theoretical framework to treat sensory inputs, synapses, and neural activity together is still lacking. The computational obstacle originates from the discrete nature of synapses, and complex interactions among these three essential elements of learning. Here, we propose a variational mean-field theory in which the distribution of synaptic weights is considered. The unsupervised learning can then be decomposed into two intertwined steps: A maximization step is carried out as a gradient ascent of the lower bound on the data log-likelihood, in which the synaptic weight distribution is determined by updating variational parameters, and an expectation step is carried out as a message passing procedure on an equivalent or dual neural network whose parameter is specified by the variational parameters of the weight distribution. Therefore, our framework provides insights on how data (or sensory inputs), synapses, and neural activities interact with each other to achieve the goal of extracting statistical regularities in sensory inputs. This variational framework is verified in restricted Boltzmann machines with planted synaptic weights and handwritten-digits learning.

Figure 1

Schematic illustration of the variational model. $N=4$ in this example (say, $i$, $j$, $k$, and $l$). The hidden layer has $P=3$ neurons. Note that $(N,P)$ can be arbitrarily finite numbers. Left: The RBM architecture receives the data of $M$ examples and encodes the hidden features of the data into synaptic connections (weights) in an unsupervised way. Here, we take into account the weight uncertainty captured by a variational distribution $q_{{\lambda }_{i\mu }}\left({\xi }_i^{\mu }\right)$, where ${\xi }_i^{\mu }$ refers to the connection between sensory neuron $i$ and hidden neuron $\mu$, and ${\lambda }_{i\mu }$ is the variational parameter. Right: The equivalent RBM model where the connection becomes the variational parameter and the additional random field applied to a hidden neuron is determined by the variance (${\mathrm{\Xi }}_{\mu }^2$) of the weighted-sum input to that hidden neuron (see the main text). $z_{\mu }$ is a standard normally distributed random variable. The equilibrium properties of the equivalent RBM are used to adjust the variational parameter improving the lower bound on the data log-likelihood.

Figure 2

Learning performance in planted RBM models ($N=100$) and real data sets ($N=784$). The inverse temperature $\beta =1$. Each learning step indicates all synapses are updated once. $B_1=B_2=1000$. Here, the Kullback-Leibler divergence refers to $\mathrm{KL}(q_{\mathit{\lambda }}\left(\mathit{\xi }\right)\parallel P\left(\mathit{\xi }\right))$, and the approximate lower bound of the data likelihood [Eqs. (5, 6, 7)] has been subtracted by its starting value. (a) Learning trajectories of RBM models with $P=3$ and orthogonal planted receptive field vectors of three hidden neurons ($M=500$). The inset shows the evolution of the Kullback-Leibler divergence and approximate lower bound. (b) Typical learning performance in planted RBM models as a function of data density $\alpha =\frac{M}{N}$. The learning performance is estimated at the 200th step. Each marker in the plot indicates the average over ten independent realizations of the model, with the error bar indicating the standard deviation. The theoretical threshold ${\alpha }_{\mathrm{thr}}=1$ for $c=0$ [12, 13] and ${\alpha }_{\mathrm{thr}}=0.596$ for $c=0.3$ [18]. (c) Learning performance vs correlation $c$ with $P=2$ and $M=500$. The average is done over ten independent realizations. (d) Learning trajectories (KL and LB per model parameter) of RBM models with $P=100$ for structured (handwritten digits, thus $N=784$) inputs of $M=1000$ images (another 1000 images for test). $B_1=B_2=500$. A localized variational parameter map (within the receptive field) is also shown.