Role of Synaptic Stochasticity in Training Low-Precision Neural Networks

Stochasticity and limited precision of synaptic weights in neural network models are key aspects of both biological and hardware modeling of learning processes. Here we show that a neural network model with stochastic binary weights naturally gives prominence to exponentially rare dense regions of solutions with a number of desirable properties such as robustness and good generalization performance, while typical solutions are isolated and hard to find. Binary solutions of the standard perceptron problem are obtained from a simple gradient descent procedure on a set of real values parametrizing a probability distribution over the binary synapses. Both analytical and numerical results are presented. An algorithmic extension that allows to train discrete deep neural networks is also investigated.

Figure 1

(Left) The training error and the squared norm against the number of training epochs for $\alpha =0.55$ and $N=10001$ averaged over 100 samples. (Right) Success probability in the classification task as a function of the load $\alpha$ for networks of size $N=1001$, 10001 averaging 1000 and 100 samples, respectively. In the inset, we show the average training error at the end of GD as a function of $\alpha$.

Figure 2

(Left) Energy of the binarized configuration versus the control variable $q_{\star }$. We show the equilibrium prediction of Eq. (9) and numerical results from the GD algorithm and a GD algorithm variant where after each update we rescale the norm of $m$ to $q_{\star }$ until convergence before moving to the next value of $q_{\star }$ according to a certain schedule. The results are averaged over 20 random realizations of the training set with $N=10001$. (Right) Entropy of binary solutions at fixed distance $d$ from the BCs of the spherical, binary, and stochastic perceptron ($q_{\star }=0.7$, 0.8, and 0.9 from bottom to top) at thermodynamic equilibrium. In both figures, $\alpha =0.55$, also $\beta =20$ for the stochastic perceptron and $\beta =\infty$ for the spherical and binary ones.