Learning may need only a few bits of synaptic precision

Learning in neural networks poses peculiar challenges when using discretized rather then continuous synaptic states. The choice of discrete synapses is motivated by biological reasoning and experiments, and possibly by hardware implementation considerations as well. In this paper we extend a previous large deviations analysis which unveiled the existence of peculiar dense regions in the space of synaptic states which accounts for the possibility of learning efficiently in networks with binary synapses. We extend the analysis to synapses with multiple states and generally more plausible biological features. The results clearly indicate that the overall qualitative picture is unchanged with respect to the binary case, and very robust to variation of the details of the model. We also provide quantitative results which suggest that the advantages of increasing the synaptic precision (i.e., the number of internal synaptic states) rapidly vanish after the first few bits, and therefore that, for practical applications, only few bits may be needed for near-optimal performance, consistent with recent biological findings. Finally, we demonstrate how the theoretical analysis can be exploited to design efficient algorithmic search strategies.

Figure 1

(a) Critical capacity ${\alpha }_c$ as a function of the number of states per synapse $L+1$, for different values of the coding rate $f$. (b) Same as in panel (a), but only for the dense (unbiased) case $f=0.5$, with a wider range of $L$, and showing a fit of the form ${\alpha }^{\infty }-\frac{a}{L^b}$ over the last part of the curve ($L\ge 5$). The fitted parameters are ${\alpha }^{\infty }\simeq 1.0, a\simeq 0.5, b=0.85$.

Figure 2

Local entropy density as a function of the distance $D$ from the reference configuration $\tilde{W}$, comparing the typical case from the Franz-Parisi analysis (black lines, marked with squares) with the large deviations case (red lines, marked with circles), at various values of the number of patterns per variable $\alpha$. The upper bound (gray curve) corresponds to the $\alpha =0$ case where every configuration is a solution. The unphysical portions of the curves where the local entropy becomes negative are dotted. For the typical case, all curves eventually go below zero at some $D_{\min }>0$, for all values of $\alpha$, i.e., typical solutions are isolated. For the large deviations case, the curves for the ${\mathrm{\Phi }}_{\text{RC}}(D,y^{☆}\left(D\right))$ case (RS analysis) and the ${\mathrm{\Phi }}_{\text{RU}}\left(D,\infty \right)$ case (1RSB analysis) yield results which are too close to be distinguished in the plot at this resolution. The “large deviations $\tilde{W}$” curve at $\alpha =1.6$ is interrupted due to numerical problems in solving the equations, but it could continue up to $D=0$, approaching the upper bound for small $\alpha$. Our results indicate that that is the case for $\alpha =1.55$, although it is not shown here since we could not produce a complete curve, again due to numerical difficulties. The curves for $\alpha =1.62$ and $\alpha =1.64$ are interrupted because the equations stop having solutions at some value of $D>0$ (${\alpha }_U$ transition, see text). The large deviations curve at $\alpha =1.3$ is also essentially indistinguishable from the RS computation performed at $y=\infty$.

Figure 3

(a) Local entropy density as a function of the distance from the reference solution $\tilde{W}$, for $L=3$ and $f=0.5$, in the approximation of RS at $y\to \infty$, for various values of $\alpha$, showing five representative curves (from top to bottom): $\alpha <{\alpha }_V, \alpha ={\alpha }_V=0.69, {\alpha }_V<\alpha <{\alpha }_U, \alpha ={\alpha }_U=0.6998, \alpha >{\alpha }_U$. (b) Derivative of the local entropy with respect to the distance $D$, for the same case as for panel (a). Curves are still arranged from top to bottom. This shows the change in concavity occurring at ${\alpha }_V$ and the gap appearing at ${\alpha }_U$.

Figure 4

(a) Transition point ${\alpha }_U$ as a function of the number of states per synapse $L+1$, for different values of the coding rate $f$, as computed in the approximation of RS at $y\to \infty$. (b) Same as panel (a), but ${\alpha }_U$ is divided by the critical capacity ${\alpha }_c$. Error bars reflect the finite precision in the determination of the values. Points for different values of $f$ are slightly shifted relative to each other for improved legibility. Despite the limited number of values, a general tendency of this value to increase with $L$ is observed (the ratio is expected to tend to 1 for $L\to \infty$), while the dependency on $f$ is less clear.

Figure 5

Comparison between different Monte Carlo based solver algorithms for one sample with $N=501, \alpha =1.2, L=4$, and $f=0.1$. The curves show in log-log scale the number of errors of the system as a function of the number of iterations (note that while the number of errors is used as the energy throughout the rest of the paper, none of the algorithms shown here uses it as its objective function). The curves shown are labeled in worst to best order: simulated annealing on $E_{\mathrm{\Delta }}$ [gray curve; see Eq. (17), more than ${10}^6$ iterations required to find a solution]; EdMC starting from random initial condition with zero-temperature dynamics (red curve, less than ${10}^4$ iterations); EdMC using BP marginals as initial condition with zero-temperature dynamics (blue curve, less than ${10}^3$ iterations); EdMC using BP marginals both as initial condition and to propose the Monte Carlo moves (green curve, less than ${10}^2$ iterations). The local-entropy landscape is clearly much smoother than the energy landscape (even when using the energy $E_{\mathrm{\Delta }}$).