Storage capacity of networks with discrete synapses and sparsely encoded memories

Attractor neural networks are one of the leading theoretical frameworks for the formation and retrieval of memories in networks of biological neurons. In this framework, a pattern imposed by external inputs to the network is said to be learned when this pattern becomes a fixed point attractor of the network dynamics. The storage capacity is the maximum number of patterns that can be learned by the network. In this paper, we study the storage capacity of fully connected and sparsely connected networks with a binarized Hebbian rule, for arbitrary coding levels. Our results show that a network with discrete synapses has a similar storage capacity as the model with continuous synapses, and that this capacity tends asymptotically towards the optimal capacity, in the space of all possible binary connectivity matrices, in the sparse coding limit. We also derive finite coding level corrections for the asymptotic solution in the sparse coding limit. The result indicates the capacity of networks with Hebbian learning rules converges to the optimal capacity extremely slowly when the coding level becomes small. Our results also show that in networks with sparse binary connectivity matrices, the information capacity per synapse is larger than in the fully connected case, and thus such networks store information more efficiently.

Figure 1

Overlap as a function of storage load $\alpha$ for the Tsodyks-Feigel'man model (TF) and the clipped Tsodyks-Feigel'man model (CTF), for $f=0.02$. The solid lines represent the theoretical prediction and squares represent the simulation results with a network of size $N=4000$ (mean and standard deviation computed over five independent realizations). Dashed lines mark the storage capacity for CTF and TF. For both models, the neuronal activity threshold $\theta$ is chosen as the one that optimizes capacity. For the parameters chosen here ($f=0.02$), $\tilde{\theta }\equiv \theta /f\approx 0.6$.

Figure 2

Comparison between storage capacity of Hebbian rules and the respective upper bounds. GUBC: Gardner upper bound for networks with continuous weights [4]. GUBB: Gutfreund and Stein upper bound for networks with binary weights [14]. See more details about capacity upper bounds in Appendix pp1-s4. (a) Storage capacity of TF and CTF models as a function of coding level $f$. When $f\to 0$, both capacities increase as $1/fln(1/f)$. Cyan and magenta lines represent the storage capacity for a sparsely connected network with continuous weights (STF) and a network with binary weights (SCTF). The storage capacities of fully connected and sparsely connected networks converge when $f\to 0$. Notice that for fully connected network, storage capacity is defined as ${\alpha }_c=p/N$ while for the sparsely connected network, storage capacity is defined as ${\alpha }_s=p/cN$, where $c\ll 1$. (b) Comparison between storage capacity and upper bounds as a function of coding level $f$. Even at $f={10}^{-2}$, for both TF and CTF models, the capacity is only around a third of the respective bounds, and thus the asymptotic solution (12) is approached very slowly. (c) Comparison between the numerical solution and asymptotic solutions. Solid lines are the numerical solutions of TF and CTF models, the dotted lines with the same color are the corresponding asymptotic solutions in the sparse coding limit [Eq. (13)], and dashed lines represent asymptotic solutions with finite coding level corrections [Eqs. (14) and (15)]. (d) Stored information per synapse as a function of coding level. When the coding level $f$ goes to 0, the information stored in synapses increases but with an extremely slow rate for both TF and CTF models. Dotted lines represents stored information of asymptotic solutions in the sparse coding limit (i.e., ${I[\alpha =(2f\left|\text{ln}f\right|)}^{-1}]$ for continuous weights case and ${I[\alpha =(\pi f\left|\text{ln}f\right|)}^{-1}]$ for binary weights case).

Figure 3

Storage capacity and information capacity for a network with sparse excitatory binary connections. (a) Storage capacity as a function of connection probability $R_1$ and coding level $f$. The storage capacity decreases when $R_1$ decreases. (b) Information capacity per active synapse, as a function of $f$ and $R_1$. The information per synapse $I_e$ increases when $f$ and $R_1$ decrease. This indicates that for learning rule equations (4) and (20), both sparse coding and sparse connectivity can improve the coding efficiency of the network. This result also indicates that the network can have an optimal $R_1$ to balance storage capacity and coding efficiency.

Figure 4

Connection probability that optimizes capacity subject to a synapse maintenance cost $\lambda$. (a) Cost function $C$ as a function of $R_1$ and $\lambda$. (b) Optimal connection probability $R_1^{*}$ for different $\lambda$. We can see that the more costly synaptic maintenance is, the sparser the resulting connectivity. In both (a) and (b), coding level is set to be $f=0.01$.