Network hierarchy and pattern recovery in directed sparse Hopfield networks

Many real-world networks are directed, sparse, and hierarchical, with a mixture of feedforward and feedback connections with respect to the hierarchy. Moreover, a small number of master nodes are often able to drive the whole system. We study the dynamics of pattern presentation and recovery on sparse, directed, Hopfield-like neural networks using trophic analysis to characterize their hierarchical structure. This is a recent method which quantifies the local position of each node in a hierarchy (trophic level) as well as the global directionality of the network (trophic coherence). We show that even in a recurrent network, the state of the system can be controlled by a small subset of neurons which can be identified by their low trophic levels. We also find that performance at the pattern recovery task can be significantly improved by tuning the trophic coherence and other topological properties of the network. This may explain the relatively sparse and coherent structures observed in the animal brain and provide insights for improving the architectures of artificial neural networks. Moreover, we expect that the principles we demonstrate here, through numerical analysis, will be relevant for a broad class of system whose underlying network structure is directed and sparse, such as biological, social, or financial networks.

Figure 1

Illustration of the real-world connectome of C. elegans which has intermediate incoherence with node height drawn using trophic levels. Data were taken from [4]. The figure was drawn using the networkx graph package [5].

Figure 2

Example of the distribution of trophic incoherence with the temperaturelike parameter $T_{\text{gen}}$ in this generative model.

Figure 3

Plot of scaled spectral radius of generated networks vs trophic incoherence following the same analytic prediction as real networks as shown in Ref. [3]. The number of nodes is always $N=500$ and the mean degree $\langle k\rangle =20$.

Figure 4

Plot of performance of 200 networks with $N=500$ and $\langle k\rangle =100$ recovering ten patterns vs trophic incoherence. Patterns are shown for the different 20% sets of nodes.

Figure 5

Plot of performance of 200 networks with $N=500$ and $\langle k\rangle =100$ recovering ten patterns vs trophic incoherence. Patterns are shown for the different 60% sets of nodes.

Figure 6

Performance of 200 networks with trophic incoherence showing patterns to the 20% lowest (blue), highest (green), and random (orange) nodes by trophic level, for $N=500$ and $\langle k\rangle =20$.

Figure 7

Performance of networks of varying degree for fixed generation temperature $T_{\text{gen}}=1$, showing patterns to the 20% lowest nodes by trophic level, for $N=500$. The average trend is shown by the red dashed line and the standard deviation is shown in the shaded area.

Figure 8

Distribution of performance for networks showing ten patterns to the lowest trophic level, highest degree, and random 20% of nodes for $N=500$ and (a) $\langle k\rangle =100$ and (b) $\langle k\rangle =20$.

Figure 9

Relationship between network performance and the ratio between the number of edges leaving the set shown the pattern and total edges in the network for $N=500$ and $\langle k\rangle =20$. Networks are of intermediate incoherence.

Figure 10

Plot of distribution of performance of 200 intermediate incoherence networks vs the integral from 0 to 1 over a parameter $\alpha$ of the curve of the number of nodes of trophic level less than $\alpha h_{\text{max}}$ for $N=500$ and $\langle k\rangle =20$. Networks are of intermediate incoherence.

Figure 11

Time series of the pattern order parameter for an average degree 100 network with 500 nodes storing 20 patterns. Each color represents the pattern most recently shown to the network. At this edge density recovery is very consistent.

Figure 12

Time series for different network components for an average degree 20 network with 500 nodes storing 4 patterns: (a) full pattern order parameter, (b) strongly connected component, (c) bottom 20% of nodes by trophic level, and (d) top 20% of nodes by trophic level. Each color represents the pattern most recently shown to the network.

Figure 13

Distribution of performance for biased networks biased towards (a) low-level nodes with $\gamma =-0.5$ and (b) high-level nodes with $\gamma =0.5$, for $N=500$ and $\langle k\rangle =20$.

Figure 14

Time series of the pattern order parameter calculated inside the different components for an average degree 20 network with 500 nodes storing 4 patterns, generated with a bias towards adding edges to lower-level nodes: (a) order parameter of the whole network and (b) largest strongly connected component only. Each color represents the pattern most recently shown to the network.

Figure 15

Performance of low-level-biased networks with nodes of average degrees 20 and 500 where the largest strongly connected component contains at least 60% of the neurons.