Neural Networks with a Redundant Representation: Detecting the Undetectable

We consider a three-layer Sejnowski machine and show that features learnt via contrastive divergence have a dual representation as patterns in a dense associative memory of order $P=4$. The latter is known to be able to Hebbian store an amount of patterns scaling as $N^{P-1}$, where $N$ denotes the number of constituting binary neurons interacting $P$ wisely. We also prove that, by keeping the dense associative network far from the saturation regime (namely, allowing for a number of patterns scaling only linearly with $N$, while $P>2$) such a system is able to perform pattern recognition far below the standard signal-to-noise threshold. In particular, a network with $P=4$ is able to retrieve information whose intensity is $\mathcal{O}(1)$ even in the presence of a noise $\mathcal{O}(\sqrt{N})$ in the large $N$ limit. This striking skill stems from a redundancy representation of patterns—which is afforded given the (relatively) low-load information storage—and it contributes to explain the impressive abilities in pattern recognition exhibited by new-generation neural networks. The whole theory is developed rigorously, at the replica symmetric level of approximation, and corroborated by signal-to-noise analysis and Monte Carlo simulations.

Figure 1

Schematic representations of the restricted Seinowskj machine (left) and its dual representation in terms of a bipartite dense associative network (right). In the former, neurons $i$, $\mu$, $\rho$ interact 3-wisely through the coupling ${\xi }_{i\mu }^{\rho }$ [see also Eq. (1)], while, in the latter, neurons $i$, $\mu$, $j$, $\nu$ interact 4-wisely through the coupling $J_{i\mu }^{j\nu }$ [see also Eq. (5)].

Figure 2

Phase diagram for the DAM with ($P=4$)-wise interactions among the $N$ neurons and a load $K=\alpha N$, as a function of the capacity $\alpha$ and of the noise level $1/\beta$. This diagram was obtained by solving the self-consistent Eqs. (11)–(15) and by identifying the retrieval phase as the region where each neural configuration corresponding to a stored pattern is a maximum of the pressure—either global ($R1$), or local ($R2$)—the spin-glass (SG) phase as the region where retrieval capabilities are lost due to prevailing “slow noise” $\alpha$, and the ergodic ($E$) phase as the region where retrieval capabilities are lost due to prevailing “fast noise” $1/\beta$ [21]. This diagram was found for a noise regime $\mathcal{O}(\sqrt{N})$ [see Eq. (8)], where the standard Hopfield would fail.

Figure 3

Expected Mattis magnetization obtained from Monte Carlo simulations run for $N=150$ and for different values of $\alpha$, as a function of $1/\beta$. Notice that, as $\alpha$ is tuned from 0.25 to 0.35, the magnetization abruptly drops even at small values of $1/\beta$, consistently with the transition from the region $R1$ to the region $R2$ found theoretically (see Fig. 2 and [21]).