Difference between memory and prediction in linear recurrent networks

Recurrent networks are trained to memorize their input better, often in the hopes that such training will increase the ability of the network to predict. We show that networks designed to memorize input can be arbitrarily bad at prediction. We also find, for several types of inputs, that one-node networks optimized for prediction are nearly at upper bounds on predictive capacity given by Wiener filters and are roughly equivalent in performance to randomly generated five-node networks. Our results suggest that maximizing memory capacity leads to very different networks than maximizing predictive capacity and that optimizing recurrent weights can decrease reservoir size by half an order of magnitude.

Figure 1

[Top (a)] MC and [bottom (b)] PC as a function of $W$ for $R_{ss}\left(t\right)=e^{-0.1\left|t\right|}$ (blue lines) and $R_{ss}\left(t\right)=\frac{1}{2}{e}^{-0.1\left|t\right|}+\frac{1}{2}{e}^{-\left|t\right|}$ (green lines), computed using Eqs. (15) and (18) in the main text. Whereas PC is maximized for some intermediate $W$ that depends on the input signal, MC is maximized in the limit $W\to 1$. When $\left|W\right|\ge 1$, the network no longer satisfies the echo state property, and so we only calculate PC, MC for $\left|W\right|<1$.

Figure 2

[Top (a)] MC and [bottom (b)] PC as a function of $N$ for $R_{ss}\left(t\right)=\frac{1}{2}{e}^{-0.1\left|t\right|}+\frac{1}{2}{e}^{-\left|t\right|}$ and $\omega ,\vec{d}$ drawn randomly: ${\omega }_i\sim \mathcal{U}[0,1],d_i\sim \mathcal{U}[-1,1]$. The signal-limited maximal PC is shown in green, whereas MC is network limited. Both MC, PC were computed using Eqs. (15) and (18) in the main text.

Figure 3

At the top (a), the hidden Markov model generating input to the nonlinear recurrent network. Edges are labeled $p\left(x\right|g\left)\right|x$, where $x$ is the emitted symbol and $p\left(x\right|g)$ is the probability of emitting that symbol when in hidden state $g$ and the arrows indicate which hidden state one goes to after emitting a particular symbol from the previous hidden state. This hidden Markov model generates a zero-mean unit-variance even process, which has the autocorrelation function $R_{ss}\left(t\right)=\{\begin{array}{cc}\hfill {(-\frac{1}{2})}^{\left|t\right|+1}& \hfill \left|t\right|\ge 1\\ \hfill 1& \hfill t=0\end{array}$. At the bottom (b), the predictive capacity of random nonlinear recurrent networks whose evolution is given by Eq. (21) with $f\left(x\right)=tanh\left(x\right)$ and entries of $W$ and $v$ drawn randomly: $W_{ij},v_i\sim \mathcal{U}[0,1]$, where $W$ then is scaled so that its largest magnitude eigenvalue has an absolute value of 1/1.1. Twenty-five random networks are surveyed at each $N$, and the blue line tracks the mean. The green line shows both the predictive capacity of the optimized one-node linear network and the upper bound from Eq. (20).