Computational capabilities of random automata networks for reservoir computing

This paper underscores the conjecture that intrinsic computation is maximal in systems at the “edge of chaos”. We study the relationship between dynamics and computational capability in random Boolean networks (RBN) for reservoir computing (RC). RC is a computational paradigm in which a trained readout layer interprets the dynamics of an excitable component (called the reservoir) that is perturbed by external input. The reservoir is often implemented as a homogeneous recurrent neural network, but there has been little investigation into the properties of reservoirs that are discrete and heterogeneous. Random Boolean networks are generic and heterogeneous dynamical systems and here we use them as the reservoir. A RBN is typically a closed system; to use it as a reservoir we extend it with an input layer. As a consequence of perturbation, the RBN does not necessarily fall into an attractor. Computational capability in RC arises from a tradeoff between separability and fading memory of inputs. We find the balance of these properties predictive of classification power and optimal at critical connectivity. These results are relevant to the construction of devices which exploit the intrinsic dynamics of complex heterogeneous systems, such as biomolecular substrates.

Figure 1

Schematic of a reservoir computing system. The input layer delivers the input signals to random nodes inside the reservoir. The readout layer receives output signals from random nodes inside the reservoir. The reservoir itself is made of a collection of computing nodes that are randomly interconnected. The reservoir creates a representation of the input signals that can be read and classified by the readout layer. Learning is performed by training only the readout layer nodes and connections.

Figure 2

The computational capability $\Delta$ of RBN reservoirs with $N=500$, $L\in [1,N]$, and $\tau \in [1,9]$. Parameters $\langle K\rangle$ and $\mathcal{T}$ are $\langle K\rangle =1$, $\mathcal{T}=10$ (a), $\langle K\rangle =1$, $\mathcal{T}=100$ (b), $\langle K\rangle =2$, $\mathcal{T}=10$ (c), $\langle K\rangle =2$, $\mathcal{T}=100$ (d), $\langle K\rangle =3$, $\mathcal{T}=10$ (e), and $\langle K\rangle =3$, $\mathcal{T}=100$ (f). The computational capability $\Delta$ varies according to $\langle K\rangle \in \{1,2,3\}$ and $\mathcal{T}=10$ in the left column and $\mathcal{T}=100$ in the right column.

Figure 3

The computational capability $\Delta$ of RBN reservoirs $\mathcal{M}$ with $N=500$ and $\langle K\rangle \in [1,4]$ summed over $L\in [1,500]$. These are calculated as ${\sum }_{L=1}^N\Delta (\mathcal{M},\mathcal{T},\tau )$, where $\mathcal{T}=10$ (a), and ${\sum }_{L=1}^N\Delta (\mathcal{M},\mathcal{T},\tau )$, where $\mathcal{T}=100$ (b). The dashed curve is a spline fit to the highest $\Delta$, illustrating that near-critical connectivity maximizes computational capability, particularly in high $\mathcal{T}$.

Figure 4

Mutual information between the input and reservoir $\mathcal{I}(I:R)$ and reservoir and output $\mathcal{I}(R:O)$. The results are for $\langle K\rangle =1.0$ (a) and (b), $\langle K\rangle =2.0$ (c) and (d), $\langle K\rangle =3.0$ (e) and (f). We have also included the intrinsic information in the input stream, reservoir, and output. For all of these cases, $\tau =1$. In an ideal reservoir, $\mathcal{I}(I:R)={\mathcal{H}}_I$ and $\mathcal{I}(R:O)={\mathcal{H}}_O$. For $\langle K\rangle =1.0$, as $L$ grows both $\mathcal{I}(I:R)$ and $\mathcal{I}(R:O)$ grow to a maximum level below the ideal level. The reservoir in these systems does not carry enough information for the output layer to solve the task. For $\langle K\rangle =2.0$, as $L$ grows, the mutual information grows and reaches the sufficient level at $L=20$. For $\langle K\rangle =3.0$, mutual information peaks near $L=5$. Small perturbation from the input provides enough information to the reservoir to reconstruct the desired output.

Figure 5

The generalization capability $G$ of a RBN-RC device $\mathcal{M}$ on the task ${\mathcal{B}}_n$ is dependent on $L$ and $\langle K\rangle$, as well as task parameters $n$ and $\mathcal{T}$. Here, the parameters are $n=3,\mathcal{T}=10$ (a), $n=3,\mathcal{T}=100$ (b), $n=7,\mathcal{T}=10$ (c), $n=9,\mathcal{T}=10$ (d). Notably, chaotic networks achieve their maximum generalization capability with a lower $L$ than ordered networks. Ordered networks possess little memory and so their performance drops as $n$ increases. On the other hand, chaotic networks perform poorly with $\mathcal{T}=100$ as opposed to $\mathcal{T}=10$, due to an inadequately fading memory.

Figure 6

The generalization capability $G$ of a RBN-RC device $\mathcal{M}$ on the task ${\mathcal{A}}_n$ is dependent on $L$ and $\langle K\rangle$ and task parameters $n$ and $\mathcal{T}$. Here, the parameters are $n=3,\mathcal{T}=10$ (a), $n=3,\mathcal{T}=100$ (b), $n=5,\mathcal{T}=10$ (c), $n=7,\mathcal{T}=10$ (d). Due to high sensitivity to the initial perturbations, the generalization capability of chaotic networks drops as the length of the input stream increases from $\mathcal{T}=10$ to 100. Ordered networks possess little short-term memory and are least robust to an increase in $n$, with generalization capability on ${\mathcal{A}}_7$ no better than chance.