Self-Learning Machines Based on Hamiltonian Echo Backpropagation

A physical self-learning machine can be defined as a nonlinear dynamical system that can be trained on data (similar to artificial neural networks) but where the update of the internal degrees of freedom that serve as learnable parameters happens autonomously. In this way, neither external processing and feedback nor knowledge of (and control of) these internal degrees of freedom is required. We introduce a general scheme for self-learning in any time-reversible Hamiltonian system. It relies on implementing a time-reversal operation and injecting a small error signal on top of the echo dynamics. We show how the physical dynamics itself will then lead to the required gradient update of learnable parameters, independent of the details of the Hamiltonian. We illustrate the training of such a self-learning machine numerically for the case of coupled nonlinear wave fields and other examples.

Popular Summary

To a large extent, the recent explosive growth of the field of machine learning has been driven by continuous improvement in electronic hardware. To surpass the limitations of current digital electronic hardware, there is a growing interest in the development of physical learning machines—physical devices that can process information and learn from data. Such devices hold the potential for greater speed, massive parallelization, and lower energy consumption compared to learning machines implemented entirely in software. The ideal goal would be to construct a physical learning machine that can learn autonomously: a self-learning machine. Here, we present a new general method to construct self-learning machines.

In our construction, both the information processing and the learning parameters are encoded in degrees of freedom of a time-reversible physical system, one where a trajectory and its time reverse are possible solutions. We present an efficient training method based on time-reversal operations that makes it possible to update the physical learning parameters in situ, with no external feedback and under no additional assumptions about the internal dynamics of the system.

Our method, being so general, could be applied to many physical systems. We believe that this result makes it feasible not only to realize self-learning machines based on mature platforms, such as integrated photonics, but also could open entirely new possibilities based on different systems.

Figure 1

Different types of physical learning machines. (a) Physical learning machine requiring feedback based on the outcome, to tune the internal “learning” parameters $\theta$ inside the physical device. (b) Self-learning machine that does not involve feedback. It updates the learning parameters in a fully autonomous way (for example, using physical backpropagation and a physical gradient update; see main text).

Figure 2

Overview: landscape of physical learning machines. Our work falls into the broad class of optimization approaches. It belongs to the small class of works that involve both physical backpropagation and self-learning, but in contrast to other approaches, our scheme is universal, i.e., independent of the specific Hamiltonian (for some related works, see main text in Shen et al. [15], Guo et al. [16], Skinner et al. [11], Psaltis et al. [8, 10]).

Figure 3

Physical backpropagation and physical gradient update. (a) Forward pass (nonlinear evolution) of the “evaluation” wave field $\mathrm{\Psi }$ in the presence of a background potential landscape provided by a “learning field” $\mathrm{\Theta }$. Any small perturbation $\delta \mathrm{\Theta }$ will affect the output, as indicated here, which in turn changes the value of the cost function. In panel (b), we show that, instead of testing the cost function response for many potential updates of the learning field $\mathrm{\Theta }$, Hamiltonian echo backpropagation automatically produces the required update in a single backward pass, via physical dynamics. After applying a time-reversal operation ($\mathrm{\Psi }\mapsto {\mathrm{\Psi }}^{*}$), the subsequent nonlinear wave-field evolution is a time-reversed version of the forward pass. A small variation $\delta \mathrm{\Psi }$ injected on top can now be used to backpropagate the error signal (variation of the cost function at the output); see main text. This then leads to the required update of the learning field $\theta$, via the physical interaction between $\mathrm{\Psi }$ and $\theta$, realizing a fully autonomous learning machine.

Figure 4

Different possible architectures of the self-learning machine. We illustrate the different possible configurations of the learning and evaluation dynamical variables, for the case of a setup consisting of wave fields interacting in 1D. We consider that the SL device is in the highlighted region, which is where the nonlinear interaction given by $H_{\mathrm{SL}}$ takes place. If both the input and output of both fields are injected from outside, all the time reversals and decay steps can be performed externally, and there is freedom to choose $H_{\theta ,\mathrm{bath}}$. If not, the Hamiltonian of the SL device, $H_{\mathrm{SL}}$, has to be engineered to have a continuously degenerate ground state in $\mathrm{\Theta }$. In panel (a), both $\mathrm{\Psi }(-T)$ and $\mathrm{\Theta }(-T)$ are wave packets injected from outside the SL device. In panel (b), the input $\mathrm{\Psi }(-T)$ is a wave packet injected from outside the device, but $\mathrm{\Theta }$ is confined inside.

Figure 5

Ingredients of a self-learning machine based on Hamiltonian echo backpropagation. (a) Sequence of events. (b) Use of “pseudodissipation via ancillary modes.” Here, the learning field $\mathrm{\Psi }$ is decomposed into a number of discrete modes. As time progresses, less and less modes participate in the physical interaction (yellow boxes), becoming ancillary modes. Both these ancillary modes and the learning field $\theta$ also take part in time reversal, however. (c) Time evolution of the learning field momentum (left) and the field itself (right) during one forward-backward pass, including the effects of time reversal (we show a single value of the field at one location). The momentum decays back to zero, but the field has been updated suitably. (d) Intrinsic dynamics of the learning field during the decay step, showing a continuously degenerate ground state (i.e., a manifold attractor). In the right panel, the effect of the training dynamics is to break this degeneracy and produce a minimum determined by the cost function.

Figure 6

Hamiltonian echo backpropagation in a simple mechanical model. In panel (a), a charged ball $\mathrm{\Psi }$ is launched with a velocity representing the input to the machine. Its trajectory is deflected depending on the position of another charged ball $\mathrm{\Theta }$, which can freely move along a rail and which experiences a momentum kick due to this interaction. Eventually, as shown in panel (b), the velocities of both balls are reversed, and the ball $\mathrm{\Psi }$ is slightly adjusted according to its deviation from the desired outcome (“error signal”). (c) Afterwards, it retraces almost exactly its initial trajectory, delivering yet another momentum kick to the $\mathrm{\Theta }$ particle. Finally, any remaining velocity of $\mathrm{\Theta }$ is dissipated and converted into a finite displacement, representing the training update. Of course, the expressive power of this machine, containing only these 2 degrees of freedom, is very limited (i.e., it can only produce very rough approximations to most input-output relations). Nevertheless, it illustrates the generality of the procedure—e.g., an arbitrary (even unknown) static force field acting on the $\mathrm{\Psi }$ particle will not spoil the learning procedure.

Figure 7

Training a self-learning machine (simulations). (a) Time evolution of the “evaluation field” intensity $|\mathrm{\Psi }{|}^2$ during a forward pass for a machine that has learned to implement XOR ($b=a_1\oplus a_2$). The region in which the interaction with the learning field occurs is enclosed by yellow dashed lines. (b) Time evolution of the learning field momentum ${\pi }_{\theta }$ during a single training step. The pattern builds up from left to right as the $\mathrm{\Psi }$ pulse propagates. It is time reversed and then partially deleted as the $\mathrm{\Psi }$ pulse travels back. The finite remainder at time $T$ is due to the injected error signal, and it produces the desired learning update. (c) Evolution of the learning field $\theta$ during many training steps. The training starts with a smooth random configuration, and there are 600 HEB steps (each for a different, randomly chosen training sample). (d) Cost function evolution during a particular training run for the model of panels (a)–(c). (e) Setup of a self-learning machine that learns to classify handwritten digits (MNIST). The outcome is a one-hot encoding of the detected digit; i.e., one of ten patches “lights up.” (f) Training error evolution of the MNIST setup for different strengths of the wave-field nonlinearity. The line colors represent a different value of the parameters ${\chi }_1$, ${\chi }_2$. Note that ${\chi }_1$ and ${\chi }_2$ are the nonlinear strengths at the convolutional and dense layers, respectively. The color that represents each value of ${\chi }_1$, ${\chi }_2$ can be seen from panel (g). In the inset, we display the cost function evolution when the SL machine is trained via HEB (blue line). We compare it with the gradient descent according to the true gradient (red line), with the same learning rate. The true gradient is obtained via autodifferentiation. In order to better compare the results, we average the cost function evolution over ten different trajectories. (g) Final validation set classification accuracy.

Figure 8

Effects of noise and nonlinear effects in the gradient estimation. (a) Relative error in the approximation of ${\partial }_{{\theta }_j}C$, where ${\theta }_j$ is the value of the learning field in the middle of the evolution domain. We compare the update of ${\theta }_j$ obtained via HEB with an accurate estimation that was computed by means of the finite difference method. The result is presented as a function of the dimensionless quantity $\epsilon C_{\max }$, where $C_{\max }$ is the maximum possible value of the cost function. (b) Root-mean-square error of the update obtained by means of HEB in the presence of noise. On the one hand, for small values of $\epsilon$, the noise overwhelms the term proportional to the gradient. (c) Analyzing the influence of a noisy time-reversal operation on training. Here, we consider a discretized version of the nonlinear Schrödinger equation, where the effect of noise is more prominent. We display the cost function during training for different values of the learning rate, at a fixed finite noise level whose standard deviation is given by ${\sigma }_{\text{noise}}^2=0.05I$ (see Appendix pp1 for details).

Figure 9

Self-learning machine based on an array of nonlinear cavities (simulations). (a) Schematics of the setup. The cavities that encode the learning degrees of freedom are placed on a 2D square lattice. Each nonlinear ${\mathrm{\Theta }}_{ij}$ cavity is coupled to its nearest neighbors, with a coupling strength given by $J$. The cavities that represent the evaluation degrees of freedom, ${\mathrm{\Psi }}_i$, are coupled to the ${\mathrm{\Theta }}_{ij}$ cavities in the corners of the lattice, and the coupling strength is given by $K$. All cavities have an identical Kerr nonlinearity whose strength is given by $g$. The input is encoded in the initial values of $\mathbf{(}{\psi }_0(-T),{\psi }_1(-T)\mathbf{)}$, while the output is given by ${\psi }_3(0)$. (b) Training error evolution for different values of the intensity of the input. We choose a reference level of intensity $I_0$ that fulfills $g{I}_{0}T=3$. (c) Robustness of the SL machine to fabrication imperfections. In particular, we consider that each cavity has a random frequency shift whose distribution is given by $\mathcal{N}(0,{\sigma }_{\mathrm{\Delta }})$. Note that, since the random shifts are on the order of $0.1\pi /T-1\pi /T$, their effect on the output value is very significant. The HEB method can successfully implement gradient descent in the imperfect machine (since it is Hamiltonian independent). However, for larger values of ${\sigma }_{\mathrm{\Delta }}$, the training process seems to converge to worse performance. Nevertheless, this result shows robustness to significant fabrication imperfections. (d) Evolution of the output as the training progresses (with an input intensity given by $I_{\mathrm{in}}=1.6I_0$). Each curve represents each different input. (e) Time evolution of the field amplitude in the output cavity during a single forward pass, once the SL machine has been trained. (f) Outputs obtained when the input is noisy. We generate noisy inputs by adding Gaussian noise to the ideal input. After feeding each input to the SL machine, a noisy output is obtained. Here, we plot the resulting outputs vs the standard deviation of the Gaussian noise. We can observe that the output remains highly concentrated around the target value when ${\sigma }_{\text{noise}}$ is in the range of 0%–3% of the input amplitude.