Physics-enhanced neural networks learn order and chaos

Artificial neural networks are universal function approximators. They can forecast dynamics, but they may need impractically many neurons to do so, especially if the dynamics is chaotic. We use neural networks that incorporate Hamiltonian dynamics to efficiently learn phase space orbits even as nonlinear systems transition from order to chaos. We demonstrate Hamiltonian neural networks on a widely used dynamics benchmark, the Hénon-Heiles potential, and on nonperturbative dynamical billiards. We introspect to elucidate the Hamiltonian neural network forecasting.

Figure 1

Hamiltonian flow. (a) Hénon-Heiles orbit wrapped many times around the hypertorus appears to intersect at the creases in this 3D projection. (b) Different colors indicating the fourth dimension show that the apparent intersections are actually separated in 4D phase space.

Figure 2

Neural network schematics. Weights (red lines) and biases (yellow spheres) connect inputs (green cubes) through neuron hidden layers (gray planes) to outputs (blue cubes). (a) NN has $2N$ inputs and $2N$ outputs. (b) HNN has $2N$ inputs and 1 output but internalizes the output's gradient in its weights and biases.

Figure 3

Hénon-Heiles flows. Two sample Hénon-Heiles flows $\{q_x,q_y\}$ for different initial conditions forecast by conventional neural network (a), Hamiltonian neural network (b), and Hénon-Heiles differential equations (c), for small, medium, and large bounded energies $0<E<1/6$. Hues code momentum magnitudes, from red to violet; orbit tangents code momentum directions. Orbits fade into the past. HNN's physics-informed forecasting is especially better than NN's at high energies.

Figure 4

Lyapunov spectrum. Integrating the variational equations estimates the Hénon-Heiles Lyapunov exponents ${\lambda }_n$ after time $t$ for NN (a), HNN (b), and true (c) for an example initial condition. HHN better approaches the expected Hamiltonian spectrum $\{-\lambda ,0,0,+\lambda \}$.

Figure 5

Lyapunov scaling. Maximum Lyapunov exponent ${\lambda }_M$ versus energy $E$ for NN, HNN, and true. HNN reproduces the correct energy exponent, while NN shows no trend at all.

Figure 6

Order to chaos. Fraction of chaotic orbits $f_c$ for random energies $E$, as inferred by the smaller alignment index $\alpha$, for conventional neural network (green circles), Hamiltonian neural network (magenta squares), and Hénon-Heiles differential equations (blue diamonds). NN is especially poor for chaotic high-energy orbits. Insets are sample orbits.

Figure 7

Introspection. (a) When an autoencoder compresses HNN forecasting, its loss function ${\mathcal{L}}_b$ (magenta squares) drops precipitously as its bottleneck layer increases past $N_b=4$ neurons, the dimensionality of the Hénon-Heiles system; when an autoencoder compresses NN forecasting, its loss function (green circles) wiggles irregularly, oblivious to this transition. (b) HNN's compressed representation $\{n_1,n_2,n_3,n_4\}$ resembles the low- or high-energy orbit $\{q_x,q_y,p_x,p_y\}$ it is forecasting, where color indicates the fourth dimension. NN hardly notices the differences between the low- or high-energy orbits.

Figure 8

Billiards potential energy. Potential energy versus position for circular billiards.

Figure 9

Billiard flows. Three sample billiard flows $\{q_x,q_y\}$ for different initial conditions, regular and bouncing from the outer circle only, regular and bouncing from both circles, and chaotic, forecast by conventional neural network (a), Hamiltonian neural network (b), and billiard differential equations (c). Hues code time, from red to violet. HNN physics bias significantly improves its dynamics forecasting over NN, which typically diverges after short times (and low-frequency colors).

Figure 10

Double pendulum. Relative energy error $\delta E/E$ for forecasted orbits versus number of training pairs $N$ for a double pendulum averaged over six different initial distributions of weights and biases for NN and HNN. Power-law fits guide the eye, and HNN's advantage grows with training pairs.