Symbolic pregression: Discovering physical laws from distorted video

We present a method for unsupervised learning of equations of motion for objects in raw and optionally distorted unlabeled synthetic video (or, more generally, for discovering and modeling predictable features in time-series data). We first train an autoencoder that maps each video frame into a low-dimensional latent space where the laws of motion are as simple as possible, by minimizing a combination of nonlinearity, acceleration, and prediction error. Differential equations describing the motion are then discovered using Pareto-optimal symbolic regression. We find that our pre-regression (“pregression”) step is able to rediscover Cartesian coordinates of unlabeled moving objects even when the video is distorted by a generalized lens. Using intuition from multidimensional knot theory, we find that the pregression step is facilitated by first adding extra latent space dimensions to avoid topological problems during training and then removing these extra dimensions via principal component analysis. An inertial frame is autodiscovered by minimizing the combined equation complexity for multiple experiments.

Figure 1

Our pregression algorithm seeks to autoencode a sequence of video frames (left) corresponding to a specific type of motion into a low-dimensional latent space (middle) where the laws of motion (right) are as simple as possible, in this example those of a quartic oscillator. In the middle figure each point corresponds to the $x$ and $y$ of the rocket in a given frame, while points having the same color or shading and being connected by a line belong to the same trajectory.

Figure 2

Architecture of our neural network: An encoder $E$ that maps images ${\mathbf{x}}_i$ into latent space vectors ${\mathbf{z}}_i$, a decoder $D$ that maps latent space vectors ${\mathbf{z}}_i$ back into images ${\mathbf{x}}_i$, and an evolution operator $U$ that predicts the next latent space vector from the two previous ones.

Figure 3

Our method can rediscover simple laws of motion even if the true images (left) are severely warped (middle) or corrupted by superimposed noise in the form of smaller distractor rockets (right).

Figure 4

Example of original (top) and rediscovered (bottom) trajectories in the latent spaces. In the top panel, each point represents the $x$ and $y$ coordinates of the rocket in each frame. In the bottom panel, each point corresponds to the two main principal components discovered in the 5D latent space. In both cases, points of the same color or shading and connected by a line belong to the same trajectory.

Figure 5

Example of original, downsampled, and reconstructed image.

Figure 6

The topological problems (middle) that prevented directly learning a 2-dimensional latent space (left) can be understood via knot theory and eliminated by instead discovering the two main principal components (right) in a learned 5-dimensional latent space. The left panel shows 16 force-free rocket trajectories, with points denoting the rocket center in each frame and points of the same color or shading corresponding to the same trajectory. The middle panel shows the corresponding points ${\mathbf{z}}_i$ produced by our encoder network trained with a 2D latent space. The right panel shows the eigenvalues obtained from a PCA on a five-dimensional learned latent space, revealing that the latent space is rather 2-dimensional because two principal components account for most of the variance.

Figure 7

Video trajectories in the 2D harmonic oscillator potential (top) look differently when mapped into the latent space using the encoder trained on that same data (right) than when mapped using the encoder trained on uniform force-free motion (left). However, the two latent spaces are equivalent up to an affine transformation.

Figure 8

Left: The five latent spaces can be unified by using affine transformations. The complexity of the equations in this unified space is minimized when having no further shift (2nd panel) or rotation (3rd panel), but a $\sim 3\%$ shear (right).