Learning Interacting Theories from Data

One challenge of physics is to explain how collective properties arise from microscopic interactions. Indeed, interactions form the building blocks of almost all physical theories and are described by polynomial terms in the action. The traditional approach is to derive these terms from elementary processes and then use the resulting model to make predictions for the entire system. But what if the underlying processes are unknown? Can we reverse the approach and learn the microscopic action by observing the entire system? We use invertible neural networks to first learn the observed data distribution. By the choice of a suitable nonlinearity for the neuronal activation function, we are then able to compute the action from the weights of the trained model; a diagrammatic language expresses the change of the action from layer to layer. This process uncovers how the network hierarchically constructs interactions via nonlinear transformations of pairwise relations. We test this approach on simulated datasets of interacting theories and on an established image dataset (MNIST). The network consistently reproduces a broad class of unimodal distributions; outside this class, it finds effective theories that approximate the data statistics up to the third cumulant. We explicitly show how network depth and data quantity jointly improve the agreement between the learned and the true model. This work shows how to leverage the power of machine learning to transparently extract microscopic models from data.

Popular Summary

We use models to make sense of data: The more complicated the data, the more complicated the model needs to be. In the past, models could be designed only from the bottom up, explaining the behavior of the whole from the already known interactions between its parts. In this work, we flip the approach to work top down. Starting from a black-box model explaining the whole, we deconstruct it into simpler interactions between a few parts. For example, a set of images of the number “3” can be described by a small set of rules for how doublets, triplets, and quadruplets of pixels interact with each other.

Key to this approach is the use of a generative neural network, which maps a complicated data distribution to a simpler one. By decomposing this mapping into interactions between simpler features, we can better understand how and why models make predictions. We hence unravel the complex, hierarchical structure that has been learned by a neural network and explain it in a form that is central to physics: interactions between degrees of freedom.

Being able to provide an understanding of how neural networks behave is becoming increasingly important today, as applications of artificial intelligence—from medical diagnosis to face recognition—progressively impact human lives.

Figure 1

Learning actions from data. We observe a physical system of interacting degrees of freedom (gray dots), whose precise interactions are unknown (shaded areas). We train a neural network on measurements of the system. The network learns in an unsupervised fashion an estimate of the distribution of training data. We extract the action from the network parameters layer by layer, using a diagrammatic language. The final action coefficients $A^{(k)}$ represent the learned interactions (pink nodes).

Figure 2

Architecture and coefficient order. Invertible network composed of multiple layers. $L_1,\dots ,L_{L+1}$ denote linear fully connected layers, ${\varphi }_1,\dots ,{\varphi }_L$ are quadratic nonlinear activation functions. Coefficients $B_l^{(k)}$ of the action $S_{H,l}$ of preactivations $h_l$ in layer $l$ of order $k$; coefficients $A_l^{(k)}$ of the action $S_{X,l}$ prior to linear layer $l$ of order $k$.

Figure 3

Teacher-student coefficient comparison for varying training set sizes $D=|\mathcal{D}|$. Both teacher and student have depth $L=1$. (a)–(d) Student coefficients $A^{(k)}$ over teacher coefficients $T^{(k)}$ up to fourth order for $D={10}^3$ in green and $D={10}^5$ in pink. (e) Training loss (full lines) and test loss (dashed lines) over training steps. (f) Cosine similarity of coefficients over number of training samples.

Figure 4

Learning an effective monomodal theory. (a) Two-dimensional example of random density with multiple maxima. White lines are level lines of learned distribution for a five layer network. All other panels show results from a $d=10$ dimensional dataset. (b)–(d) Learned over true coefficients for a three layer network on a $d=10$. We distinguish diagonal tensor entries from off-diagonal ones, where at least two indices differ. (e)–(g) Learned over true cumulants, computed from samples. Error bars are typically smaller than marker size. (h) Dissimilarity of true and learned cumulants: $1-\cos \angle (⟪x^{\otimes k}{⟫}_A,⟪x^{\otimes k}{⟫}_R)$ over training steps. We record the cumulants at logarithmically spaced intervals during training. The curves are then smoothed by averaging over ten adjacent recording steps. Shaded areas show the variation due to the estimation of the cumulants from samples. Dots indicate training stage of cumulants shown in (e)–(g).

Figure 5

Symmetry broken lattice model for networks of varying depth trained on a $d={10}^2$ dimensional dataset with $D={10}^6$ samples. (a) Sites of square lattice with periodic boundary conditions distributed on a two-dimensional torus. Colored dots show strength of external field $h$ at connected lattice sites. (b) Learned over true first-order coefficients for network depth $L=3$. (c) Distribution of learned coefficient entries $A^{(2)}$ compared to target values (black crosses) for network depth $L=3$. We distinguish self-interaction terms $A_{\mathrm{diag}}^{(2)}$ from off-diagonal entries $A_{\mathrm{off}\text{−}\mathrm{diag}}^{(2)}$. From the off-diagonal entries $A_{\mathrm{off}\text{−}\mathrm{diag}}^{(2)}$, we further separate those entries belonging to adjacent lattice sites $A_{\mathrm{off}\text{−}\mathrm{diag}}^{(2),\mathrm{adj}}$. (d) Training loss (solid curves) and test loss (dashed curves). Colors distinguish different network depths $L$. (e) Distribution of learned fourth-order self-interactions as function of network depth. The dashed line marks the target value. (f)–(h) Learned over true cumulants of up to third order. Cumulants were computed on a subset of 10 randomly chosen lattice sites. Colors distinguish different network depths $L$.

Figure 6

Inference of interactions on MNIST for digit three. (a)–(c) Images from the dataset, the linear model, and an $L=1$ layer nonlinear model, respectively. (d)–(f) Single pixel activation statistics from three distinct locations in the image. (g) Entries of the mean ${\mu }_A$ of the Gaussian theory (linear model). (h) Entries on the diagonal of the second-order coefficient $A_{\mathrm{diag}}^{(2)}$ of the linear model. (i) Training loss (full lines) and test loss (dashed lines) over training steps. Dots mark the training stages from which the coefficients of both models were extracted. (j) Mean ${\mu }_A$ for the nonlinear model if $A^{(3)},A^{(4)}$ were not present. (k)–(m) Entries on the diagonals of the remaining coefficients of the $L=1$ layer nonlinear model. White squares in (l) mark the locations of the single pixel statistics shown in (d)–(f).

Figure 7

Three-point interactions in MNIST for digit three. (a) Histogram of all entries of the third-order coefficient $A_{ijk}^{(3)}$ for $i\ne j$, $j\ne k$, and $i\ne k$, color coded according to their value. (b),(c) Triplets corresponding to the ten most negative (b) or most positive (c) values of $A_{ijk}^{(3)}$. For each triplet, we color pixels $i$, $j$, and $k$, according to the value of the interaction coefficient in $A_{ijk}^{(3)}$. Thus triplets of pixels corresponding to the same entry in $A^{(3)}$ have the same color.

Figure 8

Eigenvalue distributions of decomposed ${\chi }_l$ for networks of different depths. We decompose trained network parameters ${\chi }_l$ from Sec. 4c to the form of Eq. (b2) and distinguish eigenvalues from the decomposed form of different layers $l$.

Figure 9

Dissimilarity of parameters. Stars show the cosine similarities of the teacher and student network parameters trained on varying dataset sizes $D$. The average cosine similarity between the teacher and ${10}^2$ randomly generated random networks is marked by the gray line, the shaded area encompasses one standard deviation. The remaining markers display the cosine similarity between the teacher coefficients $T^{(k)}$ and student coefficients $S^{(k)}$.

Figure 10

Multiple local maxima in $S_R$. (a) $S_R$ along the straight line connecting two local maxima $x_0^{*},x_1^{*}$ of $S_R$ found by the optimization algorithm. (b),(c) Eigenvalues of the Hessian $H$ of $S_R$ at local maxima $x_0^{*},x_1^{*}$. All eigenvalues ${\lambda }_{H(x_i^{*})}$ are negative; therefore the action is convex down in all directions.

Figure 11

Coefficients of lattice model for networks of varying depth trained on a $d=16$ dimensional dataset with $D={10}^5$ samples. (a) Learned ($L^{(1)})$ over true ($A^{(1)}$) first-order coefficients. (b) Distribution of learned coefficient entries $A^{(2)}$ compared to target values (black crosses). Self-interaction terms are labeled $A_{\mathrm{diag}}^{(2)}$, off-diagonal entries $A_{\mathrm{off}\text{−}\mathrm{diag}}^{(2)}$. Among the off-diagonal entries $A_{\mathrm{off}\text{−}\mathrm{diag}}^{(2)}$, entries belonging to adjacent lattice sites $A_{\mathrm{off}\text{−}\mathrm{diag}}^{(2),\mathrm{adj}}$ are shown separately. (c) Training loss (full curves) and test loss (dashed curves). Colors distinguish different network depths $L$. (d) Distribution of learned fourth-order self-interactions over network depth. The dashed line marks the target value. (e) Dissimilarity of true and learned cumulants: $1-\cos \angle (⟪x^{\otimes k}{⟫}_A,⟪x^{\otimes k}{⟫}_R)$ over training steps. We record the cumulants at logarithmically spaced intervals during training. The curves are smoothed by averaging over ten adjacent recording steps. Shaded areas show the variation due to the estimation of the cumulants from samples. Dots indicate training stage of coefficients shown in (a) and (b).

Figure 12

Coefficients of lattice model without external field for networks of varying depth trained on a $d=9$ dimensional dataset with $D={10}^5$ samples. (a) Distribution of learned coefficient entries $A^{(1)},A^{(2)}$ compared to target values (black crosses). Self-interaction terms $A_{\mathrm{diag}}^{(2)}$ are shown separately from off-diagonal entries $A_{\mathrm{off}\text{−}\mathrm{diag}}^{(2)}$. Among the off-diagonal entries $A_{\mathrm{off}\text{−}\mathrm{diag}}^{(2)}$, those entries belonging to adjacent lattice sites $A_{\mathrm{off}\text{−}\mathrm{diag}}^{(2),\mathrm{adj}}$ are shown separately. (b) Distribution of learned fourth-order self-interactions compared to network depth. The dashed line marks the target value. (c) Training loss (full curves) and test loss (dashed curves). Colors distinguish different network depths $L$. (d) Dissimilarity of true and learned cumulants: $1-\cos \angle (⟪x^{\otimes k}{⟫}_A,⟪x^{\otimes k}{⟫}_R)$ over training steps. We record the cumulants at logarithmically spaced intervals during training. The curves are smoothed by averaging over ten adjacent recording steps. Shaded areas show the variation due to the estimation of the cumulants from samples. Dots indicate training stage of coefficients shown in (a) and (b).

Figure 13

Inference of interactions on MNIST for digit two. (a)–(c) Images from the dataset, the linear model, and an $L=1$ layer nonlinear model, respectively. (d)–(f) Single pixel activation statistics from three distinct locations in the image. (g) Entries of the mean ${\mu }_A$ of the Gaussian theory (linear model). (h) Entries on the diagonal of the second-order coefficient $A_{\mathrm{diag}}^{(2)}$ of the linear model. (i) Training loss (full lines) and test loss (dashed lines) over training steps. Dots mark the training stages from which the coefficients of both models were extracted. (j) Mean ${\mu }_A$ for the nonlinear model if $A^{(3)},A^{(4)}$ were not present. (k)–(m) Entries on the diagonals of the remaining coefficients of the $L=1$ layer nonlinear model. White squares in (l) mark the locations of the single pixel statistics shown in (d)–(f).

Figure 14

Three-point interactions in MNIST for digit two. (a) Histogram of all entries of the third-order coefficient $A_{ijk}^{(3)}$ for $i\ne j$, $j\ne k$, and $i\ne k$, color coded according to their value. (b),(c) Triplets corresponding to the ten most negative (b) or most positive (c) values of $A_{ijk}^{(3)}$. For each triplet, we color pixels $i$, $j$, and $k$, according to the value of the interaction coefficient in $A_{ijk}^{(3)}$. Thus triplets of pixels corresponding to the same entry in $A^{(3)}$ have the same color.

Figure 15

Three-point interactions in MNIST for digit three. (a) Histogram of all entries of the third-order coefficient $A_{ijk}^{(3)}$ for $i\ne j$, $j\ne k$, and $i\ne k$, color coded according to their value. Panels (b) and (c) show triplets corresponding to the ${10}^2$ most negative (b) or most positive (c) values of $A_{ijk}^{(3)}$. For each triplet, we color pixels $i$, $j$, and $k$, according to the value of the interaction coefficient in $A_{ijk}^{(3)}$. Thus triplets of pixels corresponding to the same entry in $A^{(3)}$ have the same color.

Figure 16

Three-point interactions in MNIST for digit two. (a) Histogram of all entries of the third-order coefficient $A_{ijk}^{(3)}$ for $i\ne j$, $j\ne k$, and $i\ne k$, color coded according to their value. Panels (b) and (c) show triplets corresponding to the ${10}^2$ most negative (b) or most positive (c) values of $A_{ijk}^{(3)}$. For each triplet, we color pixels $i$, $j$, and $k$, according to the value of the interaction coefficient in $A_{ijk}^{(3)}$. Thus triplets of pixels corresponding to the same entry in $A^{(3)}$ have the same color.