Machine learning of Kondo physics using variational autoencoders and symbolic regression

We employ variational autoencoders to extract physical insight from a dataset of one-particle Anderson impurity model spectral functions. Autoencoders are trained to find a low-dimensional, latent space representation that faithfully characterizes each element of the training set, as measured by a reconstruction error. Variational autoencoders, a probabilistic generalization of standard autoencoders, further condition the learned latent space to promote highly interpretable features. In our study, we find that the learned latent variables strongly correlate with well known, but nontrivial, parameters that characterize emergent behaviors in the Anderson impurity model. In particular, one latent variable correlates with particle-hole asymmetry, while another is in near one-to-one correspondence with the Kondo temperature, a dynamically generated low-energy scale in the impurity model. Using symbolic regression, we model this variable as a function of the known bare physical input parameters and “rediscover” the nonperturbative formula for the Kondo temperature. The machine learning pipeline we develop suggests a general purpose approach, which opens opportunities to discover new domain knowledge in other physical systems.

Figure 1

(a) The variational autoencoder architecture. Spectral functions $\mathit{x}$ on a pointwise grid are fed into the encoder and compressed into parameters ${\mu }_{\mathit{x}},{\sigma }_{\mathit{x}}$ of an $L$-dimensional Gaussian distribution in latent space. Latent activations $\mathit{z}$ are then sampled from $\mathit{z}\sim \mathcal{N}({\mu }_{\mathit{x}},\mathrm{diag}\left({\sigma }_{\mathit{x}}^2\right))$ before being fed into the decoder, which reconstructs the input spectrum. The contribution to the loss from a given $\mathit{x}, {\mathcal{L}}_{\mathit{x}}$ is shown in brief, with arrows noting which components of pipeline contribute. (b) A schematic showing that increasing $\lambda$ structures the data distribution in latent space such that the $z_i$ become statistically independent, and aligns underlying generative factors (red/blue coloring) with the latent axes.

Figure 2

(a) Final ${\mathcal{L}}_{\mathrm{RL}}$ reconstruction and ${\mathcal{L}}_{\mathrm{KLD}}$ regularization losses of $L=10$ VAEs as the regularization strength $\lambda$ increases. The highlighted yellow region is a visual aid showing the “critical region” of models with good ${\mathcal{L}}_{\mathrm{RL}}-{\mathcal{L}}_{\mathrm{KLD}}$ tradeoffs. The green circle marks the center of this critical region identifiable as an inflection point in log-log space (see Appendix pp1), while the cyan circle marks models with stronger regularization, which are presented in this work. [(b),(c)] Average KL loss ${\mathcal{D}}_{\mathrm{KL}}^{\left(i\right)}$ contributed by each individual latent variable $z_i$ as $\lambda$ increases, for models trained on (b) the $T_K$-dominated and (c) the full-parameter datasets. Latent variables are ordered by descending ${\mathcal{D}}_{\mathrm{KL}}^{\left(i\right)}$ loss. Values shown are from the models with the fewest active $z_i$ across three training runs at each $\lambda$.

Figure 3

Thick blue curves show reconstructed spectral functions corresponding to different points in the two-dimensional latent space learned by a $\lambda =0.5$ VAE on the $T_K$-dominated dataset. Thin dashed black curves show real spectral functions in the dataset whose compressed latent representations are closest to the sampled latent space point. Curves are plotted on the log-linear frequency grid (see Appendix pp1) as they are fed into the VAE.

Figure 4

Latent space learned by a $\lambda =0.5$ VAE on the $T_K$-dominated dataset, colored by (a) ${log}_{10}\left(T_K\right)$ and (b) $(U+2{\epsilon }_d)/\mathrm{\Gamma }$. The blue-open circles mark the points in latent space corresponding to the shown spectral functions. In (c) and (d), correlations between the principal components and $log{T}_K,(U+2{\epsilon }_d)/\mathrm{\Gamma }$ are plotted, whose strengths are quantified by Pearson correlation coefficients $\left|\rho \right|\le 1$, which are measures of linear correlation.

Figure 5

Reconstructed spectral functions from a $\lambda =0.3$ VAE trained on the full five-parameter dataset. Each row corresponds to a line scan in latent space where all $z_i$ are held at 0 except for one.

Figure 6

Latent space learned by the VAE on the full five-parameter dataset using $\lambda =0.3$. The data in [(a)–(c)] is precisely the same, except that the color coding differs based on the chosen physical parameter: (a) $(U+2{\epsilon }_d)/\mathrm{\Gamma }$; (b) ${log}_{10}{\mathrm{E}}_0={log}_{10}max(T_K,T,\left|B\right|)$; (c) dominant energy scale out of $\{T_K,T,B\}$. Within each cluster, points are darker if the relative factor to the next-highest energy scale is larger. Supplementary views from different angles are provided in Fig. 12.

Figure 7

(a) Chosen paths in the latent space of the VAE trained on the $T_K$-dominated dataset. Each highlighted point corresponds to a single spectral function in the dataset. (b) The same spectral functions mapped into the latent space learned by the VAE trained on the full-parameter dataset, colored as in Fig. 6. Supplementary view angles are provided in Fig. 12.

Figure 8

Results of applying symbolic regression to learn functions describing each latent variable of the VAE trained on the $T_K$-dominated dataset (Fig. 4). (a) Symbolic regression population fitness and (b) program length as the algorithm learns $f_1^{*}$. [(c),(d)] Obtained symbolic syntax trees and equivalent functions, which maximize correlation with the active latents $z_i$. [(e),(f)] Scatterplots showing $f_i^{*}$ vs $z_i$ for each latent dimension, analogous to Figs. 4 and 4. Colorbars show the density of data at each point.

Figure 9

An example of the coarse-grain mapping from $A\left(\omega \right)$ to $\mathit{x}$. The left panel shows the original spectral function on a linear $\omega$ axis. The middle panel showcases how the same spectrum distorts under the linear-logarithmic ${\omega }_i$ grid, forming the input $\mathit{x}$ to the VAE. The right panel shows the frequency values for the 333 coordinates with the exception of the single bridge point at $\omega \sim {10}^{-11}$.

Figure 10

The same performance measurements and region highlights as in Fig. 2, but on a log-log scale. Here, we can identify the “critical region” as a pronounced inflection in the ${\mathcal{L}}_{\mathrm{RL}}-{\mathcal{L}}_{\mathrm{KLD}}$ curve.

Figure 11

Additional performance measurements. (a) Final reconstruction performance of unregularized $(\lambda =0)$ VAEs as the latent dimension $L$ is varied. (b) Measurements of our disentangling metric upon increasing $\lambda$, showing that the ${\mathcal{D}}_{\mathrm{KL}}$ loss effectively drives learned features to be statistically independent. The cyan dashed line marks $\lambda =0.5$.

Figure 12

Additional view angles of each plane in the three-dimensional latent space learned by the VAE on the full five-parameter dataset in the main text, with the same colorings as in Fig. 6.

Figure 13

Latent spaces of two additional randomly initialized, independently trained $\lambda =0.3$ VAEs on the full-parameter dataset, with coloring as in Fig. 6. Both VAEs displayed here have a higher total loss than the one presented in the main text.

Figure 14

Results of applying principal component analysis (PCA) directly to the spectral functions in the full five-parameter dataset. (a) Explained variance, defined as the fraction of the variance along each PCA axis compared to the sum of all variances, for each discovered principal component. (b) The first four principal component vectors. [(c)–(e)] Scatter plots of projections of the dataset onto the first three principal components, colored respectively by the particle-hole asymmetry, ${log}_{10}{E}_0$, and the dominant energy scale [with color coding as in Fig. 6].

Figure 15

Like Fig. 14, but with a frequency-dependent normalization performed before PCA. [(a)–(e)] are as in Fig. 14. (f) An alternate 3D projection, showing that PCA4 roughly captures ${log}_{10}{E}_0$. [(g),(h)] Scatterplots showing correlation between ${log}_{10}{E}_0$ and either PCA4 or the VAE's latent $z_3$ (introduced in the main text), respectively.

Figure 16

Line scans along each latent variable $z_i$ of VAEs trained on the full five-parameter dataset for smaller regularization strengths than the $\lambda \approx 0.5$ examined in the main text having (a) $\lambda =0.002$ and (b) $\lambda =6.25\times {10}^{-5}$. The latent variables $z_i$ have been sorted according to decreasing KL loss, as in the main text. Red dashed lines mark the log-to-linear frequency grid crossover points described in [A 1]. Both models can be seen to recover finer peak details as compared to Fig. 5, though are not quite as well disentangled.