Renormalized Mutual Information for Artificial Scientific Discovery

We derive a well-defined renormalized version of mutual information that allows us to estimate the dependence between continuous random variables in the important case when one is deterministically dependent on the other. This is the situation relevant for feature extraction, where the goal is to produce a low-dimensional effective description of a high-dimensional system. Our approach enables the discovery of collective variables in physical systems, thus adding to the toolbox of artificial scientific discovery, while also aiding the analysis of information flow in artificial neural networks.

Figure 1

Feature extraction, where a high-dimensional “microscopic” description $x$ (such as the configuration of a many-particle system) is mapped to a low-dimensional feature $y=f(x)$. This is the case where the renormalized mutual information presented in this Letter is needed for feature optimization.

Figure 2

Comparing renormalized mutual information $\tilde{I}$ for several features in two representative physical scenarios. (a) Fluctuating 1D field on a lattice, with a randomly placed “wave packet” (we depict one single sample). (b) $\tilde{I}$ as a function of the size of the field fluctuations ${\sigma }_{\xi }$ for several features. Let $\overline{A_j}=1/N{\sum }_{j=1}^N{A}_j$. We consider the average field $f(x)=\overline{x_j}$, the position $j$ weighted by the field amplitude, $\overline{j{x}_j}$, or weighted by the field intensity, $\overline{j{x}_j^2}$, as well as the “normalized” feature $\overline{j{x}_j^2}/\overline{x_i^2}$ (similar to an expectation value in quantum mechanics) and the first PCA component. (c) Two-dimensional “drops” with elliptical shapes of fixed area but with fluctuating deformation amplitude $\delta r$ and orientation $\theta$ (we depict three samples). (d) $\tilde{I}$ vs max. deformation spread for the 2D feature given by PCA and for two nonlinear features sensitive to shape deformations, $f_{\mathrm{Var}}=[\overline{(x_j^{(1)}{)}^2},\overline{(x_j^{(2)}{)}^2}]$ and $f_{\text{Corr}}=[\overline{(x_j^{(1)}{)}^2},\overline{x_j^{(1)}{x}_j^{(2)}}]$, where $x_j^{(1)},x_j^{(2)}$ are the coordinates of particle $j$. In both (b) and (d) AE represents the bottleneck of a contractive autoencoder trained to reconstruct the input and NN corresponds to the feature given by a neural network optimized to maximize $\tilde{I}$. In the insets, we show the entropy $H\mathbf{(}f(x)\mathbf{)}$. This quantity is not reparametrization invariant.

Figure 3

Feature optimization and visual assessment of quality. (a) 2D non-Gaussian distribution. The obtained 1D feature $y=f(x_1,x_2)$, shown as contour lines atop the distribution $P_x(x)$, is parametrized with a neural network. Inset: PCA feature. (b) Wave packets as in Fig. 2, one by row, ordered by increasing value of the feature. The NN feature is clearly very powerful to sort the samples. (c) Liquid drops as in Fig. 2. We show how different 2D features map the deformation and the orientation of the drop. The NN builds up a representation very similar to our best handcrafted feature $f_{\text{Corr}}$.

Figure 4

Comparing the performance of a supervised regression task for different features as input. (a) For each batch of samples $x$ we calculate the feature $y=f(x)$ and train a supervised neural network to predict the provided label $z^{*}$. (b) Predicting the center of the wave packet [example from Fig. 2]. (c) Predicting the orientation and deformation of the drop [example from Fig. 2]. The optimized NN feature achieves the best performance in (b) and a performance very close to that of our best handcrafted feature (c).