Optimal Renormalization Group Transformation from Information Theory

Recently, a novel real-space renormalization group (RG) algorithm was introduced. By maximizing an information-theoretic quantity, the real-space mutual information, the algorithm identifies the relevant low-energy degrees of freedom. Motivated by this insight, we investigate the information-theoretic properties of coarse-graining procedures for both translationally invariant and disordered systems. We prove that a perfect real-space mutual information coarse graining does not increase the range of interactions in the renormalized Hamiltonian, and, for disordered systems, it suppresses the generation of correlations in the renormalized disorder distribution, being in this sense optimal. We empirically verify decay of those measures of complexity as a function of information retained by the RG, on the examples of arbitrary coarse grainings of the clean and random Ising chain. The results establish a direct and quantifiable connection between properties of RG viewed as a compression scheme and those of physical objects, i.e., Hamiltonians and disorder distributions. We also study the effect of constraints on the number and type of coarse-grained degrees of freedom on a generic RG procedure.

Popular Summary

Physical systems look very different when examined at a different length scale: The theory describing the chemical properties of water, for example, is distinct from that of the subnuclear particles comprising the molecules or the theory of surface waves on the water. The celebrated “renormalization group” (RG) framework establishes a conceptual link between such related descriptions and allows one to derive the effective theory governing behavior of a system at large distance scales, starting from microscopic models. However, the development of RG methods in real space, replicating the success of their momentum-space counterparts and applicable to disordered systems, has proven a serious challenge. We show that real-space RG can be understood from the fundamental perspective of information theory. This allows us to understand and potentially evade some of the problems that troubled those methods and paves the way for new numerical RG algorithms that employ methods of machine learning.

In our approach, we formulate RG as an optimization problem, which can be stated as compressing information about long-range physics in the most efficient way. We show that solutions achieving this information-theoretic optimality give rise to effective theories that are the simplest, in that their mathematical descriptions contain as few complicated or long-range interaction terms as possible. This holds true for clean and disordered systems, which we also demonstrate on model examples.

This abstract RG optimization problem can be addressed with numerical methods, including machine learning. Development of computationally efficient implementations should allow for the application of such techniques to more complex systems, particularly ones with correlated disorder, often encountered in soft matter physics.

Figure 1

Flow diagram of the RSMI algorithm [28]. Given a lattice and Hamiltonian $H$ (or, in practice, given Monte Carlo samples) a RBM Ansatz for the RG rule is optimized by maximizing the mutual information between the new d.o.f. $\mathcal{H}$ and the environment $\mathcal{E}$ of the original ones using stochastic gradient descent. The trained $P_{\mathrm{\Lambda }}(\mathcal{H}|\mathcal{V})$ is used to define a new effective measure and Hamiltonian $H_{\mathrm{eff}}$.

Figure 2

Schematic decomposition of the system for the purpose of defining the mutual information $I_{\mathrm{\Lambda }}(\mathcal{H}:\mathcal{E})$ (in 1D, for concreteness). The full system is partitioned into blocks of visibles $\mathcal{V}$ (yellow) embedded into a buffer $\mathcal{B}$ (blue) and surrounded by the environment $\mathcal{E}$ (green). The remaining part of the system is denoted by $\mathcal{O}$ in the main text. The conditional probability distribution $P_{\mathrm{\Lambda }}(\mathcal{H}|\mathcal{V})$ couples $\mathcal{V}$ to the hiddens $\mathcal{H}$ (red).

Figure 3

The relative difference between the renormalized NN coupling obtained from the cumulant expansion of the RSMI-favored solution (decimation) and the nonperturbative result Eq. (15). The convergence improves with increasing order of cumulant expansion $M$ and lower $K$.

Figure 4

Density plots of (a) the mutual information of the hiddens with the environment scaled to the mutual information of the visibles with the environment $I_{\mathrm{\Lambda }}(\mathcal{H}:\mathcal{E})/I(\mathcal{V}:\mathcal{E})$, (b) the ratio of the NNN to the NN coupling constants $|K_2^{\prime }(2)/K_2^{\prime }(1)|$, and (c) the ratio of the NN four-point to two-point coupling constants $|K_4^{\prime }(1,1,1)/K_2^{\prime }(1)|$. All three quantities are shown as a function of the parameters of the RG rule $\mathrm{\Lambda }=({\lambda }_1,{\lambda }_2)$. Note the inverted color scale in (b) and (c). For large enough $||\mathrm{\Lambda }|{|}^2$, a maximum of mutual information corresponds to a minimum of rangeness and $m$-bodyness, and vice versa. See the main text and Appendix pp4-s3 for details.

Figure 5

The two proxy measures of complexity of the renormalized Hamiltonian discussed in the text are shown against mutual information retained: the rangeness, i.e., the ratio of the NNN to the NN coupling constants, and the $m$-bodyness, i.e., the ratio of the NN four-point to two-point coupling constants. The mutual information is scaled to the total mutual information the block $\mathcal{V}$ shares with the environment. The curves are obtained by parametrizing the RG rule as $\lambda [\cos (\theta ),\sin (\theta )]$ and varying $\theta \in [0,\pi ]$ for different magnitudes of $\lambda$. In the physically relevant limit of large $\lambda$, the maximum of mutual information corresponds to a minimum of rangeness and $m$-bodyness. The plots are discussed in more detail in Appendix pp4-s3.

Figure 6

The mutual information $I_{\mathrm{\Lambda }}(\mathcal{H}:\mathcal{E})$ and $I({\mathcal{V}}_{\mathrm{\Lambda }}:\mathcal{E})$ for decimation (blue) and majority-rule (yellow) procedures in the 1D toy model Eq. (25). Two parameter regimes are shown: (a) strong coupling to the environment or low-temperature $K_{\mathcal{V}\mathcal{E}}$ (recall that the coupling constants contain a factor of $\beta =1/k_{B}T$) and (b) weak coupling $K_{\mathcal{V}\mathcal{E}}$; note that the absolute value of all mutual information is lower in this limit. The solid lines differ from the dashed lines of the same color by the mismatch $I({\mathcal{V}}_{\mathrm{\Lambda }}:\mathcal{E}|\mathcal{H})$ (see main text). In both parameter regimes, the dashed blue line exactly coincides with the solid blue line: For the decimation procedure, the information $I({\mathcal{V}}_{\mathrm{\Lambda }}:\mathcal{E})$ is perfectly encoded into $\mathcal{H}$. Majority rule $I_{\mathrm{\Lambda }}(\mathcal{H}:\mathcal{E})$ is inferior to decimation, even though $I({\mathcal{V}}_{\mathrm{\Lambda }}:\mathcal{E})$ is significantly larger: All that information is lost in encoding. The distinction between the two rules vanishes in the $K_{\mathcal{V}}\to \infty$ limit when both visible spins are effectively bound into a single binary variable.

Figure 7

Mutual information $I_{\mathrm{\Lambda }}(\mathcal{H}:\mathcal{E})$ in the toy model of a 2D system Eq. (26) as a function of the coupling $K_{\mathcal{V}}$ between the visibles. The coupling pattern to the visibles in different RG rules $\mathrm{\Lambda }$ is shown schematically in the (physically relevant) large-$||\mathrm{\Lambda }|{|}^2$ limit. The majority rule (and interestingly, also coupling to three spins depicted by a red line coinciding with the purple one) consistently retains more information than decimation or any coupling to two spins (blue and yellow coinciding with green). Again, the distinction vanishes for large $K_{\mathcal{V}}$ when all visible spins are bound into a single one.

Figure 8

Generic behavior of disorder probability under RG. Every point is a disorder realization (Hamiltonian). The shaded plane $K_0$ spanned by axes perpendicular to $K_0^{\perp }$ represents the subspace of two-body nearest-neighbor couplings; $K_0^{\perp }$ is the complement space of all other couplings. (a) Initially, a factorized distribution is usually assumed, schematically shown as a product of independent Gaussians. (b) After the RG step, the distribution can develop correlations depicted for simplicity in the $K_0$ plane, and additional couplings can be generated, resulting in probability mass shift out of $K_0$. The former effect is quantified by KL divergence or distance correlation, the latter by the shift of the center of mass of the distribution depicted with a green arrow (see also Fig. 9).

Figure 9

Properties of the renormalized disorder distribution as a function of the coarse-graining rule for the case of the random dilute Ising chain. The RG rules are parametrized by ${\lambda }_1$, ${\lambda }_2$, as before. (a) The distance correlation (see the main text) $\mathrm{dCor}$ between renormalized distributions of two neighboring NN couplings. The couplings are uncorrelated if and only if $\mathrm{dCor}$ vanishes; compare with Fig. 4. (b) An alternative measure of correlations is the Kullback-Leibler divergence ${\mathrm{D}}_{\mathrm{KL}}$ computed between the product of marginal distributions of neighboring couplings and their joint distribution (c) The $K_0^{\perp }$ center of mass of the disorder distribution $\mathrm{dCOM}$ (see the main text and Fig. 8). Note that all those quantities vanish when MI is maximized.

Figure 10

Logarithmic plot showing the exponential decay of the two-point correlator $K_2^{\prime }(\ell )$ with distance $\ell$ at $K=0.1$. The blue data points represent the results obtained from the cumulant expansion of the RSMI-favored solution up to tenth order, while the yellow line shows the exponential decay with decay length obtained from the first two points. For small $K$, where the cumulant expansion is expected to be accurate, the two-point correlator decays exponentially.

Figure 11

Logarithmic plot showing the exponential decay of the nearest-neighbor $m$-point correlator $K_m^{\prime }(1)$ with $m$ at $K=0.1$. The blue data points represent the results obtained from the cumulant expansion of the RSMI-favored solution up to tenth order, while the yellow line shows the exponential decay with decay length obtained from the first two points. Only points for even $m$ are present, as $K_m(1)=0$ for odd $m$ due to reasons of symmetry. Again, at small $K$, the two-point correlator decays exponentially.

Figure 12

The ratio between next-nearest-neighbor and nearest-neighbour coupling constants is plotted versus the mutual information for different orders of the cumulant expansion (see inset legend). The curves are obtained by parametrizing the RG rule by $({\lambda }_1,{\lambda }_2)=\lambda [\cos (\theta ),\sin (\theta )]$ and varying $\theta \in [0,\pi ]$.

Figure 13

The mutual information $I_{\mathrm{\Lambda }}(\mathcal{H}:\mathcal{E})$ and $I({\mathcal{V}}_{\mathrm{\Lambda }}:\mathcal{E})$ for decimation (blue) and majority-rule (yellow) procedures in the 1D periodic toy model Eq. (e1) obtained by coupling the environment spins $e_{1,2}$ in the model Eq. (25) with a coupling $K_{\mathcal{E}}=1.5$ (the value shown). Compare with Fig. 6, for which all other parameters are the same. Similarly, two parameter regimes are shown: (a) strong coupling to the environment or low-temperature $K_{\mathcal{V}\mathcal{E}}$ (recall that the coupling constants contain a factor of $\beta =1/k_{B}T$) and (b) weak coupling $K_{\mathcal{V}\mathcal{E}}$. The solid lines differ from the dashed lines of the same color by the mismatch $I({\mathcal{V}}_{\mathrm{\Lambda }}:\mathcal{E}|\mathcal{H})$ (see the main text). Note that the introduction of $K_{\mathcal{E}}$ coupling rendering the environment simply connected also in this 1D case greatly reduces the difference between the two RG rules.