Large pseudocounts and $L_2$-norm penalties are necessary for the mean-field inference of Ising and Potts models

The mean-field (MF) approximation offers a simple, fast way to infer direct interactions between elements in a network of correlated variables, a common, computationally challenging problem with practical applications in fields ranging from physics and biology to the social sciences. However, MF methods achieve their best performance with strong regularization, well beyond Bayesian expectations, an empirical fact that is poorly understood. In this work, we study the influence of pseudocount and $L_2$-norm regularization schemes on the quality of inferred Ising or Potts interaction networks from correlation data within the MF approximation. We argue, based on the analysis of small systems, that the optimal value of the regularization strength remains finite even if the sampling noise tends to zero, in order to correct for systematic biases introduced by the MF approximation. Our claim is corroborated by extensive numerical studies of diverse model systems and by the analytical study of the $m$-component spin model for large but finite $m$. Additionally, we find that pseudocount regularization is robust against sampling noise and often outperforms $L_2$-norm regularization, particularly when the underlying network of interactions is strongly heterogeneous. Much better performances are generally obtained for the Ising model than for the Potts model, for which only couplings incoming onto medium-frequency symbols are reliably inferred.

Figure 1

Coupling $J^{\mathrm{MF}}$ inferred with the mean-field approximation (12) overestimates the true coupling value $J$ (dotted line) in a system of two spins when $J$ is large (in absolute value).

Figure 2

Coupling inferred with the mean-field approximation with (a) pseudocount of intensity $\alpha$, $J^{\mathrm{PC}}$; (b) with $L_2$-norm regularization of intensity $\gamma$, $J^{L_2}$; (c) with the $L_1$-norm regularization of intensity $\gamma$, $J^{L_1}$, vs true coupling value $J$ (dotted line) for a system of two spins. Due to the symmetry $J\to -J$ only the positive quadrant is shown.

Figure 3

Relative squared error $\epsilon (B,J,\alpha )$ (18) between the inferred coupling with pseudocount of intensity $\alpha$ and the true coupling for a system of two spins as a function of the number $B$ of sampled configurations. Each panel corresponds to one value of $\alpha$ and each curve to one value of $J$. Pseudocount decreases the error on inferred couplings even when the couplings are weak, or sampling is large, but regularization that is too large leads to worse performance.

Figure 4

Mean-field prediction for the coupling ${\widehat{J}}_1$ as a function of the pseudocount strength $\alpha$ (log-log scale). The true interaction network connects three spins, with two couplings equal to ${\widehat{J}}_0$ (values shown) and one vanishing coupling ${\widehat{J}}_1$. The vertical dashed line locates ${\alpha }^{\mathrm{MF}}\simeq 0.2$.

Figure 5

Relative squared error over the couplings with (a) uniform $L_2$-norm penalty and (b) uniform pseudocount for a system of three spins, see text. The dotted lines in (b) reproduce the relative squared errors found for the $L_2$-norm regularization in (a). Note the change in the vertical axis scale.

Figure 6

Homogeneous $q$-symbol Potts model. Couplings inferred with the mean-field approximation, $J_A^{\mathrm{MF}}(J_0,q)$ (diagonal) and $J_B^{\mathrm{MF}}(J_0,q)$ (off-diagonal), vs true values coupling $J_A$ and $J_B$ in (23). Curves were obtained through a parametric representation with $J_0$ running from $-$4 to 4; longer stretches would be obtained by increasing the range of values for $J_0$. The dashed line represents the $J^{\mathrm{MF}}=J$ curve. (a) No pseudocount. (b) With a pseudocount of strength $\alpha$ (see values in the figure) and for $q=5$ symbols.

Figure 7

Optimal pseudocount strength ${\alpha }^{\mathrm{MF}}\left(q\right)$ as a function of the number of Potts symbols, $q$. The dotted line serves as a guide to the eye.

Figure 8

Heterogeneous $q$-symbol Potts model. Couplings inferred with the mean-field approximation, $J_L^{\mathrm{MF}}(J_0,J_1,q=5)$ (left) and with a pseudocount $J_L^{\mathrm{PC}}(J_0,J_1,q=5,\alpha =0.4)$ (right), vs true couplings $J_L(J_0,J_1,q)$; $L=A,B,C,D$ labels the four different branches. The value of the pseudocount strength $\alpha =0.4$ has been chosen according to Fig. 7. Curves were obtained through a parametric representation with $J_0$ running from $-$8 to 8; longer stretches would be obtained by increasing the range of values for $J_0$. Top: “Weak” bias ($J_1=3$); bottom: “strong” bias ($J_1=5$).

Figure 9

Representation of tested model parameters. Fields and nonzero couplings were selected according to model (a), all fields and couplings normally distributed with means $\overline{h}=0$, $\overline{J}=0$ and standard deviations ${\sigma }_h$, ${\sigma }_J$, respectively, or model (b), strong negative fields and couplings normally distributed with means $\overline{h}=-5$, $\overline{J}=1$ and standard deviations ${\sigma }_h$, ${\sigma }_J$. Distributions for fields are shaded dark, values for couplings are light. (c) For each model a range of ${\sigma }_h$ (dark) and ${\sigma }_J$ (light) was tested. (d) Correlations used for the MF inference were computed using $B$ samples from a Monte Carlo simulation of the model, with $B$ tested over a range from 500 to ${10}^6$. All permutations of the above parameters were considered for each choice of the network topology: (e) Erdős-Rényi graph where edges are kept with probability $p=2/N$ or $p=4/N$ and (f) $1D$ lattice with nearest-neighbor couplings. In all cases we take the system size $N=100$.

Figure 10

Typical example trajectory of inference quality as a function of the regularization strength for a single randomly chosen set of couplings and fields. At larger regularization strengths the inferred couplings are similar in magnitude to the true couplings, but at lower regularization strengths the value of the strongest inferred couplings begins to diverge. The true model is a 1D spin chain, with zero fields and couplings normally distributed with mean zero and standard deviation ${\sigma }_J=3$. $B={10}^5$ Monte Carlo samples were used to compute the correlations used for the MF inference. (a) rms error ${\Delta }_J$ (27) (circles, left axis), rank correlation ${\rho }_J$ (28) (squares, right axis), and fraction of nonzero couplings recovered $R$ (triangles, right axis), as a function of the pseudocount $\alpha$. [(b1), (b2), and (b3)] Comparison of true and inferred couplings for the three pseudocount values ${\alpha }_1,{\alpha }_2,{\alpha }_3$ shown in panel (a) by dashed vertical lines. Dashed lines mark the $J^{\inf }=J^{\mathrm{true}}$ lines. The $N$ largest (in absolute value) inferred couplings are denoted by open circles and others are denoted by squares; the cut-off number $N$ is chosen to match the number of nonzero couplings on the true interaction graph. Note the change of vertical scale between panels. The performance is maximal for $\alpha ={\alpha }_1\approx 0.2$. This value of the pseudocount agrees well with the optimal pseudocount for the Ising model ${\alpha }^{\mathrm{MF}}=0.2$ shown in Fig. 7. The rank correlation is worse for $\alpha ={\alpha }_2\approx 0.002$, as the inference procedure assigns large values to couplings equal to zero on the true interaction network, see Sec. 3c. For $\alpha ={\alpha }_3=2{10}^{-5}\simeq {\alpha }_B$ those fake couplings have essentially disappeared; the inferred couplings have much larger values than the true couplings but are correctly ordered (large rank correlation).

Figure 11

Optimal values of the regularization strength are only weakly affected by sampling depth, even when varied over the full range from $B=500$ (lightest) to $B={10}^6$ (darkest). Trajectory of the rms error ${\Delta }_J$ (27) (top), rank correlation ${\rho }_J$ (28) (middle), and fraction of nonzero couplings recovered $R$ (bottom) as the pseudocount $\alpha$ (a) and $L_2$-norm regularization strength $\gamma$ (b) is varied, averaged over ${10}^3$ sets of random couplings. Relevant values of the pseudocount are $\alpha ={\alpha }^{\mathrm{MF}}=0.2$ and ${\alpha }^B=1/B$, roughly where ${\rho }_J$ and $R$ begin to plateau for the pseudocount. Each random set of interactions has all fields set to zero. The coupling network is an Erdős-Rényi graph ($N=100$ nodes) where edges are kept with probability $p=2/N$. Nonzero couplings are normally distributed with mean zero and standard deviation ${\sigma }_J=3$. Bars denote one half standard deviation over the sample.

Figure 12

Varying the strength of the underlying interactions can shift the optimal value of the regularization strength. Trajectory of the rms error ${\Delta }_J$ (27) (top), rank correlation ${\rho }_J$ (28) (middle), and fraction of nonzero couplings recovered $R$ (bottom) as the pseudocount $\alpha$ (a) and $L_2$-norm regularization strength $\gamma$ (b) is varied, averaged over ${10}^3$ sets of random couplings, over a range of coupling distribution widths ${\sigma }_J=1$ (light), ${\sigma }_J=2$ (medium), and ${\sigma }_J=3$ (dark). Relevant values of the pseudocount are $\alpha ={\alpha }^{\mathrm{MF}}=0.2$ and ${\alpha }^B=1/B={10}^{-4}$, roughly where ${\rho }_J$ and $R$ plateau for the pseudocount. Each random set of interactions has all fields set to zero. The coupling network is an Erdős-Rényi graph ($N=100$ nodes) where edges are kept with probability $p=2/N$. Nonzero couplings are normally distributed with mean zero and standard deviation ${\sigma }_J$. MF couplings were inferred from correlations computed from $B={10}^4$ Monte Carlo samples of the true model. Bars denote one half standard deviation over the sample.

Figure 13

Performance of the pseudocount (a) and $L_2$-norm regularization (b) differ as a function of the regularization strength, particularly in the behavior of the rank correlation. The rms error ${\Delta }_J$ (27) (circles, left axis), rank correlation ${\rho }_J$ (28) (squares, right axis), and fraction of nonzero couplings recovered $R$ (triangles, right axis), as a function of pseudocount $\alpha$ and $L_2$-norm regularization strength $\gamma$, averaged over ${10}^3$ sets of random couplings. Dashed lines in (a) mark $\alpha ={\alpha }^{\mathrm{MF}}=0.2$ and ${\alpha }^B=1/B={10}^{-3}$, roughly where ${\rho }_J$ and $R$ plateau for the pseudocount. Each random set of interactions has all fields $h=-5$. The coupling network is an Erdős-Rényi graph where edges are kept with probability $p=4/N$. Nonzero couplings are normally distributed with mean $\overline{J}=1$ and standard deviation ${\sigma }_J=2$. MF couplings were inferred from correlations computed from $B={10}^3$ Monte Carlo samples of the true model. Bars denote one half standard deviation over the sample.

Figure 14

Scatter plot of the couplings $J_{ij}(a,b)$ for the homogenous Potts model for $q=5$ (filled circles) and for $q=21$ (triangles) symbols; perfect sampling. (a) No pseudocount. (b) With pseudocount. Each panel shows results from three realizations with different sets of couplings ($L=10$). Black solid lines correspond to the analytical predictions of Sec. 3e1. Colors show values of $p_i^a{p}_j^a$, here equal to $1/q^2$ for all interacting sites and for all symbols, see right scale.

Figure 15

Scatter plot of the couplings $J_{ij}(a,b)$ for the heterogenous-A Potts model for $q=5$ symbols, with perfect sampling. (a) No pseudocount. (b) With pseudocount ($\alpha =0.4$). Each panel shows results from five realizations with different sets of couplings and fields ($L=2$). Insets: Distributions of the frequencies $p_i^a$. Black solid lines correspond to the analytical predictions of Sec. 3e1. Colors show values of $p_i^a{p}_j^a$, see right scale.

Figure 16

Scatter plot of the couplings $J_{ij}(a,b)$ for the heterogenous-B Potts model for $q=5$ symbols, with perfect sampling. (a) No pseudocount. (b) With pseudocount ($\alpha =0.4$). Each panel shows results from five realizations with different sets of couplings and fields ($L=2$). Insets: Distributions of the frequencies $p_i^a$. Black solid lines correspond to the analytical predictions of Sec. 3e1. Colors show values of $p_i^a{p}_j^a$, see right scale.

Figure 17

Heterogenous-B Potts model for $q=5$ symbols, for various depths of sampling. (a) Small pseudocount $\alpha ={\alpha }^B=\frac{1}{B}$. (b) Large (optimal) pseudocount $\alpha ={\alpha }^{\mathrm{MF}}=0.4$. Each panel shows results from one realization of the Potts model with random couplings and fields ($L=2$) and three sets of $B$ sampled configurations (for finite $B$).

Figure 18

Scatter plot of the Frobenius norms of the inferred couplings vs their true values for the pseudocount strengths $\alpha =\frac{1}{B}$ (a) and $\alpha =0.4$ (b). Same heterogeneous-B model and same conditions as in Fig. 17. Lines locate the largest Frobenius norm corresponding to a pair of sites $(i,j)$ which are not neighbors on the one-dimensional graph, i.e., which have zero true coupling.