Network inference in stochastic systems from neurons to currencies: Improved performance at small sample size

The fundamental problem in modeling complex phenomena such as human perception using probabilistic methods is that of deducing a stochastic model of interactions between the constituents of a system from observed configurations. Even in this era of big data, the complexity of the systems being modeled implies that inference methods must be effective in the difficult regimes of small sample sizes and large coupling variability. Thus, model inference by means of minimization of a cost function requires additional assumptions such as sparsity of interactions to avoid overfitting. In this paper, we completely divorce iterative model updates from the value of a cost function quantifying goodness of fit. This separation enables the use of goodness of fit as a natural rationale for terminating model updates, thereby avoiding overfitting. We do this within the mathematical formalism of statistical physics by defining a formal free energy of observations from a partition function with an energy function chosen precisely to enable an iterative model update. Minimizing this free energy, we demonstrate coupling strength inference in nonequilibrium kinetic Ising models, and show that our method outperforms other existing methods in the regimes of interest. Our method has no tunable learning rate, scales to large system sizes, and has a systematic expansion to obtain higher-order interactions. As applications, we infer a functional connectivity network in the salamander retina and a currency exchange rate network from time-series data of neuronal spiking and currency exchange rates, respectively. Accurate small sample size inference is critical for devising a profitable currency hedging strategy.

Figure 1

Network inference for the kinetic Ising model. Inference mean-square error (MSE, black) and discrepancy $(D$, gray) are shown as functions of the number of iterations for large observed configurations, $L/N^2=1$ (a) and few observed configurations, $L/N^2=0.2$ (c). Predicted couplings vs actual couplings for $L/N^2=1$ (b) and $L/N^2=0.2$ (d). The inference errors are obtained for naïve mean-field (nMF), Thouless-Anderson-Palmer (TAP), exact mean-field (eMF), maximum likelihood estimation (MLE), and free energy minimization (FEM), for various numbers of observed configurations, $L/N^2$ from 0.2 to 1 in the limit of weak coupling, $g=1$ (e), and in the limit of stronger coupling, $g=2$ (f), $g=3$ (g), and $g=4$ (h). A system size $N=100$ is used. A learning rate $\alpha =1$ is used for MLE.

Figure 2

Efficiency of inference. Number of iterations per spin (a) and real computational time (b) by using MLE vs FEM for various coupling strengths $g$ from 1 to 4 and number of observed configurations $L/N^2$ from 0.2 to 1. A system size $N=100$ is used. A learning rate $\alpha =1$ is used for MLE.

Figure 3

Effectiveness of FEM in inferring network with specific structures. Given true coupling weights of $N=40$ (a) and 350 (e) spin variables with non-Gaussian distributions, typical time series of their activities are generated, (c) and (g). Predicted coupling weights are obtained for different data lengths $L/N^2=0.5,1$, and 4 from left to right, (d) and (h). The image is converted from the photograph of the 2018 Gerber baby, Lucas Warren (with permission from Gerber).

Figure 4

Accurate inference of higher-order coupling strengths. Linear (a) and quadratic (b) coupling strengths in the nonlinear kinetic Ising model are predicted from FEM. Here the true coupling strengths are normally distributed with a system size $N=40$. Three different data lengths, $L=1.6\times {10}^4$ (gray), $6.4\times {10}^4$ (blue), and $2.56\times {10}^5$ (red), are examined.

Figure 5

Inference of coupling strengths between neurons, external local fields, and neuronal activities. From activities of 100 neurons (a), the neuronal network (b) and external local field $H_i^{\mathrm{ext}}$ (c) are predicted. The red and blue edges represent positive and negative couplings, respectively. Edge direction is clock-wise. Inferred correlation covariances $C_{ij}$ are compared with actual correlation covariances $C_{ij}^{\text{true}}$ (d). Inference accuracy of remaining neuronal activities vs number of input neurons selected based on large $|W_{ij}|$ (filled black circles), and randomly selected (empty blue circles). Error bars represent the standard deviation from 50 random trials (e). Neuronal activities are reconstructed with 10 input neurons, selected based on large $|W_{ij}|$ (f), and randomly selected (g).

Figure 6

Inference of coupling strengths between currency exchange rates. Normalized exchange rates relative to EUR of 11 currencies are plotted with different colors representing distinct currencies (a). A raster representation of binarized exchange rate fluctuations is plotted with black dots representing increase, white dots decrease. Average power spectrum obtained from a Fourier transform of exchange rate fluctuations vs time-window size in which error bar represents standard deviation from different currencies (c). The currency networks are predicted for different periods, e.g., from the years of 2012 to 2013 (d), 2014 to 2015 (e), and 2016 to 2017 (f). The network for the whole data, from 2000 to 2017, is also predicted (g). The red and blue edges represent positive and negative couplings, respectively. Edge direction is clock-wise. Predicted covariances are shown to compare with observed covariances $C_{ij}^{\text{true}}$ [(d)–(g), lower]. Cumulative profit vs time period with trading every day (without $D$, black) and trading only on days specified by lower model discrepancy (with $D$, red) strategies (h). Profit per transaction using our strategy is plotted as a function of time-window size (i).