Nonparametric estimation of Fisher information from real data

The Fisher information matrix (FIM) is a widely used measure for applications including statistical inference, information geometry, experiment design, and the study of criticality in biological systems. The FIM is defined for a parametric family of probability distributions and its estimation from data follows one of two paths: either the distribution is assumed to be known and the parameters are estimated from the data or the parameters are known and the distribution is estimated from the data. We consider the latter case which is applicable, for example, to experiments where the parameters are controlled by the experimenter and a complicated relation exists between the input parameters and the resulting distribution of the data. Since we assume that the distribution is unknown, we use a nonparametric density estimation on the data and then compute the FIM directly from that estimate using a finite-difference approximation to estimate the derivatives in its definition. The accuracy of the estimate depends on both the method of nonparametric estimation and the difference $\mathrm{\Delta }\theta$ between the densities used in the finite-difference formula. We develop an approach for choosing the optimal parameter difference $\mathrm{\Delta }\theta$ based on large deviations theory and compare two nonparametric density estimation methods, the Gaussian kernel density estimator and a novel density estimation using field theory method. We also compare these two methods to a recently published approach that circumvents the need for density estimation by estimating a nonparametric $f$ divergence and using it to approximate the FIM. We use the Fisher information of the normal distribution to validate our method and as a more involved example we compute the temperature component of the FIM in the two-dimensional Ising model and show that it obeys the expected relation to the heat capacity and therefore peaks at the phase transition at the correct critical temperature.

Figure 1

Schematic drawing in one dimension with points of estimation $\theta$ and $\theta +\mathrm{\Delta }\theta$. The gray area is the hypersphere. $\varepsilon$ is the radius of the hypersphere in units of $\mathrm{\Delta }\theta$.

Figure 2

A comparison between Gaussian KDE, DEFT, and EMST for FI estimation. The top figure shows median FI estimates using the different methods. Error bars represent 5 and 95 percentiles. The bottom figure shows the relative errors. The values were computed with $N={10}^4,\varepsilon =0.05$, and 100 repetitions at each $\sigma$. The same samples were used by all three methods. The KDE and EMST estimates were shifted by $\pm 0.02$ along the $\sigma$ axis for clearer presentation.

Figure 3

The median relative error as a function of $\varepsilon$ for different values of $\sigma$. FI stands for the computed value and $g_{\sigma \sigma }$ the analytic value. The shaded areas and error bars in the inset indicate the 5 and 95 percentiles computed over 100 repetitions of the computation with $N=2\times {10}^4$.

Figure 4

Relative error in the computation of the FI for $\sigma =1.0$ as a function of both $\mathrm{\Delta }\sigma$ and $N$. Computed using DEFT with 100 repetitions per point. Dashed line represents the $\varepsilon =0.1$ line and the dash-dotted line is the $\mathrm{\Delta }\sigma =0.35$ line. Unlike the previous plots, here we compute the absolute-value relative error to avoid problems with the logarithmic color-bar scale.

Figure 5

Blue continuous curve is the $TT$ component of the FIM and the green dots are the heat capacity in the two-dimensional Ising model on a $25\times 25$ grid. Shaded blue and green regions indicate the 5 and 95 percentiles computed from five simulations. The inset shows the ratio of FI to heat capacity ($g_{TT}{T}^2{C}_h^{-1}{L}^{-2}$) which according to Eq. (13) is equal to 1 (black horizontal line in the inset).

Figure 6

Comparison between the three methods reviewed in the main text for $\varepsilon =0.1$. All other parameters are equal to those in Fig. 2.

Figure 7

Comparison between the three methods reviewed in the main text for $\varepsilon =0.2$. All other parameters are equal to those in Fig. 2.

Figure 8

A plot of the accuracy of EMST as a function of $\varepsilon$ with the same parameters as in Fig. 3.