Bias reduction in the estimation of mutual information

This paper deals with the control of bias estimation when estimating mutual information from a nonparametric approach. We focus on continuously distributed random data and the estimators we developed are based on a nonparametric $k$-nearest-neighbor approach for arbitrary metrics. Using a multidimensional Taylor series expansion, a general relationship between the estimation error bias and the neighboring size for the plug-in entropy estimator is established without any assumption on the data for two different norms. The theoretical analysis based on the maximum norm developed coincides with the experimental results drawn from numerical tests made by Kraskov et al. [Phys. Rev. E 69, 066138 (2004)]. To further validate the novel relation, a weighted linear combination of distinct mutual information estimators is proposed and, using simulated signals, the comparison of different strategies allows for corroborating the theoretical analysis.

Figure 1

Mutual information and mean estimation error using the different estimators ${\widehat{\mathcal{I}(X,Y)}}_{\mathrm{basic},\mathrm{E}},{\widehat{\mathcal{I}(X,Y)}}_{\mathrm{basic},\mathrm{M}},{\widehat{\mathcal{I}(X,Y)}}_{\mathrm{mixed},\mathrm{E}}$, and ${\widehat{\mathcal{I}(X,Y)}}_{\mathrm{mixed},\mathrm{M}}$ with 100 trials. (a) Mutual information (in nats) estimated with varying $\beta ,d=3,n=512$. (b) Mutual information (in nats) estimated with different signals lengths, $\beta =0.4,d=3$. (c) Mean estimation error $\overline{\widehat{\mathcal{I}(X,Y)}-\mathcal{I}(X,Y)}$ (in nats) with varying dimension, $\beta =0.5,n=512$.