Density Estimation on Small Data Sets

How might a smooth probability distribution be estimated with accurately quantified uncertainty from a limited amount of sampled data? Here we describe a field-theoretic approach that addresses this problem remarkably well in one dimension, providing an exact nonparametric Bayesian posterior without relying on tunable parameters or large-data approximations. Strong non-Gaussian constraints, which require a nonperturbative treatment, are found to play a major role in reducing distribution uncertainty. A software implementation of this method is provided.

Figure 1

Density estimation using field theory. (a) A Gaussian mixture distribution $Q_{\mathrm{true}}=\frac{2}{3}\mathcal{N}(-2,1)+\frac{1}{3}\mathcal{N}(2,1)$ within the $x$ interval $(-15,15)$. (b) A histogram $R$ of $N=30$ data points sampled from $Q_{\mathrm{true}}$ and discretized to $G=100$ grid points. (c) The corresponding estimate $Q^{*}$ computed by DEFT using $\alpha =3$ and the same grid as in (b). (d) One hundred distributions sampled from the Laplace-approximated posterior $p_{\mathrm{Lap}}(Q|\mathrm{data})$, which accounts for uncertainty in $\ell$ as well as in $Q$. (e) One hundred distributions generated using importance resampling of the Laplace ensemble. The differential entropies of the illustrated distributions are provided.

Figure 2

Performance of DEFT. (a) DEFT, KDE, and DPMM were used to analyze data from two different $Q_{\mathrm{true}}$ distributions: the Gaussian mixture from Fig. 1 (left) and a Pareto distribution, $Q_{\mathrm{true}}(x)=3x^{-4}$, confined to the $x$ interval (1,4) (right). (b) One hundred data sets of size $N=10$ and one hundred data sets of size $N=100$ were generated for each $Q_{\mathrm{true}}$. For each data set, $Q^{*}$ was computed by DEFT (using $G=100$ and $\alpha =1$, 2, 3, or 4), by KDE, or by DPMM. Violin plots (with median indicated) show the resulting Kullback-Leibler divergences $D_{\mathrm{KL}}(Q_{\mathrm{true}}\parallel Q^{*})$. (c) $p$ values quantifying, for each simulated data set, the location of $D_{\mathrm{KL}}(Q_{\mathrm{true}}\parallel Q^{*})$ within the distribution of $D_{\mathrm{KL}}(Q\parallel Q^{*})$ values observed for $Q\sim p(Q|\mathrm{data})$.

Figure 3

DEFT applied to Higgs boson data. (a) A reconstruction of Fig. 4 from Ref. [11]. Dots (black) indicate the invariant masses of four-lepton decay events histogrammed across $G=37$ bins of width 3 GeV each. Also shown are the number of events expected, based on Standard Model simulations, from either background decay processes (blue) or from the decay of a Higgs boson with mass of 125 GeV (red). (b) The optimal density estimate $Q^{*}$ (black), along with 100 posterior samples $Q\sim p(Q|\mathrm{data})$ (olive) computed by DEFT using the histogram data in panel (a).