Quality of uncertainty estimates from neural network potential ensembles

Neural network potentials (NNPs) combine the computational efficiency of classical interatomic potentials with the high accuracy and flexibility of the ab initio methods used to create the training set, but can also result in unphysical predictions when employed outside their training set distribution. Estimating the epistemic uncertainty of a NNP is required in active learning or on-the-fly generation of potentials. Inspired from their use in other machine-learning applications, NNP ensembles have been used for uncertainty prediction in several studies, with the caveat that ensembles do not provide a rigorous Bayesian estimate of the uncertainty. To test whether NNP ensembles provide accurate uncertainty estimates, we train such ensembles in four different case studies and compare the predicted uncertainty with the errors on out-of-distribution validation sets. Our results indicate that NNP ensembles are often overconfident, underestimating the uncertainty of the model, and require to be calibrated for each system and architecture. We also provide evidence that Bayesian NNPs, obtained by sampling the posterior distribution of the model parameters using Monte Carlo techniques, can provide better uncertainty estimates.

Figure 1

The ground truth (blue dash-dotted lines) and the training points (blue dots and dotted vertical lines) are shown together with predictions from eight models trained on the same data but initialized differently. In the top panel we plot the logarithm of the standard deviation between the models. The model uses the hyperbolic tangent as an activation function. The top panel shows the standard deviation of the model prediction as a function of $r$ on a semilogarithmic scale.

Figure 2

Similar to Fig. 1, training data (blue dots) sampled from the ground truth (blue dash-dotted line) were used to train different models (dotted lines). The models are initialized independently and also have different architectures, ensured via a random assignment of activation function to every neuron in the hidden layer. The estimate of the uncertainty is improved due to reduced bias.

Figure 3

Similar to Fig. 1, the training data (blue dots) are sampled from the ground truth (blue dash-dotted line). The dotted lines show the predictions of models sampled from the posterior parameter distribution, and the orange area depicts one standard deviation from the mean.

Figure 4

The prediction of a force component ($y$ axis) against the DFT-calculated force ($x$ axis) for the validations sets ${\mathcal{D}}_{300}^{\mathrm{Al}}, {\mathcal{D}}_{600}^{\mathrm{Al}}$, and ${\mathcal{D}}_{1000}^{\mathrm{Al}}$ in blue, orange, and green, respectively. The RMSE of the forces is given in brackets in the legend.

Figure 5

Histograms of the difference of the predicted and calculated force components reveal an approximately Gaussian-distributed error with increasing deviation with higher temperature.

Figure 6

The ensemble uncertainty ${\sigma }_{I\alpha }$ [calculated via Eq. (5)] is plotted against the error ${\epsilon }_{I\alpha }$, calculated as the squared difference between the predicted value and the label, for three validation set ${\mathcal{D}}_{1500}^{\mathrm{Al}}, {\mathcal{D}}_{1500}^{\mathrm{Al}}$, and ${\mathcal{D}}_{1500}^{\mathrm{Al}}$ in blue, orange, and green, respectively. The red solid line gives the line of smallest slope the bounds the data points from above. In the top panel, the reliability diagram shows the mean error for all data points with a given uncertainty, with the same color encoding.

Figure 7

For the validation set ${\mathcal{D}}_{1500}^{\mathrm{Al}}$ we plot the recall or TPR as a heat map over different true errors and predicted errors (left panel). On the right, we plot the precision or PPV for the same dataset.

Figure 8

The true error distribution of snapshots is plotted for all configurations of the validation set ${\mathcal{D}}_{1500}^{\mathrm{Al}}$ for configurations that have a predicted uncertain above (top panel) or below (bottom panel) a given threshold. The black bars display the true error histogram for all configurations. The histograms being very similar to each other shows that for ${\mathcal{D}}_{1500}^{\mathrm{Al}}$ it is not possible to single out high-error configurations via a threshold on the model uncertainty.

Figure 9

We plot the DFT forces against the predicted forces of deepmd for the validation sets ${\mathcal{D}}_1^{{\mathrm{H}}_2\mathrm{O}}, {\mathcal{D}}_2^{{\mathrm{H}}_2\mathrm{O}}$, and ${\mathcal{D}}_3^{{\mathrm{H}}_2\mathrm{O}}$. RMSE is given in brackets inside the legend

Figure 10

The predicted uncertainty ${\sigma }_{I\alpha }$ against the true error in liquid water for the validation sets (${\mathcal{D}}_1^{{\mathrm{H}}_2\mathrm{O}}, {\mathcal{D}}_2^{{\mathrm{H}}_2\mathrm{O}}$, and ${\mathcal{D}}_3^{{\mathrm{H}}_2\mathrm{O}}$), shown as green, orange, and blue dots, respectively.

Figure 11

Recall (TPR) and precision (PPV) for the ${\mathcal{D}}_3^{{\mathrm{H}}_2\mathrm{O}}$ validation set for liquid water.

Figure 12

Histogram of the true error for different subsets of ${\mathcal{D}}_3^{{\mathrm{H}}_2\mathrm{O}}$, based on the predicted model uncertainty. The black bars show the distribution of error distribution of the entire validation set. With increasing model uncertainty threshold, the histogram becomes skewed towards higher-error configurations.

Figure 13

We plot the CCSD(T) forces against the predicted forces of the NNP ensemble for the validation sets ${\mathcal{D}}_{1...5}^{{\mathrm{C}}_6{\mathrm{H}}_6}$. RMSE is given in brackets inside the legend.

Figure 14

The predicted uncertainty ${\sigma }_{I\alpha }$ against the true error of an isolated benzene for the validation sets ${\mathcal{D}}_1^{{\mathrm{C}}_6{\mathrm{H}}_6}, {\mathcal{D}}_3^{{\mathrm{C}}_6{\mathrm{H}}_6}$, and ${\mathcal{D}}_5^{{\mathrm{C}}_6{\mathrm{H}}_6}$, shown as green, orange, and blue dots, respectively.

Figure 15

Recall (TPR) and precision (PPV) for the ${\mathcal{D}}_5^{{\mathrm{C}}_6{\mathrm{H}}_6}$ validation set.

Figure 16

Histogram of the prediction error for different subsets of ${\mathcal{D}}_5^{{\mathrm{C}}_6{\mathrm{H}}_6}$, selected based on the ensemble model uncertainty. The black bars show the distribution of error distribution of the entire validation set. With increasing the model uncertainty threshold, the histogram becomes skewed towards higher-error configurations.