A guide to constraining effective field theories with machine learning

We develop, discuss, and compare several inference techniques to constrain theory parameters in collider experiments. By harnessing the latent-space structure of particle physics processes, we extract extra information from the simulator. This augmented data can be used to train neural networks that precisely estimate the likelihood ratio. The new methods scale well to many observables and high-dimensional parameter spaces, do not require any approximations of the parton shower and detector response, and can be evaluated in microseconds. Using weak-boson-fusion Higgs production as an example process, we compare the performance of several techniques. The best results are found for likelihood ratio estimators trained with extra information about the score, the gradient of the log likelihood function with respect to the theory parameters. The score also provides sufficient statistics that contain all the information needed for inference in the neighborhood of the Standard Model. These methods enable us to put significantly stronger bounds on effective dimension-six operators than the traditional approach based on histograms. They also outperform generic machine learning methods that do not make use of the particle physics structure, demonstrating their potential to substantially improve the new physics reach of the Large Hadron Collider legacy results.

Figure 1

Feynman diagram for Higgs production in weak boson fusion in the $4\ell$ mode. The red dots show the Higgs-gauge interactions affected by the dimension-six operators of our analysis.

Figure 2

Kinematic distributions in our example process for three example parameter points. We assume an idealized detector response to be discussed in Sec. 2d2. Left: Transverse momentum of the leading (higher-$p_T$) jet, a variable strongly correlated with the momentum transfer in the process. The dip around 350 GeV is a consequence of the amplitude being driven through zero, as discussed in the text. Right: Separation in azimuthal angle between the two jets.

Figure 3

Basis points ${\theta }_c$ and some of the morphing weights $w_c(\theta )$ for our example process. Each panel shows the morphing weight of one of the components $c$ as a function of parameter space. The weights of the remaining 13 components (not shown) follow qualitatively similar patterns. The dots show the position of the basis points ${\theta }_c$; the big black dot denotes the basis point corresponding to the morphing weight shown in that panel. Away from the morphing basis points, the morphing weights can easily reach $\mathcal{O}(100)$, with large cancellations between different components.

Figure 4

Score vector as a function of kinematic observables in our example process. Left: First component of the score vector, representing the relative change of the likelihood with respect to small changes in ${\mathcal{O}}_W$ direction. Right: Second component of the score vector, representing the relative change of the likelihood with respect to small changes in ${\mathcal{O}}_{WW}$ direction. In both panels, the axes show two important kinematic variables. We find that the score vector is clearly correlated with these two variables.

Figure 5

Illustration of some key concepts with a one-dimensional Gaussian toy example. Left: Classifiers trained to distinguish two sets of events generated from different hypotheses (green dots) converge to an optimal decision function $\mathit{s}(x|{\theta }_0,{\theta }_1)$ (in red) given in Eq. (17). This lets us extract the likelihood ratio. Right: Regression on the joint likelihood ratios $r(x_e,z_e|{\theta }_0,{\theta }_1)$ of the simulated events (green dots) converges to the likelihood ratio $\mathit{r}(x|{\theta }_0,{\theta }_1)$ (red line).

Figure 6

Illustration of some key concepts with a one-dimensional Gaussian toy example. Left: Probability density functions for different values of $\theta$ and the scores $t(x_e,z_e|\theta )$ at generated events $(x_e,z_e)$. These tangent vectors measure the relative change of the density under infinitesimal changes of $\theta$. Right: Dependence of $\log \mathit{p}(x|\theta )$ on $\theta$ for fixed $x=4$. The arrows again show the (tractable) scores $t(x_e,z_e|\theta )$.

Figure 7

Illustration of some key concepts with a one-dimensional Gaussian toy example. Left: Full statistical model $\log \mathit{r}(x|\theta ,{\theta }_1)$ that we are trying to estimate. Right: Available information at the generated events $(x_e,z_e)$. The dots mark the joint likelihood ratios $\log r(x_e,z_e|{\theta }_0,{\theta }_1)$, the arrows the scores $t(x_e,z_e|{\theta }_0,{\theta }_1)$.

Figure 8

Schematic neural network architectures for point-by-point (top), agnostic parametrized (middle), and morphing-aware parametrized (bottom) estimators. Solid lines denote dependencies with learnable weights, dashed lines show fixed functional dependencies.

Figure 9

Uncertainty $\mathrm{\Delta }\log \widehat{r}(x_e|\theta ,{\theta }_1)$ of morphing-aware estimators due to uncertainties $\mathrm{\Delta }\log {\widehat{r}}_c(x_e|\theta ,{\theta }_1)$ on the individual basis ratios as a function of $\theta$. We fix ${\theta }_1$ as in Eq. (47) and show one random event $x_e$, the results for other events are very similar. We assume iid Gaussian uncertainties on the $\log {\widehat{r}}_c(x|\theta ,{\theta }_1)$ and use Gaussian error propagation. The white dots show the position of the basis points ${\theta }_c$. Small uncertainties in the individual basis estimators ${\widehat{r}}_c(x_e|\theta ,{\theta }_1)$ are significantly increased due to the large morphing weights and can lead to large errors of the combined estimator $\widehat{r}(x_e|\theta ,{\theta }_1)$.

Figure 10

True vs estimated likelihood ratios for a benchmark hypothesis ${\theta }_0=(-0.5,-0.5{)}^T$. Each dot corresponds to one event $x_e$. The CASCAL (right, red), RASCAL (right, orange), and SALLY (middle, blue) techniques can predict the likelihood ratio extremely accurately over the whole phase space. All new techniques clearly lead to more precise estimates than the traditional histogram approach (left, orange).

Figure 11

True vs estimated expected log likelihood ratio. Each dot corresponds to one value of ${\theta }_0$, where we take the expectation over $x\sim \mathit{p}(x|{\theta }_{\mathrm{SM}})$. The new techniques are less biased than the histogram approach (left, orange).

Figure 12

Comparison of the point-by-point, parametrized, and morphing-aware versions of CARL. Top left: True vs estimated likelihood ratios for a benchmark hypothesis ${\theta }_0=(-0.5,-0.5{)}^T$, as in Fig. 10. Each dot corresponds to one event $x_e$. Top right: True vs estimated expected log likelihood ratio, as in Fig. 11. Each dot corresponds to one value of ${\theta }_0$, where we take the expectation over $x\sim p(x|{\theta }_{\mathrm{SM}})$. The parametrized estimator outperforms the point-by-point one and particularly the morphing-aware version.

Figure 13

Performance of the techniques as a function of the training sample size. As a metric, we show the mean squared error (left) and trimmed mean squared error on $\log \mathit{r}(r|{\theta }_0,{\theta }_1)$ weighted with a Gaussian prior, as discussed in the text. Note that we do not vary the size of the calibration data samples. The number of epochs is increased such that the number of epochs times the training sample size is constant; all other hyperparameters are kept constant. The SALLY method works well even with very little data, but plateaus eventually due to the limitations of the local model approximation. The other algorithms learn faster the more information from the simulator is used.

Figure 14

Learning curve of the parametrized models. The solid lines show the metrics evaluated on the training sample, the dots indicate the performance on the validation sample. Note that these numbers are not comparable to the metrics in Table 4 and Fig. 13, which are weighted with the prior in Eq. (62). These results also do not include calibration. Left: Mean squared error of $\log \widehat{r}(x|{\theta }_0,{\theta }_1)$. Right: Binary cross entropy of the classification based on $\widehat{s}(x|{\theta }_0,{\theta }_1)$ between the numerator and denominator samples. The solid grey line shows the “optimal” performance based on the true likelihood ratio. The CASCAL and RASCAL techniques converge to a performance close to the theoretic optimum. The CARL approach (green), based on minimizing the cross entropy, shows signs of overfitting. All machine-learning-based methods outperform traditional histograms (dashed orange).

Figure 15

Expected likelihood ratio with respect to the Standard Model along a one-dimensional slice of the parameter space. We assume 36 observed events and the SM to be true. For each estimator, we generate five sets of predictions with different reference hypotheses, independent data samples, and different random seeds. The lines show the median of this ensemble, the shaded error bands the envelope. All machine-learning-based methods reproduce the true likelihood function well, while the doubly differential histogram method underestimates the likelihood ratio in the region of negative Wilson coefficients.

Figure 16

Expected exclusion contours based on asymptotics at 68% C.L. (innermost lines), 95% C.L., and 99.7% C.L. (outermost lines). We assume 36 observed events and the SM to be true. As test statistics, we use the profile likelihood ratio with respect to the maximum-likelihood estimator. For each estimator, we generate five sets of predictions with different reference hypotheses, independent data samples, and different random seeds. The lines show the median of this ensemble, the shaded error bands the envelope. The new techniques based on machine learning, in particular the CASCAL and RASCAL techniques, lead to expected exclusion contours very close to those based on the true likelihood ratio. An analysis of a doubly differential histogram leads to much weaker bounds.

Figure 17

Expected exclusion contours based on the Neyman construction with toy experiments at 68% C.L. (innermost lines), 95% C.L., and 99.7% C.L. (outermost lines). We assume 36 observed events and the SM to be true. As test statistics, we use the likelihood ratio with respect to the SM. All machine-learning-based methods let us impose much tighter bounds on the Wilson coefficients than the traditional histogram approach (left, dotted orange). The Neyman construction guarantees statistically correct results: no contour based on estimators excludes parameter points that should not be excluded. The expected limits based on the CASCAL (right, lavender) and RASCAL (right, red) techniques are virtually indistinguishable from the true likelihood contours.

Figure 18

Ratio of the estimated likelihood ratio $\widehat{r}(x|{\theta }_0,{\theta }_1)$ to the joint likelihood ratio $r(x,z|{\theta }_0,{\theta }_1)$, which is conditional on the parton-level momenta and other latent variables. As a benchmark hypothesis we use ${\theta }_0=(-0.5,-0.5{)}^T$, the events are drawn according to the SM. The spread common to all methods shows the effect of the smearing on the likelihood ratio. The additional spread in the histogram, CARL, and ROLR methods is due to a poorer performance of these techniques.

Figure 19

Expected exclusion contours based on the Neyman construction with toy experiments at 68% C.L., 95% C.L., and 99.7% C.L. with smearing. We assume 36 observed events and the SM to be true. As test statistics, we use the likelihood ratio with respect to the SM. In the setup with smearing we cannot these results to the true likelihood contours. But since the Neyman construction is guaranteed to cover, these expected limits are correct. The new techniques, in particular CASCAL (right, dashed red) and RASCAL (right, dash-dotted orange), allow us to set much tighter bounds on the Wilson coefficients than a traditional histogram analysis (left, dotted orange).

Figure 20

Example distributions to illustrate the doubly differential histogram analysis (top), the SALLY technique (middle), and the CASCAL method (bottom). The left panels show the different spaces in which the densities and their ratios are estimated. On the right we show the corresponding distributions of the estimated ratio $\widehat{r}$ (solid) and compare them to the true likelihood ratio distributions (dotted). We use the benchmark ${\theta }_0=(-0.5,-0.5{)}^T$ (blue) and ${\theta }_1$ as in Eq. (47) (orange).

Figure 21

Calibration curves for different estimators, comparing the uncalibrated (raw) estimator to the estimator after probability calibration. The calibration curve for the truth prediction is a cross-check for consistency; we do not actually use calibration for the truth predictions. For the local score regression technique, we show the value of $\widehat{h}(x|{\theta }_0,{\theta }_1)$ (essentially the log likelihood ratio in the local model) versus the estimated likelihood ratio after density estimation.

Figure 22

Receiver operating characteristic (ROC) curves of true positive rates (TPR) vs false positive rates (FPR) for the classification between the benchmark scenarios ${\theta }_0=(-0.5,-0.5{)}^T$ and ${\theta }_1$ as in Eq. (47). The ROC Area under curve (AUC) based on the true likelihood is 0.6276. The results show how much the probability distributions for these two hypotheses overlap.