Extending the search for new resonances with machine learning

The oldest and most robust technique to search for new particles is to look for “bumps” in invariant mass spectra over smoothly falling backgrounds. We present a new extension of the bump hunt that naturally benefits from modern machine learning algorithms while remaining model agnostic. This approach is based on the classification without labels (CWoLa) method where the invariant mass is used to create two potentially mixed samples, one with little or no signal and one with a potential resonance. Additional features that are uncorrelated with the invariant mass can be used for training the classifier. Given the lack of new physics signals at the Large Hadron Collider (LHC), such model-agnostic approaches are critical for ensuring full coverage to fully exploit the rich datasets from the LHC experiments. In addition to illustrating how the new method works in simple test cases, we demonstrate the power of the extended bump hunt on a realistic all-hadronic resonance search in a channel that would not be covered with existing techniques.

Figure 1

The probability to reject the SM as a function of $q$ for fixed values of $p$ as indicated in the legend when only considering events with $Y=1$. The expected background is fixed at 1000 and the expected BSM is fixed at 20. When $q=0$, there is no signal and therefore the rejection probability is 5%, by construction. When $q=p=1$, $Y$ is not useful, but the probability to reject is above 5% simply because there is an excess of events inclusively.

Figure 2

An illustration of the CWoLa procedure for the simple two-dimensional uniform example presented in Sec. 3. The left plot shows the mass distribution for the three mass bins, which is uniform for the background. The blue line is the total number of events and the other lines represent thresholds on various neural networks described in the text leading up to Fig. 5. The center plots show the $(x,y)$ distribution for the events in each mass bin with truth labels (purple for background and yellow for signal). The black square is the true signal region for this example model, with signal distributed uniformly inside. The right plot shows the combined distribution in the $(x,y)$ plane with CWoLa labels that can be used to train a classifier even without any truth-level information (red for target window, blue sideband).

Figure 3

The CWoLa-labeled data can be used to construct an estimate for the optimal classifier $h(x,y)=\frac{p_b(x,y)+p_s(x,y)}{p_b(x,y)}$. The top path shows an estimate constructed by histogramming the observed training events in the $(x,y)$ plane. The bottom path shows an estimate constructed by using a neural network trained as described in the text, which can be efficiently generalized to higher dimensional distributions. The optimal classifier would be 1 outside of the small box centered at the origin and 1.75 inside the box.

Figure 4

The classifier $h(x,y)$ constructed from four independent training runs on the same example two-dimensional model dataset described in the text. The thick contours represent the cuts that would reduce the events in the test data target window by a factor of ${\epsilon }_{\text{test}}=10\%$, 5%, 1%.

Figure 5

The mass distribution after various threshold on the neural network classifier. The flat background fit from the sideband regions are the dashed lines, and the statistical uncertainty for each bin is shown by the error bars. The top histogram is the model before any threshold in the $(x,y)$ plane, and from top to bottom respectively the histogram is given for efficiency thresholds of 10%, 5%, 1%, 0.2%. The significance is $\mathcal{S}=3\sigma$, $9.4\sigma$, $10.8\sigma$, and $3.4\sigma$ for respectively no threshold, 10%, 5%, and 1%. The 0.2% threshold reduces the signal to no statistical significance.

Figure 6

Top left: Histogram of significance at a test threshold of 6% for 100 NN trained on the example toy model data (blue), and 100 NN trained on a control dataset with no signal present (green). The dashed green line gives the expected significance of $13.9\sigma$ for the example dataset with ideal cuts. Top right: Histogram of significance for ensembles with expected signal strength of $15\sigma$ with ideal cuts. The blue is $(N_b=10000,N_s=300,w_s=0.2)$ at a test threshold of 6%, the green is $(N_B=10000,N_s=150,w_s=0.1)$ at a test threshold of 1.5%, and the red is $(N_B=10000,N_s=75,w_s=0.05)$ at a test threshold of 0.4%. For each ensemble, 100 independent instances of the dataset are generated and one NN is trained on each dataset. Bottom: Histogram of significance for ensembles with $(N_b=10000,N_s=300)$ and varying $w_s$. Blue is $w_s=0.2$ at a test threshold of 6%, green is $w_s=0.4$ at a test threshold of 24%, and red is $w_s=1.0$ [for which the background and signal distribution in $(x,y)$ are identical] at a test threshold of 50%. For each ensemble, 100 independent instances of the dataset are generated and one NN is trained on each dataset. The dashed line gives the expected significance for each ensemble.

Figure 7

Illustration of the nested cross-validation procedure. Left: The dataset is randomly partitioned bin-by-bin into five groups. Center: For each group $i\in \{1,2,3,4,5\}$ (the test set), an ensemble classifier is trained on the remaining groups $j\ne i$. There are four ways to split the four remaining groups into three for training and one for validation. For each of these four ways, many classifiers are trained and the one with best validation performance is selected. The ensemble classifier is then formed by the average of the four selected classifiers (one for each way to assign the training/validation split). Right: Data are selected from each test group using a threshold cut from their corresponding ensemble classifier. The selected events are then merged into a single $m_{\mathrm{res}}$ histogram.

Figure 8

Left: $m_{JJ}$ distribution of dijet events (including injected signal, indicated by the filled histogram) before and after applying jet substructure cuts using the NN classifier output for the $m_{JJ}\simeq 3\mathrm{TeV}$ mass hypothesis. The dashed red lines indicate the fit to the data points outside of the signal region, with the gray bands representing the fit uncertainties. The top dataset is the raw dijet distribution with no cut applied, while the subsequent datasets have cuts applied at thresholds with efficiency of ${10}^{-1}$, ${10}^{-2}$, $2\times {10}^{-3}$, and $2\times {10}^{-4}$. Right: Local $p_0$-values for a range of signal mass hypotheses in the case that no signal has been injected (left), and in the case that a 3 TeV resonance signal has been injected (right). The dashed lines correspond to the case where no substructure cut is applied, and the various solid lines correspond to cuts on the classifier output with efficiencies of ${10}^{-1}$, ${10}^{-2}$, and $2\times {10}^{-3}$.

Figure 9

Events projected onto the 2D plane of the two jet masses. The classifiers are trained to discriminate events in the signal region (left plot) from those in the sideband (second plot). The third plot shows in red the 0.2% most signal-like events determined by the classifier trained in this way. The rightmost plot shows in red the truth-level signal events.

Figure 10

2D projections of the 12D feature-space of the signal region dataset. First column: All signal region events. Second column: Truth-level simulated signal events are highlighted in red. Third column: 0.2% most signal-like events selected by the classifier described in Sec. 5 are highlighted in red. Fourth column: Highlighted in red are the 0.2% most signal-like events selected by a classifier trained on the same sample but with true-signal events removed.

Figure 11

Truth-label ROC curves for taggers trained using CWoLa with varying number of $WX$ signal events, compared to those for a dedicated tagger trained on pure $WX$ signal and background samples (dashed black) and one trained to discriminate $W$ and $Z$ jets from QCD (dot-dashed black). The CWoLa examples have $B=81341$ in the signal region and $S=(230,352,472,697,927)$.

Figure 12

Toy test statistic distributions. Black data points: $10^3$ toys with NN training and cross-validation procedure; error bars represent $\sqrt{N}$ Poisson uncertainty. Orange histogram: $10^5$ toys with random selection (no NN training or cross validation). Blue: Asymptotic formula.

Figure 13

Dijet invariant mass distributions in the resonance mass scan in the case that no signal is present. In each plot, the signal and sideband regions used for training the tagger are indicated by the vertical dashed lines. The top distribution in each plot is the original dijet mass distribution, and the subsequent distributions have had cuts applied at efficiencies of ${10}^{-1}$, ${10}^{-2}$, $2\times {10}^{-3}$, and $2\times {10}^{-4}$ respectively. The dashed red curves are fits determined by weighted least squares, and the gray bands the corresponding systematic uncertainty in the fit. In the lowest distribution of each plot the fit is poor as the presence of low-count bins makes the least-squares fit inappropriate—the fit line is kept to guide the eye, but the fit uncertainty band is omitted.

Figure 14

Dijet invariant mass distributions in the resonance mass scan in the case that the signal is present (indicated by the filled histogram). Otherwise, the plots are as described in the caption to Fig. 13.