Top quark as a probe of heavy Majorana neutrino at the LHC and future collider

Right-handed (RH) Majorana neutrinos play a crucial role in understanding the origin of neutrino mass, the nature of dark matter and the mechanism of matter-antimatter asymmetry. In this work, we investigate the observability of heavy Majorana neutrino through the top quark neutrinoless double beta decay process $t \to b \ell^+ \ell^+ j j$ at hadron colliders. By performing detector level simulation, we demonstrate that our method can give stronger limits on the light-heavy neutrino mixing parameters $|V_{eN, \mu N}|$ in the mass range of 15 GeV $


INTRODUCTION
The discovery of neutrino oscillations in solar, atmospheric, reactor and accelerator experimental data indicates that neutrinos are massive and mixed [1]. This stands for a robust evidence for new physics beyond the Standard Model (SM). The seesaw mechanism provides a natural framework for generating a small Majorana neutrino mass [2][3][4][5][6][7][8][9][10][11][12]. There are many models that could incorporate the Majorana mass, such as SO(10) Supersymmetric (SUSY) grand unification [13][14][15] and other grand unified theories (GUT) [16][17][18], which naturally links the tiny neutrino masses with new physics at the GUT-scale. Thus, the typical scale for heavy Majorana neutrino mass M N in GUT is of order the GUT-scale.
Among various seesaw models, the Type-I seesaw extension of the SM is the simplest version, in which three singlet RH neutrinos (N Rα ) are included [2][3][4][5]. The full mass term of neutrinos can be written as, where i = 1, 2, 3 is the generation index and α = e, µ, τ is the RH neutrino flavor index. After the electroweak symmetry breaking, the neutrino mass matrix in the flavor basis {ν C Li , N Rα } is given by, The diagonalization of Eq. 2 leads to the mixing between light and heavy neutrinos. Then, the light-flavor neutrinos ν iL can be expressed as a combination of light and heavy mass eigenstates, where the mixing matrices U and V satisfy the unitary condition U U † + V V † = I. If M N is of order GUT-scale, the light-heavy neutrino mixing parameters V 2 iN ∼ M D /M N are usually too tiny to produce sizable effects in various physical processes. However, if M N is allowed to be much lower, then the induced effects of such heavy Majorana neutrinos can be searched for in some existing experiments. Therefore, it is crucial to explore the possibilities of implementing seesaw mechanism at low energies from both theoretical and experimental researches.
Heavy neutrinos that have masses below TeV scale in the low-scale Type-I seesaw can be directly produced at colliders (for a review, see e.g. [38]). By searching for the process e + e − → N (→ ℓW, ν Li Z, ν Li H)ν Li , the LEP experiment puts a 95% C.L. upper limit on the mixing parameter |V eN,µN | 2 < O(10 −5 ) in a heavy neutrino mass range between 80 and 205 GeV [52]. Recently, the CMS collaboration has performed searches for heavy Majorana neutrinos through the Drell-Yan process qq → W * → N ℓ and photon initial process γq → W * q ′ → N ℓq ′ in trilepton and same-sign dilepton final states, which give the current stringent limits on |V eN,µN | 2 from O(10 −5 ) to unity in the masses of heavy neutrinos between 20 GeV and 1600 GeV [53,54].
As a top-rich environment, the Large Hadron Collider (LHC) can produce copious top quark events and may give a good opportunity to test the low-scale seesaw models [55][56][57]. In this work, we will demonstrate that the top quark neutrinoless double beta decay process t → bjjℓ + ℓ + (see Fig. 1) provides a new way to search for the GeV scale-electroweak scale heavy Majorana neutrino, which will give the signature of the same-sign dilepton plus multi-jet through tt production at the LHC. In the following calculations, we will show that our strategy has a better sensitivity of probing the light-heavy neutrino mixing parameter V µN than other existing methods when 15 GeV m N 80 GeV.

SEARCH FOR 0νββ DECAY OF TOP QUARK
As a phenomenological study, we will parameterize the low-scale Type-I seesaw as a single RH Majorana neutrino mass scale M N and a single flavor light-heavy neutrino mixing V iN . Such a framework allows us to remain agnostic of the detailed UV-physics, yet still capture the feature of low-scale Type-I seesaw. The effective interactions between Majorana neutrinos and charged leptons in the mass eigenstates is given by, In our study, we assume that the mixing effects in other flavors ℓ ′ = ℓ are sub-dominant. This will enables us to derive generic bounds on the mixing parameter, which can be translated or scaled appropriately in the context of particular neutrino mass models. Besides, we will focus on the semileptonic decay of the W boson, because it is impossible to determine whether the leptonic channel is induced by the Majorana neutrino. In Fig. 1, we present the dependence of the normalized branching ratio of the top rare decay channel Fig. 1, we can find that the normalized branching ratio can be as large as 10 −4 to 10 −2 when 15 GeV < m N < m W . With the increase of m N , the branching ratio will decrease rapidly due to the suppression of phase space. As such, we will focus on the kinematical region of m N < m W in our study.
In order to understand the mass generation mechanism and flavor structure of neutrino sector, it is essential to measure the mixing parameter and mass of each kind of neutrino at the LHC. It should be noted that the limit on the mixing parameter |V τ N | 2 strongly depends on the identification of same-sign di-tau, which is quite difficult and has low efficiency in the realistic collider simulation at the LHC. Therefore, we estimate that the current LHC sensitivity for the mixing parameter V ℓN (ℓ = e, µ) can be improved significantly for the heavy neutrino mass range of interest, i.e. 15 GeV < m N < m W . Then, we will carry out the Monte Carlo simulation of the following signature, where t → bℓ + ℓ + jj andt → bjj. The contribution of the process pp → tt → 2b + ℓ − ℓ − + 4j is also included. Since there are two same sign leptons (2SSLs) plus multi-jets in our signal, the main SM backgrounds include: • multiple prompt leptons: they mainly come from events with two vector bosons, such as W ± W ± +jets and ttW ± . Besides, the processes of W Z+jets and ZZ+jets can lead to 2SSLs, if one or more of the leptons fail the reconstruction or selection criteria.
• misidentified leptons: The fake leptons can be misidentified hadrons that are from heavy-flavor jets. These fake leptons are generally less isolated than a prompt lepton from a W/Z boson decay. The main contribution arises from tt events.
• sign mismeasurement: The events that have two opposite-sign leptons with jets could contaminate our signal due to the mismeasured sign of leptons. But the mismeasurement rate of the sign of an electron or muon is usually small, which will be not considered in this analysis.
Besides, it should be mentioned that the leptons arising from b and c decay can also mimic our signals because of the large heavy flavor production cross section. However, these leptons are usually too soft to satisfy our criterion of two prompt same sign leptons with P T > 10GeV . According to the CMS analysis, their contributions are expected to be negligible [54]. We generate the parton-level signal and background events by using MadGraph5 aMC@NLO [58]. Within the framework of CheckMATE2 [59], we then implement parton showering and hadronization by Pythia-8.2 [60], while the detector effects are simulated by tuned Delphes3 [61]. Jets-clustering is done by FastJet [62] with the anti-k t algorithm [63]. In the simulation we assume a 70% btagging efficiency. To include the higher order QCD corrections, we normalize the leading order cross sections of tt and ttW ± to their NNLO and NLO values, respectively [64,65]. Signal (ℓ ± ℓ ± ) t̄t t̄tW ± W ± W ± +jets Signal (ℓ ± ℓ ± ) tt tt ℓ ± ℓ ± ℓ ±̄j e s Signal (ℓ ± ℓ ± ) t̄t t̄tℓ ± ℓ ± ℓ ±̄j ets Signal (ℓ ± ℓ ± ) t̄t t̄tℓ ± ℓ ± ℓ ±̄j et Figure 2. Kinematical distributions of signal pp → ℓ ± ℓ ± + 2b + 4j and backgrounds tt, ttW ± and W ± W ± +jets at 14 TeV LHC. The benchmark point is mN 4 = mN 5 = 15 GeV and V ℓN = VµN = 1.
In Fig. 2, we present the kinematical distributions of the signal and SM background events at the 14 TeV LHC. We can see that the backgrounds tt and ttW ± have less events with same-sign muons than our signal (top-left panel). Since there are no neutrinos in our signal process,  Table I. Cutflow of cross sections of the signal process pp → ℓ ± ℓ ± + 2b + 4j and background processes pp → tt, ttW ± , W W +jets at the 14 TeV LHC. The cross sections are in the unit of pb. The benchmark point is chosen as mN = 15 GeV and V ℓN = 1.
the missing transverse energy / E T of the signal events are smaller than that of background events (top-right panel). Besides, we note that the two same-sign muons in the signal events come from the decay of the same top quark and hence tend to be closer. While two muons in the backgrounds arise from the decays of different parent particles. Thus they separate from each other, which can be seen in the distribution of ∆R ℓℓ (lower-left panel). Furthermore, the reconstruction of top quark also plays an important role in discriminating the signal from backgrounds. To do so, we present the cluster mass m bjjℓℓ , in which the leading b jet and two soft jets are used. This is because the jets from the top five-body decay t → bℓ + ℓ + jj are averagely softer than those in the top quark decay t → bjj. We can see that most of the signal events distribute around m bjjℓℓ ∼ 200 GeV, which can be used to suppress W ± W ± +jets background events.
According to the above discussions, we apply the following cuts to select the signal events in our analysis: • Cut-1: We require a pair of same-sign leptons in the final states. Each of leptons should satisfy p T (ℓ) > 10 GeV and |η| < 2.8.
• Cut-2: At least 6 jets with p T (j) > 15 GeV and |η| < 3.0 in the final states are required.
• Cut-3: The missing transverse energy is required to satisfy / E T < 25 GeV.
• Cut-5: We demand at least one b-jet with p T (b) > 20 GeV in the final states.
• Cut-6: We require the reconstructed invariant mass m bjjℓℓ lie in the range of [130, 350] GeV.
In Table I, we show the cutflow of cross sections of the signal process pp → ℓ ± ℓ ± + 2b + 4j and background processes pp → tt, ttW ± , W W +jets at the 14 TeV LHC.
It can be seen that the tt process is the dominant background, which is followed by ttW ± process. After imposing the requirement of same-sign muons, the cross section of tt process is reduced to the same order as that of signal process. Then, the small / E T < 25 GeV, large jet multiplicity N (j) ≥ 6 and 0.4 < ∆R < 2.5 conditions further suppress the cross sections of all backgrounds by an order of O(10 2 ). In the end, the cluster mass cut will remove the W W +jets background events and reduce the ttW ± background events to negligible level. Thus with the above cuts, we expect we can have a promising sensitivity of probing our parameter space because of very few background events.
In order to estimate the signal significance (α), we adopt the following formula, in which S and B stand for number of signal and background events after our cuts, respectively. L is the integrated luminosity of the collider. It should be mentioned that the main systematic uncertainty is related to the misidentified-lepton backgrounds. By combining other sources, we include a systematic uncertainty of β = 5% in our calculations. Other limits are also shown: the electroweak precision measurements [44], search for neutral heavy leptons produced in Z decays at LEP [52], and searches of same-sign leptons [53] and trileptons events [54] at the LHC.
In Fig. 3, we present 2σ exclusion limits of the signal process pp → tt → µ ± µ ± + 2b + 4j on the plane of m N versus |V µN | 2 . To compare our results with others, we also plot the limits from electroweak precision measurements, LEP search for neutral heavy leptons produced in Z decays, and LHC searches of same-sign leptons and trileptons events. It can be seen that the light-heavy neutrino mixing parameter |V µN | 2 > 2.5 × 10 −6 can be excluded at 2σ level in the masses of heavy neutrinos between 15 GeV and 80 GeV at 13 TeV LHC with the luminosity of 36 fb −1 , which is stronger than other existing bounds. Such a limit will be further improved to |V µN | 2 > 2.3 × 10 −6 at future HL-LHC with the luminosity of 3000 fb −1 . s =14TeV, L=3ab -1 , β=5% Figure 4. Same as Fig. 4, but for the signal process pp → 2e ± + 2b + 4j. The indirect constraint from the search of neutrinoless double beta decay with the GERDA experiment [43] is plotted as well.
Besides, we also studied the sensitivity of the electron channel pp → e ± e ± +2b+4j in Fig. 4. The exclusion limits on the mixing parameter |V eN | 2 is of the same order as |V µN | 2 , which is |V eN | 2 > 1.1 × 10 −6 at 13 TeV LHC with the luminosity of 36 fb −1 and |V eN | 2 > 7.2 × 10 −7 at the HL-LHC. This bound is weaker than that from the current GERDA search of neutrinoless double beta decay of 76 Ge [43]. However, it should be mentioned that 0νββ constraint is usually model dependent. For example, there is no such a limit in models with nearly-degenerate heavy Majorana neutrinos [66]. Also, in radiative neutrino mass models, the cancelation effects between different amplitudes induced by the Majorana phase [67] can significantly weaken this bound. Therefore, it is still necessary to analyze the electron channel while performing a complementary search for heavy neutrinos at colliders.
Finally, we comment that our analyses have not included pileup effects. In a fully realistic analysis it is important to include the effects from pileup and the effects of applying the appropriate pileup removal techniques [68][69][70]. For the analyses presented in this paper, however, events are selected with two hard same-sign leptons so we expect such additional considerations will not significantly alter the results.

CONCLUSIONS
Many low-scale Type-I seesaw models predict the heavy neutrinos below TeV scale. In order to test the mass generation mechanism and flavor structure of neutrinos in these models, it is crucial to measure the lightheavy neutrino mixing parameters V iN and masses of heavy neutrinos M N . In this paper, we proposed to probe the heavy Majorana neutrino in top quark neutrinoless double beta decay through tt production process, which gives a distinctive signature ℓ ± ℓ ± + 2b + 4j at the LHC. After performing the detector level simulation, we find that the mixing parameters |V eN | 2 > 1.1 × 10 −6 and |V µN | 2 > 2.5×10 −6 in the mass range of 15 GeV< m N < 80 GeV can be excluded at 2σ level at 13 TeV LHC with the luminosity of 36 fb −1 , which have surpassed other existing collider bounds. The future HL-LHC will further improve these limits. Therefore, we conclude that searching for the neutrinoless double beta decay of top quark will provide a new better way to probe the heavy Majorana neutrino in the mass range of 15 GeV< m N < 80 GeV at the LHC.