Constraining top quark flavor violation and dipole moments through three and four top quark productions at the LHC

In this paper, we examine the sensitivity of the three-top quark production at the LHC to the top quark flavor-changing neutral currents (FCNC) as well as the sensitivity of the four-top production to the strong and weak dipole moments of the top quark. Upper limits at $95\%$ CL on the branching fractions of $\mathcal{B}(t\rightarrow qX)$, where $X = g, Z, \gamma, H$ and $q=u,c$, are set by performing an analysis on three-top events in the same-sign dilepton channel. We consider the main sources of the background processes and a realistic detector simulation is performed at the center-of-mass energy of 14 TeV. In the second part of this work, based on the recent upper limits which have been set on the four-top quark cross section by the ATLAS and CMS collaborations from 13 TeV data, we constrain the top quark strong and weak dipole moments. The bounds on the top quark dipole moments are presented using the future LHC prospects for four-top quark cross section measurement.


I. INTRODUCTION
In proton-proton collisions at the LHC, mostly top quarks are produced in pair via strong interaction [1,2] or singly via weak interactions [1,2].However, the large center-of-mass energy of proton-proton collisions at the LHC opens the possibility to have three [3,4] or four-top quark productions.Searches for four-top quark production using the Run I dataset at √ s = 8 TeV have been performed by both the CMS [5,6] and ATLAS [7,8] experiments, with no observed excess of data above the background expectation.The searches of both experiments have been updated using the 13 TeV data using different final states [9][10][11][12][13][14][15].The cross section of the four-top quark production at the LHC with the center-of-mass energy of 13 TeV is around σ(pp → t tt t) = 9 fb.
Within the SM at leading-order, the three-top quarks are produced in association with either a W boson or a jet with a total rate of around 2 fb [3,4].Although three and four-top rates are extremely small with respect to the t t [16,17] or single top quark production by around five order of magnitudes, these processes are particularly sensitive to New Physics (NP) beyond the Standard Model (SM).Beyond the SM scenarios predict enhancements in tt tX and t tt t production cross section [3,4,[18][19][20][21][22][23].Vector-like quarks, SUSY with R-parity violation are examples of the BSM scenarios which affect their rates and some have been experimentally studied [5,7,9,11,[24][25][26].
So far, the top quark is the heaviest discovered particle with its mass close to the scale of electroweak symmetry breaking and Yukawa coupling near one, y t ∼ 1.Therefore, one may expect that possible NP effects would show up in top quark production or decays [27][28][29][30][31][32][33][34][35][36][37][38].NP can be seen either directly in new particles production or through the indirect effects via the higher order corrections.Indeed, the observation of indirect indications is important because it provides hints to search for new physics before direct observation.Within the SM framework, the branching fractions of the rare decays of the top quark t → qX, with X = g, γ, Z, Higgs and q = u, c, are extremely small and are of the order of 10 −14 − 10 −12 [39].Due to smallness of these branching fractions, the current and future experiments would not be able to measure them.Such transitions in the SM are only possible at loop-level and are significantly suppressed because of the Glashow-Iliopoulos-Maiani (GIM) mechanism [40].On the other part, it has been found that many SM extensions potentially can relax the suppression in top quark FCNC decays from GIM mechanism leading to considerable enhancements for B(t → qX) [39].This happens because of the appearance of several loop diagrams with new particles mediated inside them.Beyond the SM scenarios like technicolor, SUSY models, two-Higgs doublet models predict much higher branching fractions of the order of ∼ 10 −10 to 10 −6 which are larger than the SM values [39,[41][42][43][44][45][46][47].Searches for FCNC in the top quark sector have been followed by various experiments in the past years, including ALEPH, DELPHI, L3 and OPAL experiments at LEP [48][49][50][51], H1 and ZEUS experiments at HERA [52][53][54][55][56], CDF and D0 experiments at the Tevatron [57][58][59][60][61], and CMS and ATLAS experiments at the LHC.
The most stringent limits on the top FCNC branching fractions come from the LHC experiments.
The CMS collaboration presented the result of their search for the FCNC through single-top-quark production in association with a photon [62].Based on an integrated luminosity of 19.8 fb −1 in pp collisions at a center-of-mass energy of 8 TeV, upper limits on B(t → qγ) and B(t → qZ) through FCNC single-top-quark production in association with a photon or a Z boson are obtained [62,63].
The upper limits on the branching fractions at 95% CL are: The most recent search for the tqH FCNC has been done by looking at the events with either single top quark FCNC production associated with a Higgs boson and t t production with FCNC decay of one of the top quarks [64][65][66].The search is based on the Higgs boson decay into a b b pair and uses the data corresponding to an integrated luminosity of 35.9 fb −1 at √ s = 13 TeV.The observed upper limits at 95% CL on the branching fractions of FCNC top quark decays are [65]: These upper limits are the most stringent ones to date.Among all the FCNC top quark decays, the upper limits on B(t → qg) has been tightly bounded.Using the data collected at the center-of-mass energy of 8 TeV corresponding to an integrated luminosity of 20.3 fb −1 by the ATLAS detector, the observed upper limits on the branching fractions are found to be [67]: The CMS experiment search for the FCNC tqg provides the following limits at the 95% CL [68]: As it can be seen, the ATLAS experiment has obtained stronger bound on B(t → cg) than the CMS experiment and vice versa for the B(t → ug).So far, there have been a lot of studies on the top quark FCNC in proton-proton, electron-positron and electron-proton colliders [69][70][71][72][73][74][75][76][77][78][79][80].In the first part of this work, we perform a search for the FCNC interactions separately at the tuX and tcX vertices at the LHC by looking for events with only three-top quark in the final state.
We focus on the decay channels in which two same-sign top quarks decay into a W boson and a b-quark, followed by the W boson decay to an electron or muon and a neutrino.The same-sign dilepton decay channels are considered because it has a clean signature and does not suffer from large background contribution.
In the second part of the work, we study the strong and electroweak dipole moments of the top quark.In the SM, at leading order dipole interactions do not exist, however, electroweak radiative corrections generate both strong and weak dipole moments for the top quark.The size of top quark dipole moments are so small that the LHC would not be able to observe them.But, there are several well-motivated beyond the SM theories which contribute to these dipoles and lead to sizable enhancements which make the dipoles accessible by the LHC detectors [81][82][83].As a consequence, any observation of considerable deviations from zero would be an indication to beyond the SM physics.Here, we investigate the sensitivity of the four-top quark production to the weak and strong top quark dipole moments and upper limits are set on them using the present and prospects upper limits on the four-top quark production cross section.
The present article is organized as follows: In Section II, we present the theoretical framework for describing the top quark FCNC couplings and dipole moments which is based on the effective field theory approach.The sensitivity estimations of three-top quark production on FCNC couplings are given in Section III.Section IV is dedicated to probe the strong and weak dipole moments using the four-top quark production.Finally, Section V is dedicated to summarize the results and conclusions.

II. THEORETICAL FRAMEWORK AND ASSUMPTIONS
In this paper, we assume that new physics effects in four-top-quark production are not going to be directly discovered at the LHC (like new heavy degrees of freedom).These kinds of effects could be described by an effective Lagrangian with the following form [84,85]: where c i coefficients are dimensionless by which the new physics effects are parameterized with dimension-six operators O i .The scale of new physics is denoted by Λ and the O i are a complete set of dimension-six operators which satisfy the SM symmetries, i.e. the Lorentz and the SU (3) × SU (2) × U (1) gauge symmetries.Due to the observed excellent agreement between the predications of the SM and data, the deviations from the SM are expected to be small.Therefore, when the effects of these new operators are computed we restrict ourselves only to the SM interference the operators O d=6 i , unless there does not exist any interference.
There are different dimension-six operators which contribute to the four-top-quark production at the LHC.For instance, the impacts of full set of four-fermion qqtt dimension-six operators on the four-top production have been studied in Ref. [86].In this paper, the aim is to probe the impacts of dimension-six operators which contain FCNC (tqg, tqZ, tqγ, tqH) as well as those contribute to gt t and Zt t on the four-top-quark production at the LHC.Following on the notation presented in Ref. [84], the operators , O where ζ qt , η qt , X qt , κ qt and λ qt are the real parameters which determine the strength of FCNC interactions with gluon, Higgs, Z and photon, respectively.At tree-level, in the SM, all the above coefficients are zero, i.e.
The effective Lagrangian including dimension-six operators for gt t and Zt t can be written as: and The couplings d = 0.0 while the QCD and electroweak corrections at loop level generate very tiny values for the weak and strong dipole moments [87][88][89][90][91].The values of X L tt and X R tt parameters are equal to one and zero in the SM, respectively.
The most stringent bounds on d g V and d g A are derived from the low energy measurements.In particular, d g V is constrained using the rare B-mesons decays [87] and d g A is probed through the neutron electric dipole moment (d n ) [92].The 95% CL limits from the low energy experiments are [87,92]: The combination of the LHC and Tevatron top quark pair cross section measurements leads to the following 95% CL regions [38]: As it can be seen, the indirect constraints are tighter than the direct ones by one order of magnitude.
The weak dipole moments d Z V and d Z A have been studied at the LHC and a future electronpositron collider using the t tZ production [93].At the LHC, both d Z V and d Z A are expected to be probed down to the order of 0.15 with 300 fb −1 of integrated luminosity of data which would be well-improved to 0.08 using 3000 fb −1 integrated luminosity of data.Limits from the electroweak precision data are found to be at the same order.A future e − e + collider at the center-of-mass energy of 500 GeV with 500 fb −1 would be able to achieve the limits of 0.08 on |d Z A | and [−0.02, 0.04] on the d Z V [93].

III. SENSITIVITY OF THREE-TOP QUARK PRODUCTION TO FCNC COUPLINGS
In this section, we study the sensitivity of the three-top quark production cross section to the FCNC interactions of tqg, tqγ, tqZ, and tqH.In the SM at leading order, similar to the single top quark production, three-top quark events are produced in association with a light-jet, or a b-jet, or associated with a W boson, i.e.
The sum of cross section of all processes amounts to around 1.9 fb at the LHC with the centerof-mass energy of 14 TeV.The presence of FCNC couplings tqX, X = g, γ, Z, H, would lead to production only three-top quark (which does not exist at leading order in SM). Figure 1  the other FCNC interactions are similar except that the gluon should be replaced by a photon, a Z-boson, or a Higgs boson.
In this work, the search is performed for all FCNC couplings of tqg, tqγ, tqZ, and tqH independently, i.e. one is switched on at a time.We also do the analysis separately for q = u and q = c.
We concentrate on a very clean signature with two same-sign leptons, where lepton could be either an electron or a muon.Therefore, the signal events are generally characterized by the presence of exactly two isolated same-sign charged leptons, large missing transverse energy, and several jets from which three of them come from b-quarks.We perform the analysis for 300 and 3000 fb −1 of the LHC at the center-of-mass energy of 14 TeV and present the upper limits on the branching fractions of B(t → qX) at 95% CL.

A. Event simulation and selection
In this section, the simulation tools and techniques as well as the event selection and reconstruction are described.The process of signal is taken as three-top quarks followed by the leptonic decay of two same-sign top quarks and hadronic decay of the other top quark.As a result, the final state consists of two same-sign charged leptons, at least five jets from which three are originating from b-quarks, and missing transverse energy.In this exploratory study, the background processes such as t tZ, t tW , SM four-top, t tW W , t tZZ and W W Z are considered.For both signal and the background processes, we use MadGraph5_aMC@NLO package [94].It automatically generates the code for obtaining the production rates.The leading order parton distribution functions of NNPDF 3.0 [95] are employed as the input for the calculations and the mass of the top quark is set to 173 GeV.To calculate the three-top quark cross sections in the presence of the FCNC couplings, the effective Lagrangians introduced above are implemented into the FeynRules package [96,97] and is exported into a UFO module [98] which is connected to MadGraph5_aMC@NLO 1 .
To obtain the cross section of three-top versus the FCNC couplings, the calculations in the presence of ζ qt , λ qt , κ qt and η qt are done assuming various values: ±1.0, ±2.0 and fit the resulting cross sections to quadratic polynomials.Considering each FCNC coupling at a time, the cross sections times the related top quark effective coupling strengths are given in the first column of As it can be seen, the three-top rate has a significant dependence on the tqg FCNC coupling which can be understood by considering the appearance of diagrams like gu(c) → tg → tt t.The cross section for the tqZ FCNC coupling are presented for both cases of γ µ and σ µν -couplings.
Larger rate is observed for the σ µν -coupling which is due to the dependence of the interaction on the Z boson momentum. 1The UFO file is taken from http://feynrules.irmp.ucl.ac.be/wiki/GeneralFCNTop To perform the whole simulation chain, Pythia [99] is used for showering and hadronization.
Jets are reconstructed using the anti-k t algorithm [100] with a cone size of 0.4.Delphes framework [101] is employed for performing a comprehensive CMS detector [102] response simulation which considers a tracker, calorimeters, and a muon system with a realistic magnetic field configuration  I and Table II, respectively.The cross section values in both tables are given in fb unit.The SM four-top, t tW W , t tZZ, and t tH are found to have small contributions in the background composition.Sum of the cross section of all these processes after all cuts are found to be 0.02 fb.The three-top production in association with a jet or a W boson in the SM also could contribute to the background.After all cuts, the rate of SM three-top and W W Z are found to be of the order of 10 −3 fb and 10 −4 fb which are neglected in this analysis.At this stage, we go on to set upper limits on the signal rates at 95% CL.In order to determine the expected limits on the FCNC branching fractions, a Bayesian approach with a flat and positive prior on the cross sections of signals are used.Then the upper limits on signal rates are translated into the upper bounds on the FCNC branching fractions B(t → qX).The 95% CL constraints on various FCNC branching fractions are summarized in Table III for two scenarios of integrated luminosities 300 and 3000 fb −1 of data.The results from a recent ATLAS experiment analysis which is based on t t process with one of the top quark decays via FCNC and another one decays in standard way are also presented for comparison [105].As the comparison with the ATLAS limits shows, the three-top process would be able to reach similar sensitivity to ATLAS in the tqH FCNC coupling.However, it should be noted that the limits from the three-top process could be considerably improved taking into account the other three-top quark signatures.In addition, employing more powerful variables and tools (like a multivariate technique [107]) to discriminate signal events from backgrounds would lead to better sensitivities.Taking into account the next to leading order QCD corrections to the signal processes which includes the three-top plus a light jet, would significantly tighten the upper bounds on the FCNC branching fractions.

IV. SENSITIVITY OF FOUR-TOP QUARK PRODUCTION TO THE TOP QUARK WEAK AND STRONG DIPOLE MOMENTS
In this section, we explore the sensitivity of four-top quark production in pp collisions at the center-of-mass energy of 13 TeV to the strong (d g A,V ) and weak (d Z A,V ) top quark electric and magnetic dipole moments.The representative leading order Feynman diagrams including the contributions of strong dipole moments as filled circles are displayed in Fig. 3.
The contributions of the top quark (weak)chromoelectric (d g,Z A ), (weak)chromomagnetic (d g,Z V ) dipole moments, coming from O 33  uW and O 33 uGφ and O 33 uBφ operators, to the tt tt production rate is determined with the MadGraph5_aMC@NLO [94].By considering at most an effective vertex in each diagram, which means up to O(Λ −2 ), the total four-top cross section becomes at most a quadratic function of dipole moments:

σ(pp
where SM four-top cross section is denoted by σ SM and the linear terms are the interference between the SM and new physics and its contribution is at Λ −2 order.The quadratic terms in the cross section are corresponding to the power of Λ −4 which are the first contributing terms for the strong and weak dipole moments.The four-top cross section is symmetric with respect to d g,Z A = 0 because the cross section is a CP-even observable.To find the coefficients of cross sections in Eq. ( 14), the calculations with different values of d g,Z A and d g,Z V are done and fit the obtained cross sections to quadratic polynomials.
In Ref. [12], the CMS experiment has presented the results of a search for four-top quark production based on a dataset corresponding to an integrated luminosity of 35.9 fb −1 in proton-proton collisions at √ s = 13 TeV.The analysis relies on selecting events containing either a same-sign lepton pair or at least three leptons (e, µ) topologies.The observed signal significance is found to be 1.0 standard deviation and the cross section is measured to be 16.9 +13.8 −11.4 fb which is in agreement with the standard model prediction.In Ref. [12], the results are interpreted to limit the top quark Yukawa coupling (y t ) which leads to y t /y SM t < 2.1 at the 95% confidence level.
The ATLAS experiment search for the four-top quark production is based on the single electron or muon with large transverse momentum and a high jet multiplicity topology [9].The analysis has been performed using 3.2 fb −1 of pp collisions at the center-of-mass energy of 13 TeV and to improve the search sensitivity, events are classified based on the jet and b-jet multiplicities.An upper limit of 190 fb on the four-top cross section (21 times the SM value) is set at the 95% CL.
In the ATLAS study, upper bounds on the four-fermion contact interaction and a universal extra dimensions (UED) model parameters have been set.In our study, we only use the four-top cross section measurement done by the CMS experiment that is used as it is the most restrictive results to date.
The upper limits at 95% CL on the strong (d g A,V ) and weak (d Z A,V ) dipole moments using the CMS experiment measurement [12], which is the most recent one, are presented in Table IV.The resulted bounds are compatible with the ones obtained from top quark pair cross sections at the Tevatron and the LHC.Although the results are looser, this study complements the capabilities of other channels at the LHC.The four-top channel would not be able to compete with the t tZ and t tZZ channels to probe the weak dipole moments of the top quark.

A.
Future prospect for the top quark dipole moments To complete our study towards accessing more sensitivity to new physics effect at the LHC, it is important to have an estimate of the sensitivity of the four-top quark production using the Run-II LHC reach.In this section, we derive the limits at 95% CL on the branching fractions of the FCNC transitions and the top quark strong and weak couplings using the possible future reach of the LHC to measure the four-top cross section.In Ref. [14], a novel strategy to search for four-top production based on the same-sign dilepton and the trilepton channels is presented which allows to avoid the huge backgrounds in the full-hadronic and mono-leptonic decay channels.Signal features such as large jet and b-jet multiplicity and a Z-mass veto in the opposite-and same-flavor dilepton spectrum, are used to suppress the main background and increase the ratio of signal-to-background ratio as well as the signal significance.Using the suggested strategy leads to reach the following upper limit on the four-top signal strength using 300 fb −1 integrated luminosity of data at √ s = 13 TeV [14]: where σ exp tt tt is the expected cross section of the four-top production using 300 fb −1 of data.Considering this estimation, we redo our analysis to determine the sensitivity of four-top production to the top quark dipole moments.In Table IV, the projections of the constraints on the strong and weak dipole moments using the four-top quark with 300 fb −1 are depicted.We note that while the indirect constraints on d g V and d g A , obtained from the rare B-meson decays and from the neutron electric dipole moment (EDM), and the direct searches from t t events are stronger, the results of this search are complementary.Comparison of the achieved sensitivity to d Z V and d Z A with those could be obtained from the LHC and ILC presented in Section II, shows that the four-top rate does not provide competitive bounds with the t tZ channel.

V. SUMMARY AND CONCLUSIONS
Due to the very small production rate of the three-and four-top quark in the SM which are few fb at the LHC, the observation of these processes is quite challenging.However, both the threeand four-top processes give rise to interesting set of final states depending on various top quarks decay modes.The CMS and ATLAS collaborations have measured the upper limits on four-top production cross section in same-sign dilepton and trilepton topologies which does not suffer from large background contributions.These measurements could be directly exploited to confine the NP effects.
Uncommon decays of the top quark with extremely small rates via the flavor violating in the vertices of tqg, tqZ, tqγ, and tqH are attracting much attention as they are considerably sensitive to several SM extensions.The predictions for the branching fractions of the rare decay modes t → qX, X = g, Z, γ, H and q = u, c, in the SM are expected to be quite small so that they are unobservable at the LHC.However, new physics scenarios predict significant enhancement in the rare top quark branching fractions by many order of magnitudes.Consequently, observation of such decay modes would indicate the existence of beyond the SM.
In this work, we demonstrate that the three-top quark production is sensitive to the FCNC couplings.By performing a realistic detector simulation and the main background processes in the same-sign dilepton channel, upper limits on the FCNC branching fractions are obtained.Upper limits of the order of 10 −4 at 95% CL on the branching fractions of t → qH, t → qg and t → qZ are set.We also examine the sensitivity of the four-top cross section to the strong and weak top quark dipole moments.Upper limits are set on the top quark dipole moments which are compatible with the other bounds extracted from the other processes at the LHC.
and are related to strong (weak) magnetic and strong (weak) electric dipole moments of the top quark, respectively.In the SM, at tree-level, d g(Z) V = 0.0 and d g(Z) A

Figure 1 :
Figure 1: A representative leading order Feynman diagram of the tqg FCNC contributions to the three-top quark production at the LHC including the lepton decay of the W boson from the top quark decay and hadronic decays of the W boson from the anti-top quark.
of the CMS detector.The b-tagging efficiency and misidentification rates for light-flavor quarks are assumed to be dependent on the jet transverse momentum.They are taken as[103]:b-tagging efficiency [p T ] = 0.85 tanh(0.0025pT ) 25 1 + 0.063p T , misidentification rate for c-jets[p T ] = 0.25 tanh(0.018pT ) 1 1 + 0.0013p T ,(13)misidentification rate for light-jets[p T ] = 0.01 + 0.000038p T .The efficiency of b-tagging for a jet with p T = 40 GeV is 60% and the misidentification rates for cand light-flavor jets are 14% and 1%, respectively.The events are selected by applying the following simple criteria: (I) Two same-sign charged lepton with p T > 10 GeV, |η| < 2.5, m > 10 GeV, (II) Missing Transverse Energy (MET) > 30 GeV, (III) At least five-jets with p T > 20 GeV, |η j | < 2.5, ∆R( , j) > 0.4, ∆R(j 1 , j 2 ) > 0.4, (IV) At least three b-jets.The b-jet multiplicity for the tqg and tqH FCNC couplings are depicted in the left panel of Figure 2. As it is expected, the FCNC signals peak at three while for the backgrounds the peak is at two.The minimum cut of 10 GeV on the invariant mass of the same-sign dilepton is useful to reject events with pairs of same-sign energetic leptons from the heavy hadrons decays.The invariant mass distribution of dilepton is presented in the right side of Figure 2.There is a shift in the tqg signal scenarios with respect to the background and the tqH signals.This is due to the fact that the tqg FCNC interactions are momentum dependent.More kinematic cuts and complicated variables could be used to suppress the background contributions and enhance signal significance, however including such variables is beyond the scope of this exploratory analysis and is left to a future work.At this point, it is important to mention that the trigger of such events could be either based on only the presence of same-sign dilepton with loose isolation requirements or based on the existence of same-sign dilepton with lowered transverse momentum thresholds without any isolation requirement, but requiring hadronic activities in the event [104].The cross sections of various scenarios of FCNC signals and the main background processes after imposing different cuts are presented in Table

Figure 2 :
Figure 2: The b-jet multiplicity (left) and invariant mass of dilepton (right) for tqg and tqH signal scenarios obtained from MadGraph simulation at leading-order at √ s = 14 TeV.Sum of all backgroundprocesses is presented as well.

Figure 3 :
Figure 3: Illustrative leading order Feynman diagrams for t tt t production representing the effect of strong dipole moments as filled circles.
33uW and O33φu give contributions to gt t and Zt t interactions.The most general effective Lagrangian describing the FCNC interactions can be parameterized as:

Table II :
Cross section of the main background processes in fb after different set of cuts.±± , |η | < 2.5, p T > 10 GeV, m > 10 GeV

Table III :
[105]pper limits on the tqX FCNC at 95% C.L obtained at the √ s = 14 TeV based on the integrated luminosities of 300 and 3000 fb −1 .The HL-LHC results from a recent ATLAS experiment study which uses t t process are presented for comparison[105].

Table IV :
Limits on d g,Z V and d g,Z A at 95% CL corresponding to current and future four-top cross section measurements.Coupling Current four-top with 35.6 fb −1 Future four-top with 300 fb −1