Charming Top Decays with Flavor Changing Neutral Higgs Boson and WW at Hadron Colliders

We investigate the prospects for discovering a top quark decaying into one light Higgs boson ($h^0$) along with a charm quark ($c$) in top quark pair production at the CERN Large Hadron Collider (LHC) and future hadron colliers. A general two Higgs doublet model is adopted to study the signature of flavor changing neutral Higgs (FCNH) interactions with $t \to c h^0$, followed by $h^0 \to WW^* \to \ell^+ \ell^- +\notE_T$, where $h^0$ is the CP-even Higgs boson and $\notE_T$ stands for missing transverse energy from neutrinos. We study the discovery potential for this FCNH signal and physics background from dominant processes with realistic acceptance cuts as well as tagging and mistagging efficiencies. Promising results are found for the LHC running at 13 TeV and 14 TeV center-of-mass energy as well as future pp colliders at 27 TeV and 100 TeV.


I. INTRODUCTION
The Standard Model has been very successful in explaining almost all experimental data to date, culminating in the discovery of the the long awaited standard Higgs boson at the CERN Large Hadron Collider (LHC) [1,2]. The most important experimental goals of the LHC, future high energy hadron colliders, and the International Linear Collider (ILC) are to study the Higgs properties and to search for new physics beyond the Standard Model (SM) including additional Higgs bosons and flavor changing neutral Higgs (FCNH) interactions.
In the Standard Model there is one Higgs doublet, which generates masses for both vector bosons and fermions. There is no explanation for the large differences among Yukawa couplings of fermions with the Higgs boson. In addition, there are no flavor changing neutral currents (FCNC) mediated by gauge interactions or by Higgs interactions at the tree level. The top quark is the most massive elementary particle ever discovered. The fact that the Higgs boson (h 0 ) is lighter than the top quark (m t > M h ) makes it possible for the top quark to decay into the Higgs boson along with a charm quark (t → ch 0 ) kinematically. At the one loop level, the branching fraction of t → ch 0 is 3 × 10 −15 for M h = 125 GeV [3][4][5]. If this decay mode is detected in the near future, it would indicate a large tree-level coupling or a significant enhancement from beyond SM loop effects.
A general two Higgs doublet model (2HDM) usually contains flavor changing neutral Higgs (FCNH) interactions if there is no discrete symmetry to turn off tree-level FCNC [6,7]. In 1991, it was pointed out that top-charm FCNH coupling could be prominent [8] if the Yukawa couplings of fermions and the Higgs boson are comparable to the geometric mean of their mass [9]. A special two Higgs doublet model for the top quark (T2HDM) [10] might provide a reasonable explanation why the top quark is much more massive than other elementary fermions. In the T2HDM, top quark is the only elementary fermion acquiring its mass from a special Higgs doublet (φ 2 ) with a large vacuum expectation value (v 2 ≫ v 1 ). Since the up and charm quarks couple to another Higgs doublet (φ 1 ), there are FCNH interactions among the up-type quarks. The down type quarks have the same interactions as those in the SM.
The LHC has become a top quark factory. The production cross section of top quark pair (σ tt ) is approximately 820 pb in pp collisions with a 13 TeV center-of-mass energy (CM) energy ( √ s), and it becomes σ tt ≃ 970 pb at √ s = 14 TeV [13][14][15][16][17]. For an integrated luminosity of L = 100 fb −1 at √ s = 13 TeV, the LHC has produced more than 8 × 10 7 top quark pairs (tt) for m t ≃ 173.2 GeV [18,19]. For the same integrated luminosity at √ s = 14 TeV, the number of (tt) pairs generated would increase to about 1 × 10 8 . Thus, the LHC will provide great opportunities to study electroweak symmetry breaking as well as other important properties of the top quark and the Higgs boson. Most ATLAS and CMS measurements of the 125 GeV Higgs boson (h 0 ) are consistent with expectations for the Standard Model. The branching fractions of the standard Higgs boson are presented in Table I [20][21][22]. In a general two Higgs doublet model, let us consider the light Higgs scalar (h 0 ) as the SM Higgs boson in the alignment limit [23,24].
It is clear that the most probable decay channels are bb and W W with branching fractions B(h 0 → bb) ≃ 0.58 and B(h 0 → W W * ) ≃ 0.22 as show in Table I. However, the light Higgs boson was first discovered with h 0 → γγ and h 0 → ZZ * → 4ℓ, because these channels have less background and better mass resolutions. In the past few years, several theoretical studies and experimental searches have been completed for the charming top FCNH decay t → ch 0 with (a) h 0 → bb [25][26][27]30], (b) h 0 → ZZ * [28], (c) h 0 → γγ [29,30], and (d) Higgs decays into multileptons [31][32][33]. Recently, the ATLAS collaboration has placed tight limits on the FCNH branching fraction for t → ch 0 and the Yukawa coupling λ tch with Higgs boson decaying into multileptons [34] B(t → ch 0 ) ≤ 0.16% , and λ tch ≤ 0.077 , for the effective Lagrangian The LHC limits for the branching ratios can be translated to a limit on the flavor changing Yukawa coupling by a simple rescaling. It is a good approximation to consider a simple numerical relation between the FCNH Yukawa coupling (λ tch ) and the branching fraction of In this article, we focus on the discovery potential of the LHC in the search for the FCNH top decay t → ch 0 followed by h 0 → W W * → ℓ + ℓ − νν. We have evaluated production rates with full tree-level matrix elements including Breit-Wigner resonances for both the signal and the physics background. In addition, we optimize the acceptance cuts to effectively reduce the background with realistic b-tagging and mistagging efficiencies. Promising results are presented for the LHC with √ s = 13 TeV and √ s = 14 TeV as well as for future hadron colliders at √ s = 27 TeV and 100 TeV, for High Luminosities (HL) [36][37][38][39] of L = 300 fb −1 and 3000 fb −1 . Section II shows the production cross sections for the Higgs signal and the dominant background, as well as our strategy to determine the reconstructed masses for the top quark and the Higgs boson. Realistic acceptance cuts are discussed in Section III. Section IV presents the discovery potential at the LHC for √ s = 13 TeV and 14 TeV, as well as for future hadron colliders with for √ s = 27 TeV and 100 TeV. Our optimistic conclusions are drawn in Section V.

II. THE HIGGS SIGNAL AND PHYSICS BACKGROUND
In this section we present the cross section for the FCNH Higgs signal in pp collisions (pp → tt → tch 0 → bjj cℓℓνν + X, ℓ = e, µ) as well as for the dominant physics background processes. Figure 1 shows the Feynman diagram of top quark pair production in pp collisions from gluon fusion and quark-antiquark fusion, followed by one top quark decaying into a Higgs boson and a charm quark, while the other top quark decays into bW → bjj.

A. The Higgs Signal in Top Decay
Applying the Lagrangian in Eq. [1] with general Yukawa interactions for the light Higgs boson and fermions, we obtain the decay width of t → ch 0 where Let us define two variables, Combining LHC Higgs data and B physics, a recent study found constraints ρ tc ≤ 1.5 and ρ ct ≤ 0.1 [40]. That implies ǫ c < ∼ 0.2ρ tc for ρ tc ≃ 1. Hence we can write our decay width as, Using m t = 173.2 GeV, M h = 125.1 GeV and m c = 1.42 GeV [20], we obtain Since we have m c ≪ m t , r c ≪ 1, and ǫ c < ∼ 0.2|ρ tc |, it is a very good approximation to consider For typical values of parameters cos(β − α) = 0.1, |ρ tc | ∼ 1 and |ρ ct | ∼ 0.1, we have and For simplicity, we may adopt the following effective Lagrangian to study FCNH Yukawa interactions for the light CP-even Higgs boson (h 0 ) with the top quark (t) and the charm quark (c) where It is the effective coupling of the FCNH Yukawa coupling. Then the decay width for t → ch 0 [8] becomes We assume that the total decay width of the top quark is Then the branching fraction of t → ch 0 becomes As a case study, let us take the FCNH Yukawa couplings to be the geometric mean of the Yukawa couplings of the quarks that is also known as the Cheng-Sher (CS) Ansatz [9] or with m t = 173.2 GeV and m c = 1.42 GeV. Then the branching fraction of t → ch 0 becomes GeV. In general, we will consider g htc =ρ tc cos(β − α)/ √ 2 withρ tc and cos(β − α) as free parameters. We employ the programs MadGraph [41,42] and HELAS [43] to evaluate the exact matrix element for the FCNH signal in top decays from gluon fusion and quark-antiquark annihilation, where ℓ = e or µ. The cross section of the Higgs signal in FCNH top decays at the LHC and future hadron colliders for pp → tt → tch 0 → bjj cℓ + ℓ − νν + X is evaluated with the parton distribution functions of CT14LO [44,45] with a common value Q = M tt = the invariant mass of tt, for the renormalization scale (µ R ) and the factorization scale (µ F ). This choice of scale leads to a K factor of approximately 1.8 for top quark pair production. We have used the computer program Top++ [17] to evaluate higher order corrections. In addition, we have checked the tree-level signal cross section with narrow width approximation. That is, the cross section σ(pp → tt → tch 0 → bjj cℓ + ℓ − νν + X) is calculated as the product of cross section times branching fractions: In our analysis, we consider the FCNH signal from both tt → tch 0 → bjjcℓ + ℓ − ν ℓνℓ and tt → ch 0t →bjj cℓ + ℓ − ν ℓνℓ . In every event, we require that there should be one b jet and three light jets (j = u, d, s, c, or g in physics background). In addition, there are two leptons (ℓ = e or µ) and neutrinos, which will be lead to missing transverse energy (E / T ). Unless explicitly specified, q generally denotes a quark (q) or an anti-quark (q) and ℓ will represent a lepton (ℓ − ) or anti-lepton (ℓ + ). That means our FCNH signal leads to the final state of bjj cℓ + ℓ − ν ℓνℓ or bjjjℓ + ℓ − + E / T .

B. The Physics Background
The dominant physics background to the final state of bjjcℓ + ℓ − νν comes from top quark pair production along with two light jets (ttjj), pp → ttjj → bbjjW W → bbjjℓ + ℓ − νν + X, where every top quark decays into a b−quark as well as a W boson (W → ℓν) and a bjet is mis-identified as a c-jet. We have also considered backgrounds from pp → ttW → bbjjW W → bbjjℓ + ℓ − νν + X with one W boson decaying into jj, and pp → bbjjW W → bbjjℓ + ℓ − νν + X, excluding the contribution from ttjj and ttW . In addition, we have included pp → ccjjW W → ccjjℓ + ℓ − νν + X and pp → jjjjW W → jjjjℓ + ℓ − νν + X where j = u, d, s, or g. We evaluate the cross section of physics background in pp collisions with proper tagging and mistagging efficiencies. In our analysis, we adopt updated ATLAS tagging efficiencies [46,47]: the b tagging efficiency is∼ 70%, the probability that a c-jet is mistagged as a b-jet (ǫ c ) is approximately 14%, while the probability that any other jet is mistagged as a b-jet (ǫ j ) is 1%.

C. Mass Reconstruction
In this subsection, we demonstrate that the proposed Higgs signal comes from top quark pair production with tt → bjj ch 0 → bjj cℓ + ℓ − + E / T . We discuss our strategy to determine the reconstructed top mass as the invariant mass of bjj from t → bW → bjj along with another top quark decays into a Higgs boson and a charm quark t → ch 0 . Furthermore, we employ cluster transverse mass distributions for ℓ + ℓ − and cℓ + ℓ − with missing transverse energy (E / T ) from neutrinos. These distributions have broad peaks near M h and m t respectively as the kinematic characteristics of t → ch 0 → cℓ + ℓ − + E / T . Applying suitable cuts on the cluster transverse mass M T (ℓℓ, E / T ) as well as M T (cℓℓ, E / T ), we can greatly reduce the physics background and enhance the statistical significance for the Higgs signal.
In our analysis, we assume that the FCNH signal comes from top quark pair production with one top quark decaying into a charm quark and a Higgs boson (t → ch 0 → cW W → cℓ + νℓ −ν ) while the other decays hadronically (t → bW → bjj). In every event, there is one tagged b-jet and three light jets. Let us choose the pair of light jets that minimize |M jj − m W | and |M bjj − m t | as j 1 j 2 and label the other jet as j 3 ≃ c. That means, for a correctly reconstructed event, j 1 and j 2 are the products of a W decay such that their invariant mass distribution peaks at M j 1 j 2 ≃ m W . For a background event, one b is likely coming from the top decay t → bW → bjj while the other is either a mistagged c or a light quark jet coming from W decay, or a real b quark coming from the decay oft.
We present the invariant mass distributions for M j 1 j 2 and M bj 1 j 2 in FIG. 2 for the Higgs signal (tt → tch 0 ) and the dominant background (ttjj) with basic Cuts from CMS [51]: where p T (ℓ 1 ) ≥ p T (ℓ 2 ) and ∆R ≡ (∆φ) 2 + (∆η) 2 . It is clear to see that M j 1 j 2 distribution peaks at m W while dσ/dM bjj has a peak at m t . In a good reconstruction, the remaining light jet, j 3 ∼ c should reproduce the top quark mass with the momenta of charged leptons and neutrinos. To reconstruct the Higgs mass and top mass for t → ch 0 → cℓ + ℓ − + E / T , we use cluster transverse mass M T (ℓℓ, E / T ) and M T (cℓℓ, E / T ) [48,49], defined below, and where p T (ℓℓ) or p T (cℓℓ) is the total transverse momentum of all the visible particles and M ℓℓ or M cℓℓ is the invariant mass. Figure 3 presents the cluster transverse mass distributions (dσ/dM T (ℓℓ, E / T ) and (dσ/dM T (cℓℓ, E / T ) for the Higgs signal in pp collisions, dσ/dM T (pp → tt → tch 0 → tcW W → bjjcℓ + ℓ − + E / T + X (fb/GeV), with basic cuts defined in Eq. [22], as well as |M jj − m W | ≤ 0.15 × m W and |M bjj − m t | ≤ 0.20 × m t . the cluster transverse mass distributions for ℓ + ℓ − and cℓ + ℓ − for the Higgs signal (tt → tch 0 ) and the dominant background (ttjj) with basic cuts defined in Eq. [22] as well as invariant mass cuts Note that dσ/dM T (ℓℓ, E / T peaks near M h while dσ/dM T (cℓℓ, E / T has a peak near m t .
It is clear that there are pronounced peaks at m W and m t in the invariant mass distributions of jets as shown in FIG. 2. We can also see broad peaks near M h and m t in the cluster transverse mass distributions: where M * is the value of invariant mass or cluster transverse mass with a peak of the distribution. These distributions provide powerful selection tools to remove physics background while maintaining the Higgs signal.

III. REALISTIC ACCEPTANCE CUTS
To study the discovery potential of this charming FCNH signal from top decays at the LHC, we have applied realistic basic cuts listed in Eq. [22] and tagging efficiencies for b−jets. In addition to basic cuts we apply cuts on invariant mass of jets and cluster transverse mass of ℓℓ and cℓℓ to effectively veto the background events: Gaussian smearing of momenta: for jets and for charged leptons with individual terms added in quadrature.

IV. DISCOVERY POTENTIAL AT THE LHC
Applying all realistic cuts, we present our results for the Higgs signal at the LHC with √ s = 13 TeV and √ s = 14 TeV as well as cross sections for future hadron colliders with √ s = 27 TeV and √ s = 100 TeV in Table II. Here we have kept cos(β − α) = 0.1. Later we will vary it from 0.01 to 0.2 for discovery contours. Cross sections for dominant background processes are presented in Table III.
To estimate the discovery potential at the LHC we include curves that correspond to the minimal cross section of signal (σ S ) required by our discovery criterion described in the following. We define the signal to be observable if the lower limit on the signal plus background is larger than the corresponding upper limit on the background with statistical fluctuations  or equivalently, Here L is the integrated luminosity, σ S is the cross section of the FCNH signal, and σ B is the background cross section. The parameter N specifies the level or probability of discovery. We take N = 2.5, which corresponds to a 5σ signal. For Lσ B ≫ 1, this requirement becomes similar to where N S is the signal number of events, N B is the background number of events, and N SS is the statistical significance, which is commonly used in the literature. If the background has fewer than 25 events for a given luminosity, we employ the Poisson distribution and require that the Poisson probability for the SM background to fluctuate to this level is less than 2.87 × 10 −7 , i.e. an equivalent probability to a 5-sigma fluctuation with Gaussian statistics. Figure 4 shows the Higgs signal cross section as a function ofρ tc , along with cross section of total background and the most dominant background process (ttjj) for the CERN Large Hadron Collider with √ s =13 and 14 TeV. We have also shown, minimum cross section required for 5σ significance at L = 36.1f b −1 and higher luminosities for the future HL LHC [36,37], i.e L = 300 and 3000 f b −1 . All tagging efficiencies and K factors discussed above are included. Our analysis suggests an improvement in the reach of ATLAS at a luminosity of 3000 f b −1 , which gets better at higher energies(HE-LHC), i.e √ s =27 and 100 TeV, as shown in Figure 5. We present the 5σ discovery reach at the LHC for (a)  Figure 7 shows the discovery contours for √ s =27 and 100 TeV. High energy (HE) LHC with high luminosity (HL) is quite promising as it nearly covers the entire parameter space that we have used in our analysis. . Also shown is the current limit on λ tch =ρ tc cos(β − α) (red dotdash) set by ATLAS [34]. The shaded region above this curve is excluded at 95% CL.

V. CONCLUSIONS
It is a generic possibility of particle theories beyond the Standard Model to have contributions to tree-level FCNH interactions, especially for the third generation quarks. These contributions arise naturally in models with additional Higgs doublets, such as the special two Higgs doublet model for the top quark (T2HDM), or a general 2HDM. In the alignment (dot). Also shown is the current limit on λ tch =ρ tc cos(β − α) (red dotdash) set by ATLAS [34]. The shaded region above this curve is excluded at 95% CL.
limit, the light Higgs boson (h 0 ) resembles the standard Higgs boson, and it has a mass below the top mass. This could engender the rare decay t → ch 0 . We investigated the prospects for discovering such a decay at the LHC, focusing on the channel where tt are pair produced and subsequently decay, one haronically and the other through the FCNH mode. The primary background for this signal is a ttjj with both top quarks decaying leptonically. This background involves one b−jet mis-tagged as a c jet,and two other light jets, along with two leptons and missing transverse energy. Nonetheless, by taking advantage of the available kinematic information, we can reconstruct the resonances of the signal and reject much of the background.
Based on our analysis, we find that LHC at √ s = 14 TeV, with L = 3000 fb −1 , can probe to as low as B(t → ch 0 ) ≃ 1.17 × 10 −3 , λ tch =ρ tc cos(β − α) ≃ 0.069. It gets better with √ s = 27 TeV and √ s = 100 TeV, which can reach upto B(t → ch 0 ) ≃ 6.1 × 10 −4 , λ tch ≃ 0.048 and B(t → ch 0 ) ≃ 2 × 10 −4 , λ tch ≃ 0.028 respectively. We look forward to being guided by more new experimental results as we explore interesting physics of electroweak symmetry breaking (EWSB) and FCNH interactions. While the properties of the Higgs boson goes under further scrutiny as data accumulate, perhaps a dedicated FCNH t → ch 0 search should be undertaken, for upcoming HL LHC and further HE-LHC as well as future high energy hadron collider with a CM energy of 100 TeV.