Flavor changing top decays to charm and Higgs with $\tau \tau$ at the LHC

We investigate the prospects of discovering the top quark decay into a charm quark and a Higgs boson ($t \to c h^0$) in top quark pair production at the CERN Large Hadron Collider (LHC). A general two Higgs doublet model is adopted to study flavor changing neutral Higgs (FCNH) interactions. We perform a parton level analysis as well as Monte Carlo simulations using \textsc{Pythia}~8 and \textsc{Delphes} to study the flavor changing top quark decay $t \to c h^0$, followed by the Higgs decaying into $\tau^+ \tau^-$, with the other top quark decaying to a bottom quark ($b$) and two light jets ($t\to bW\to bjj$). To reduce the physics background to the Higgs signal, only the leptonic decays of tau leptons are used, $\tau^+\tau^- \to e^\pm\mu^\mp +\slashed{E}_T$, where $\slashed{E}_T$ represents the missing transverse energy from the neutrinos. In order to reconstruct the Higgs boson and top quark masses as well as to effectively remove the physics background, the collinear approximation for the highly boosted tau decays is employed. Our analysis suggests that a high energy LHC at $\sqrt{s} = 27$ TeV will be able to discover this FCNH signal with an integrated luminosity $\mathcal{L} = 3$ ab$^{-1}$ for a branching fraction ${\cal B}(t \to ch^0) \agt 1.4 \times 10^{-4}$ that corresponds to a FCNH coupling $|\lambda_{tch}| \agt 0.023$. This FCNH coupling is significantly below the current ATLAS combined upper limit of $|\lambda_{tch}| = 0.064$.


I. INTRODUCTION
The discovery of the Higgs boson in 2012 [1,2] completes the experimental observation of the particle spectrum predicted by the Standard Model (SM). A primary goal of the high luminosity and higher energy Large Hadron Collider (LHC) is the precision testing of the SM and the search for physics beyond the Standard Model (BSM), especially the interactions of the Higgs boson, the top quark, and sources of CP violation. Several experimental searches [3][4][5][6] are being performed to improve the understanding of Higgs boson interactions with SM particles and to search for possible extensions of the Higgs sector.
We adopt the Yukawa Lagrangian in a general two Higgs doublet model [45,46] as where P L,R ≡ (1∓γ 5 )/2, c β−α ≡ cos(β −α), s β−α ≡ sin(β −α), α is the mixing angle between neutral Higgs scalars, tan β ≡ v 2 /v 1 [47] is the ratio of the vacuum expectation values of the two Higgs doublets, Q F is the fermion charge, and the κ matrices are diagonal and fixed by fermion masses to κ F = √ 2m F /v with v ≈ 246 GeV, while the matrices ρ contain both diagonal and off-diagonal elements with free parameters. In addition, F, U, D, L represent elementary fermions, up-type quarks, down-type quarks, and charged leptons, respectively. The matrix elements ρ are the FCNH couplings to the fermions. Almost all experimental data are consistent with the Standard Model [48,49], which implies all two Higgs doublet models must be in the decoupling [50] or the alignment limit [51,52] with one SM-like light scalar (h 0 ) that has a mass of 125 GeV.
Recently the ATLAS Collaboration [40] combined several channels to search for t → ch 0 with h 0 → bb, h 0 → τ τ with at least one hadronic tau decay, h 0 → W W * , ZZ * , τ + τ − (same sign 2 , 3 ), and h 0 → γγ, and put a strong constraint on the branching fraction B(t → ch 0 ) ≤ 1.1 × 10 −3 . This leads to an upper limit on the FCNH Yukawa coupling |λ tch | for the effective Lagrangian, with the relation between λ tch and the t → ch 0 branching fraction [53] being In this article, we investigate the discovery potential of the top quark decay into a charm quark and a Higgs boson (t → ch 0 ) followed by the Higgs boson decaying into τ + τ − in top quark pair production at the CERN Large Hadron Collider (LHC). To investigate the discovery potential of a flavor changing neutral Higgs boson signal with low physics background, we consider only the leptonic decays of the tau leptons, τ + τ − → e ± µ ∓ + / E T , where / E T is the missing transverse energy in the event from the neutrinos. This is complementary to the ATLAS searches for same charge dileptons.
We perform a parton level analysis as well as a Monte Carlo simulation using Pythia 8 [54] and Delphes [55] to study the FCNH decay of one top quark while the other top quark decays hadronically to a bottom quark (b) and two light jets: pp → tt → bW ± ch 0 → bjjcτ + τ − + X. We have calculated the production rates using the full tree level matrix elements including the Breit-Wigner resonance for both signal and background process. In addition, we optimize our acceptance using a standard selection based technique, as well as using a boosted decision tree to improve the signal to background ratio and statistical significance.
Since we did not apply charm tagging, our analysis is suitable for a general search for t → qh 0 , q = u, c. Many previous studies have adopted the Cheng-Sher Ansatz [56] as the benchmark Yukawa coupling where q = u, c and v ≈ 246 GeV is the Higgs vacuum expectation value. The FCNH couplings as the geometric mean for top and charm quarks is which has been excluded by recent ATLAS experiment [40]. For simplicity, we assume λ tch λ tuh and focus on the search for t → ch 0 . To verify that the associated quark is a charm, we will need to apply charm tagging.
There are several aspects to note in this analysis. To reconstruct the Higgs boson and the top quark, the collinear approximation of tau decays [57] is used. The collinear approximation for tau decays with physical momentum fractions x i (0 < x i < 1), where x i = p( i )/p(τ ), i = 1, 2, more effectively reduced the physics background than the centrality requirement suggested in Refs. [37,40]. Furthermore, the energy of the charm quark in the top quark rest frame provides good acceptance for the FCNH top signal while rejecting background [32,58]. Promising results are presented for the LHC with √ s = 14 TeV and 27 TeV.

II. HIGGS SIGNAL AND EVENT SELECTIONS
This section presents the cross section for the FCNH signal t → ch 0 from top quark pair production and outlines our search strategy for this signal at the LHC. We focus on the discovery channel with one top quark decaying hadronically (t → bjj), while the other top quark decays into a charm quark and a Higgs boson (h 0 ) followed by h 0 → τ + τ − → e ± µ ∓ + / E T . Unless explicitly specified, q generally denotes a quark (q) or an anti-quark (q) and ± will represent an e ± or µ ± . This means our FCNH signal has the final state of pp → tt → bjjce ± µ ∓ + / E T + X, where X represents all other particles produced in pp collisions. Since the mass of the Higgs boson is much greater than the tau lepton's mass (M h m τ ), the tau leptons are highly boosted. Therefore, the collinear approximation of the tau decay [57] is employed to reconstruct the Higgs boson mass and the top quark mass.
At parton level, our analysis employs MadGraph5-aMC-NLO [59] to calculate tree-level cross sections for the full process pp → tt → bjjch 0 → bjjcτ + τ − +X along with the collinear approximation of tau decays [57]. The parton level cross section is evaluated using the CT14LO parton distribution functions (PDFs) [60]. For simplicity, the factorization scale (µ F ) and the renormalization scale (µ R ) are chosen to be the invariant mass of the top quark pair (M tt ). With the above scale choices and PDFs, our current estimates suggest a K-Factor of ≈ 1.8, and is approximately the same for all three energies ( √ s = 13, 14, and 27 TeV), investigated for top quark pair production at the LHC. The K-factors are calculated using TOP++ [61].
This analysis employees the full tree-level matrix elements to evaluate the cross section for the FCNH signal and physics background. In addition, a consistency check for the tree-level signal cross section has been performed in the narrow width approximation by calculating the cross section σ(pp → tt → tch 0 → bjj c ± 1 ∓ 2 / E T + X) as the product of cross section times branching fractions: To evaluate the branching fraction of t → ch 0 , the effective Lagrangian in Eq. 3 is employed. The resulting decay width is then obtained as with r h = M h /m t and r c = m c /m t . Assuming that the total decay width of the top quark is the branching fraction of t → ch 0 is Comparing this with the Yukawa Lagrangian in Eq. 1, we can express To present the results, the more convenient free parametersρ tc and cos(β − α) are chosen for the FCNH Yukawa couplings.
In the event level analysis, parton level samples are generated from MadGraph using TauDecay-UFO [62] to model τ decays, and then the sample is processed with Pythia 8 [54] and Delphes [55] to generate events with hadronization, showering, and detector effects. In addition, the MLM-matching/merging [63] algorithm is applied to match the additional hadronized jets in each event with partons to avoid double counting jets that are generated by parton showering from final state radiation for all background processes.
To provide a realistic estimate for production rates at the LHC, we evaluate the cross section for the FCNH signal and physics background in pp collisions with the proper tagging and mistagging efficiencies. The ATLAS tagging efficiencies [64] are adopted to evaluate the cross section for the FCNH signal and physics background. The b tagging efficiency is 0.7, the probability that a c-jet is mistagged as a b-jet ( c ) is approximately 0.14, while the probability that a light jet (u, d, s, g) is mistagged as a b-jet ( j ) is 0.01.

A. Event Selections
Our FCNH signal comes from top quark pair production with one top quark decaying into a charm quark and a Higgs boson while the other top quark decays to an all hadronic final state. Every event is required to contain at least four jets, including exactly one that is identified as a b jet. In addition, there are two opposite charge leptons of different flavor (e ± µ ∓ ) with missing transverse energy from neutrinos.
We adopt the following basic requirements, which are similar to the ATLAS and CMS h 0 → τ + τ − studies [69].
Since the b-quark jet is selected through tagging, this leaves three jets to be identified as two light-flavor jets and a c-quark jet. The two light-flavor jets, j 1 j 2 , are selected by minimizing |M jj − m W | and |M bjj − m t |. The remaining jet is labeled as the c-quark jet. For the event to be correctly reconstructed, j 1 and j 2 must result from the decay of a W boson, therefore their invariant mass distribution M j 1 j 2 peaks at M W ≈ 80.4 GeV and M bj 1 j 2 has a peak at m t ≈ 173.2 GeV. Using the ATLAS mass resolution [65], the reconstructed W and top quark masses are required to lie in the mass windows ∆M j 1 j 2 = 0.20M W and ∆M bj 1 j 2 = 0.25m t .

B. Higgs Mass Reconstruction
For the FCNH signal, t → ch 0 → cτ + τ − → ce ± µ ∓ + / E T , the reconstruction is performed two ways: (a) using the invariant τ + τ − mass from the Higgs decay and the invariant mass of cτ + τ − from the top quark decay, which have sharp peaks near M H and m t , and (b) as in Ref. [41], using the cluster transverse masses of + − and c + − , which have broad peaks near M H and m t .
The Higgs boson mass can be reconstructed by applying the collinear approximation [66][67][68] on the τ decay products and the missing transverse momentum 2-vector, p / T . Taking x i to be the momentum fractions carried away from the decay by i , i = 1, 2, we have: This yields two equations for x 1 and x 2 that can be solved to reconstruct the two original τ 4-momenta p µ . Physically x i is constrained to 0 < x i < 1, i = 1, 2, which reduces the background. Figure 1 presents the invariant mass distributions M col (τ τ ) and M col (c, τ τ ), which have sharp peaks near the Higgs boson and top quark masses, respectively. We require the reconstructed Higgs boson mass and top quark mass to lie in the mass windows ∆M τ τ = 0.20M h and ∆M cτ τ = 0.25m t using the ATLAS mass resolution [69]. We note that improvements in the discovery potential are possible by reducing the τ pair mass resolution ∆M τ τ . Furthermore, we employ the cluster transverse mass distributions for e ± µ ∓ and ce ± µ ∓ with missing transverse energy (E / T ) from the neutrinos to confirm the Higgs boson mass and top quark mass reconstruction. These distributions have broad peaks near M h and m t , respectively, as the kinematic characteristic of t → ch 0 → c e ± µ ∓ + / E T . The cluster transverse mass [70] is defined as where C = ± ∓ or c ± ∓ , p T ( ) or p T (c ) is the total transverse momentum of all the visible particles, while M and M c are the invariant cluster masses. In most cases, the physics background can be reduced and the statistical significance for the Higgs boson signal enhanced if we apply a suitable requirement on the cluster transverse mass distributions [41] M T ( , E / T ) and M T (c , E / T ). We have found that acceptance requirement on M τ τ and M cτ τ is much more effective than mass requirement on the cluster transverse masses. After the mass selection on the collinear invariant mass, the effects of additional requirements on the cluster transverse mass are negligible.

C. Centrality of Missing Transverse Energy
To further suppress the physics background, the authors of Refs. [37,40] suggest the use of the centrality of the missing transverse energy (C MET ) where φ 1,2 are the azimuthal angles of the two leptons (e or µ) in the transverse plane, and φ MET is the azimuthal angle of the transverse missing energy. Figure 2 shows the centrality C MET for the FCNH signal from t → ch 0 and the dominant background ttjj. This is found to be less stringent than the requirement on the physical momentum fractions 0 < x i < 1, i = 1, 2, which leads to C MET > 1.

III. THE PHYSICS BACKGROUND
The dominant background to the signal is from ttjj, j = q or g. Here both top quarks decay leptonically (t → b ν) to the desired final state combination of leptons. This comprises more than 95% of the total background. The other dominant contribution is from pp → bbjjτ τ → bbjje ± µ ∓ + / E T + X and pp → bbjjW + W − → bbjje ± µ ∓ + / E T + X (without a tt contribution) as well as ttW ± and ttZ. For all of the backgrounds, one b jet is selected while the other b jet is mis-identified as a light jet. Events with two b-jets having p T (b) > 20 GeV and |η(b)| < 2.5 are vetoed [71,72]. We calculate the cross section for each of the backgrounds separately using MadGraph and apply the same event selection procedure as for the signal. The irreducible background from pp → ttZ + X and pp → tth 0 + X with the subsequent decay of Z → τ + τ − and h 0 → τ + τ − are negligible after all acceptance requirements and the two b veto are imposed.   We scale our backgrounds to NLO using K-factors of 1.8 for tt + 2j, bbjjτ τ , and bbjjW + W − for all energies i.e. √ s= 13, 14, and 27 TeV. For ttW and ttZ we use the following K-factors calculated with MadGraph5-aMC-NLO.
After applying the event acceptance criteria, we reconstruct the invariant mass variables M j 1 j 2 , M bj 1 j 2 , M τ τ , and M cτ τ , as discussed in the previous section. In addition, the energy of the charm quark (E c ) in the rest frame of the top quark is reconstructed to discriminate the t → ch 0 signal from background [32,58]. For the flavor changing top decay of t → ch 0 , the E c distribution exhibits a peak at the following value, Requiring 29 GeV < E c < 54 GeV, the background is significantly reduced while most of the signal is maintained. Figure 3 presents the energy distributions of the charm quark in the top quark rest frame. From the invariant mass and the charm quark energy distributions at the parton and event levels, the following mass requirements are deduced  These requirements are chosen to remove the physics background in a manner that maximizes the statistical significance of the FCNH signal.

IV. DISCOVERY POTENTIAL AT THE PARTON LEVEL
Applying all the selection criteria at parton level for √ s = 13, 14 and 27 TeV, our signal cross sections for λ tch = 0.064 are shown in Table II. The cross sections with λ tch = 0.01 are also presented for a simple estimate to find the cross sections for other values of this FCNH Yukawa coupling. The cross sections for the backgrounds after applying the selection requirements are presented in Table III.   Figure 4 presents the estimated statistical significance (N SS ) as a function of λ tch / √ 2 for the parton level analysis, where N SS is calculated using [75], Here N S and N B are number of signal and background events, respectively.    Table IV presents a comparison between this study and our previous study for t → ch 0 → cW W * → ce ± µ ∓ + / E T [41]. This analysis suggests that h 0 → τ τ is much cleaner, because the Higgs boson mass is fully reconstructed and the energy of the charm quark in the top quark rest frame improves the statistical significance using the optimized requirements.  , 300f b −1 (medium green solid) and L = 3000f b −1 (light green dashed) Also shown is the current limit on λ tch =ρ tc cos(β − α) (red dotted dashed) set by ATLAS [40].

V. EVENT LEVEL ANALYSIS WITH BOOSTED DECISION TREES
In this section, we present the event level analysis using the event generator Pythia 8 [54] and the detector simulation program Delphes [55]. From this analysis, the cross sections for the FCNH signal and the backgrounds are shown in Table V  For the event level analysis, the mass resolutions are worse than at the parton level. Therefore, the mass selection window is relaxed and the selected events are used to train and then process it through the BDT, which contains 1000 trees at a depth of 5. The BDT response is shown in Fig 6. The BDT is employed to optimize the selection requirements and improve the statistical significance. Event selection using the BDT classifier improves the statistical significance of the analysis relative to using an event based selection on kinematic and acceptance variables only. Table VI shows that the BDT analysis improves the statistical significance by more than a factor of two.  Table VII presents the 95% confidence level limits on λ tch at √ s = 13, 14 and 27 TeV using an integrated L = 300 and 3000 fb −1 . In addition, the minimum λ tch for 5σ discovery at the LHC is presented in Table VIII. We conclude that it will be difficult to discover this channel at 13 and 14 TeV colliders in this channel, but a 27 TeV high energy collider holds promise for this signature.  3000 fb −1 . It is clear that the high energy LHC at √ s = 27 TeV with a high luminosity L = 3000 fb −1 significantly improves the discovery potential of t → ch 0 beyond the current ATLAS limit [40] λ tch = 0.064. √ s = 27 TeV with 300f b −1 (medium green solid) and L = 3000f b −1 (light green dashed), as well as the event level discovery contours and 3σ contour (yellow dashed) Also shown is the current limit on λ tch =ρ tc cos(β − α) (red dotted dashed) set by ATLAS [40].
We have illustrated the improvement in statistical significance achieved by using a boosted decision trees classifier relative to a cut based analysis. To avoid overtraining the BDT due to low statistics in the event level analysis, the more restrictive parton level invariant mass requirements are relaxed. We then rely on the BDT to optimize the selections on the kinematic variables. Our goal is to improve the significance by using the BDT to set the requirements on the invariant masses and the charm-quark energy, which is a strong signal to background discriminant. We encourage our experimental colleagues to include the charmquark energy as an effective discriminant to further improve the potential of detecting this FCNH signature at the the LHC.

VI. CONCLUSIONS
Many beyond the Standard Model theories contain tree-level contributions to FCNH interactions, especially for the third generation fermions. These contributions arise naturally in models with additional Higgs doublets, such as the special two Higgs doublet model for the top quark (T2HDM) [77], or a general 2HDM [45,46]. In the decoupling [50] or the alignment limits [51,52], the light Higgs boson (h 0 ) resembles the standard model Higgs boson with a mass less than the top quark. This could engender the rare decay t → ch 0 .
We investigated the prospects for such a discovery at the LHC, focusing on the tt production channel and their subsequent decay, where one decays hadronically and the other through the FCNH mode. The primary background for this signal is ttjj with both top quarks decaying leptonically. This background contains one b jet mis-identified as a c jet, and two additional light jets, along with two leptons and missing transverse energy. Taking advantage of the available kinematic information, the h 0 and top quark masses in the signal can be reconstructed and much of the background rejected.
Based on our parton level analysis, we find that the LHC at √ s = 14 TeV, with L = 3000 fb −1 , can probe to as low as B(t → ch 0 ) ≈ 2.5 × 10 −4 with λ tch ≈ 0.033. At √ s = 27 TeV, the reach is B(t → ch 0 ) ≈ 1.4 × 10 −4 with λ tch ≈ 0.023. The event level analysis implies that there are technical challenges to reach the discovery potential of the parton level analysis, especially, improving efficiencies and mass reconstruction with high precision for final states with missing transverse energy from neutrinos.
In summary, we have made several significant contributions to search for charming top decays with an associated Higgs boson: (i) The t → ch 0 → τ + τ − → ce ± µ ∓ has not been previously investigated as a dedicated discovery channel.
(ii) We demonstrate the effectiveness of reconstructing the Higgs boson and the top quark masses by applying the collinear approximation to the tau decays.
(iii) We show that the requirement on the momentum fractions 0 ≤ x i ≤ 1, i = 1, 2 is more effective at removing background and improving the significance than the requirement on centrality (C MET > 0).
(iv) Our requirement on the energy of the charm quark (E c ) in the top quark rest frame significantly reduces the background and improves the significance.
(v) We have performed the first investigation of the discovery potential of t → ch 0 → τ + τ − → ce ± µ ∓ for a high energy pp collider at √ s = 27 TeV.
There are two useful features in the τ + τ − channel: (a) the reconstruction of M h and m t invariant masses applying the collinear approximation, and (b) the selection requirement on the charm quark energy in the top quark rest frame for reducing the physics background. This leads to the τ + τ − discovery channel having a better reach in λ tch by a factor of approximately two over the W + W − channel.
We look forward to being guided by new experimental results as we explore the interesting physics of EWSB and FCNH. While the properties of the Higgs boson undergo scrutiny as data is accumulated, dedicated FCNH searches for t → ch 0 and φ 0 → tc +tc, φ 0 = H 0 , A 0 should be undertaken for the upcoming high luminosity LHC and future high energy pp colliders.