Constraining $t \to u$ flavor changing neutral Higgs coupling at the LHC

We study the constraints on $t\to u$ flavor changing neutral Higgs (FCNH) coupling, and how it may be explored further at the Large Hadron Collider (LHC). In the general two Higgs doublet model, such transitions can be induced by a nonzero $\rho_{tu}$ Yukawa coupling. We show that such couplings can be constrained by existing searches at the LHC for $m_H$, $m_A$ and, $m_{H^+}$ in the sub-TeV range, where $H$, $A$ and $H^+$ are the exotic $CP$-even, $CP$-odd and charged scalars. We find that a dedicated $ug\to t H/tA \to t t \bar u$ search can probe the available parameter space of $\rho_{tu}$ down to a few percent level for $200\,\mbox{GeV} \lesssim m_H,\,m_A \lesssim 600$ GeV, with discovery possible at high luminosity. Effects of how other extra top Yukawa couplings, such as $\rho_{tc}$ and $\rho_{tt}$, dilute the sensitivity of the $\rho_{tu}$ probe are discussed.


I. INTRODUCTION
The 125 GeV scalar boson h, only discovered [1] in 2012, combines with the longitudinal components of the massive vector bosons to form the weak scalar doublet of the Standard Model (SM). But with one scalar doublet established naturally brings in the question of a second doublet, i.e. the so-called [2] two Higgs doublet model (2HDM). Although it is popular [2] to use a discrete symmetry to impose "Natural Flavor Conservation" [3] so all "dangerous" flavor changing neutral Higgs (FCNH) couplings are removed, it is also well known that this may not be necessary [2]. Indeed, upon the discovery of h, the t → ch decay [4] search was advocated [5] and quickly pursued by ATLAS [6] at the LHC, and further efforts are recorded [7] by the Particle Data Group (PDG). As another example, CMS saw early on with 8 TeV data some hint [8] for h → τ µ decay. Though it subsequently disappeared [7], it did bring about considerable interest in FCNH couplings.
As elucidated in Ref. [5], the t → ch decay occurs via the c γ ρ tc coupling, where c γ ≡ cos γ is the mixing angle of h with the CP -even scalar boson H of the exotic doublet, which is the one that carries the FCNH ρ tc coupling. With subsequent Higgs property studies [9][10][11], it became clear that h resembles very closely the Higgs boson of SM, and the h-H mixing angle c γ seems rather small. This may be the reason behind the non-observation [7] of t → ch so far, without implying ρ tc to be small. Demonstrating [12] that there is quite some parameter space for c γ to be small in the 2HDM context, it was advocated that mass-mixing hierarchy suppression [4] of FCNH couplings involving lighter generation fermions, augmented by the smallness of c γ ("alignment"), can explain the absence of low energy FCNH effects without the need to invoke NFC. Thus, extra Yukawa couplings are rather general in the 2HDM setting and should be pursued experimentally, and not just at the LHC. The "Model III" of Ref. [4] was therefore elevated to the general 2HDM (g2HDM), even promoted [13] as a possible future "SM2", the SM with two Higgs doublets.
Having introduced the g2HDM, we write down the couplings of the CP -even scalars h, H and CP -odd scalar A to fermions as [5,12,14] where L, R ≡ (1 ∓ γ 5 )/2, i, j = 1, 2, 3 are generation indices and summed over, c γ = cos γ and s γ = sin γ, and u, d, are up and down type quarks and charged leptons, respectively. The matrices λ f ij ≡ √ 2m f i δ ij /v are the usual Yukawa couplings related to mass in SM, whereas ρ f ij are in general non-diagonal and complex. We do not consider H + effects in this work, but consider ρ tuinduced processes at the LHC, including ug → tH/tA production (see Fig. 1). We refrain from quoting the Higgs potential for g2HDM here. Instead, we treat the scalar boson masses as parameters, but state that we have checked that they satisfy the usual requirements of perturbativity, positivity and unitarity, as well as other constraints such as electroweak oblique parameters (see e.g. Refs. [15][16][17]).
In the experimental pursuit of t → ch, one actually searches for t → ch, uh simultaneously. It turns out that the bound on t → uh is not better than t → ch, i.e. the current 95% C.L. bound from ATLAS [18] gives based on 36.1 fb −1 data at 13 TeV, which is better than the CMS result [19] based on similar amount of data. This may seem surprising since single top production via ρ tu is taken into account. One may think that ρ tu should naturally be much smaller than ρ tc , but this is not based   [20] that B → µν decay probes the ρ tu ρ τ µ product. The process will be pursued by Belle II [21], where a deviation of the ratio R µ/τ B = B(B → µν)/B(B → τν) from the SM expectation of 0.0045 would indicate [20] nonzero ρ tu in g2HDM. What can LHC do to check ρ tu = 0? In this paper we focus on ug → tH/tA → ttū production, i.e. ug → tH/tA (see Fig. 1) followed by H/A → tū, leading to same-sign top signature.
In the next section we first summarize the constraint on ρ tu from searches at the LHC, including tttt search. We turn to ug → tH/tA → ttū (conjugate process always implied unless specified) in Sec. III and use it to constrain or discover the ρ tu coupling [22]. We focus on m A , m H ∈ (200, 600) GeV, which is allowed in g2HDM [15][16][17]. Heavier m A , m H are possible, but discovery prospect is reduced due to rapid fall off in parton luminosities. As the ρ tc -induced cg → tH/tA → ttc process [23][24][25][26] (see also Refs. [27][28][29][30]) can be misidentified as ug → tH/tA → ttū due to inefficient c-jet tagging, we outline a procedure to distinguish between the two processes. We comment briefly on the effect of the diagonal ρ tt coupling in Sec. IV, before offering our conclusion.

II. CURRENT CONSTRAINTS ON ρtu
As stated, our actual knowledge of the strength of ρ tu is actually quite poor.
The h boson couples to tu as c γ ρ tu , hence B(t → uh) search constrains ρ tu coupling for finite c γ . The latest ATLAS result based on 36.1 fb −1 data at 13 TeV sets the 95% C.L. limit B(t → uh) < 1.1 × 10 −3 [18], as given in Eq. (2), which is better than the CMS limit [19] of B(t → uh) < 4.7 × 10 −3 based on 35.9 fb −1 . We illustrate the ATLAS limit [18] in Fig. 2 as the blue shaded region in the c γ -ρ tu plane, while the weaker CMS limit is not displayed. Taking c γ = 0.2 as example, one gets |ρ tu | 0.5 at 95% C.L., which is rather weak, and weakens further for smaller c γ .
Stronger constraints on ρ tu arise from the tttt, or 4t search, which does not depend on c γ . Let us first focus on the CMS 4t search, which is based on 137 fb −1 at 13 TeV, i.e. with full Run 2 data [31], more than three times the data size of the preceding study [32]. Depending on the number of charged leptons (e, µ) and b-tagged jets, the search in Ref. [31] is divided into several signal regions (SRs) and two control regions (CRs), with the baseline selection criterion of at least two same-sign leptons. We find that the most stringent constraint on ρ tu arises from the control region of ttW , which is denoted as CRW [31]. Induced by the ρ tu coupling, the ug → tH/tA → ttū process would contribute to this CRW.
CRW of the CMS 4t search [31] is defined as containing two same-sign leptons plus two to five jets with two b-tagged. The selection cuts are as follows. Leading (subleading) lepton transverse momentum should satisfy p T > 25 (20) GeV. The pseudorapidity of electrons  To reduce the Drell-Yan background with a chargemisidentified electron, events with same-sign electron pairs with m ee < 12 GeV are rejected. With these selection cuts, CMS reports 338 observed events in CRW, while the expected total number of events (SM backgrounds plus 4t) is at 335 ± 18 [31].
To calculate our limits, we generate signal events using MadGraph5 aMC@NLO [34] (denoted as Mad-Graph5 aMC) at leading order (LO) with default parton distribution function (PDF) set NN23LO1 [35], interface with PYTHIA 6.4 [36] for showering and hadronization, and MLM matching [37] prescription for matrix element and parton shower merging. The event samples are then fed into Delphes 3.4.2 [38] for fast detector simulation, where we follow the CMS-based detector analysis for CRW, utilize the default b-tagging efficiency and light-jet rejection, with jets reconstructed via anti-k T algorithm. The effective model is implemented in FeynRules [39].
The ρ tu -induced process pp → tH/tA → ttū (nonresonant ug → ttū and t-channel H/A exchange uu → tt processes are included) with both top quarks decaying semileptonically contributes to CRW of CMS 4t search. Setting all other ρ ij = 0, we estimate the contribution for ρ tu = 1 and then scale the cross section by |ρ tu | 2 , assuming narrow H/A widths with B(H/A → tū) = 50%. We then demand that the sum of the number of events expected from SM and those from ρ tu -induced processes agree with the observed number of events within 2σ uncertainty of expectations. We display the 2σ exclusion limits obtained via CRW in Fig. 3 as the purple shaded regions, where we assume Gaussian behavior for simplicity. That is, we simplify and do not follow the more precise estimation [40] of exclusion limits using likelihood function with Poisson counting.  ATLAS has also searched for 4t production [41] with 139 fb −1 , but categorizing into different SRs and CRs. Again, the CR for ttW , called CRttW2 , is the most relevant. It is defined as at least two same-sign leptons (e ± µ ± or µ ± µ ± ), plus at least four jets with at least two b-tagged. The same-sign leptons are required to have p T > 28 GeV with |η µ | < 2.5 and |η e | < 1.5. All jets should satisfy p T > 25 GeV and |η| < 2.5. If the number of b-jets is equal to two, or the number of b-jets is ≥ 3 but with no more than 5 jets, the scalar p T sum over all jets and same-sign leptons (note the difference in definition from CMS), H T , should satisfy H T < 500 GeV. Unlike CRW for CMS, ATLAS does not give the observed number of events in CRttW2 , but provides a figure of comparison between data and prediction in the variable p T (see Ref. [41] for definition). We extract [42] from this figure the number of expected and observed events for CRttW2 , finding 378 ± 10 and 380, respectively, where we have simply added the errors in quadrature for the expected events from each p T bin.
To extract the constraint, we follow the same event selection procedure as before, but use the ATLAS-based detector card of Delphes. Assuming that the number of events for pp → tH/tA → ttū plus SM stay within 2σ of the expected number of events, we illustrate the exclusion limits from ATLAS CRttW2 by the cyan shaded regions in Fig. 3. Mainly due to differences in selection cuts, the ATLAS constraint on ρ tu is weaker. From CMS 4t search we find ρ tu 0.13-0.15 is still allowed for 200 GeV m H 400 GeV, while slightly larger values open up for m H > 400 GeV. In this vein, we stress that we have illustrated for |m H − m A | = 50 GeV, as there is strong cancellation between ug → tH → ttc and ug → tA → ttc amplitudes for H, A that are nearly degenerate in mass and width.
We remark that supersymmetry search in similar event topologies can in principle constrain ρ tu . However, such analyses now typically require H T and/or missing energy that are too large for our purpose. The selection criteria could be relaxed with R-parity violation, e.g. the ATLAS search [43] for squark pair production, but the selection cuts are still too strong to give meaningful constraint. We note further that the ATLAS search for new phenomena in events [44] with same-sign dileptons and b-jets (36.1 fb −1 at 13 TeV) has similar SRs, but the cuts are again strong and the selection criteria different, such that it does not give relevant constraint for our study.

III. SAME-SIGN TOP SIGNATURE FROM ρtu
Even though the existing CMS 4t search with full LHC Run 2 data can set meaningful constraints on ρ tu , it is not optimized for ug → tH/tA → ttū search. In this section, we perform a dedicated study of the ug → tH/tA → ttū process at the LHC, targeting exclusion or discovery of a stand-alone ρ tu coupling.

A. Discovery and Exclusion Limits
The pp → tH/tA + X → ttū + X process can be searched for in events containing same-sign dilepton (ee, µµ, eµ), at least three jets with at least two b-tagged and one non-b-tagged jet, plus E miss T , which we denote as same-sign top. The final state topology will also receive contribution from uu → tt via t-channel A/H exchange which we include as signal. The dominant backgrounds are ttZ, ttW , 4t and tth, while 3t+W , 3t+j and tZ+ jets are subdominant. In addition, if the lepton charge gets misidentified (charge-or Q-flip), with the misidentification efficiency at 2.2 × 10 −5 [44][45][46], the tt+ jets and Z/γ * + jets processes would also contribute. We remark that the CMS study [47] with similar final state topology but with slightly different cuts finds the "nonprompt" backgrounds at ∼ 1.5 times the ttW background, which is significant. As the nonprompt backgrounds are not properly modeled in Monte Carlo simulations, we simply add this component to the overall background at 1.5 times the ttW background after selection cuts. We generate signal and background events as in the previous section at LO via MadGraph5 aMC for √ s = 14 TeV, follow the same showering, hadronization and ME, and parton shower merging and matching. We adopt here the default ATLAS-based detector card of Delphes. The LO ttW − (ttW + ), ttZ, 4t, tth and, tZ+ jets cross sections are normalized to next-to-leading order K factors 1. 35 [49], and 1.56 [50], respectively. We assume the same K factor fortZ+ jets background for simplicity. The Q-flip Z/γ * + jets and tt+ jets backgrounds are corrected to next-to-next-to-leading (NNLO) order cross sections by 1.27 [51] and 1.84 [52], respectively. We utilize FEWZ 3.1 [53] to obtain the NNLO factor for Z/γ * + jets background. The signal cross sections and 3t + W , 3t + j backgrounds are kept at LO.
To reduce backgrounds, we follow a cut based analysis that is different from CRW of CMS 4t search, and optimize for pp → tA/tH + X → ttū + X as follows. The leading (subleading) lepton should have p T > 25 (20) GeV, while |η| < 2.5 for both leptons. All three jets should satisfy p T > 20 GeV and |η| < 2.5. The missing energy in each event should satisfy E miss T > 30 GeV. The separation ∆R between a lepton and any jets (∆R j ), between the two b-jets (∆R bb ), and between any two leptons (∆R ) should all satisfy ∆R > 0.4. We finally demand that selected events should satisfy H T > 300 GeV, where H T is defined according to ATLAS,i.e. including the p T of the two leading leptons.
We plot in Fig. 4 Tables I and II, respectively. We have assumed m H to be lighter than m A . To estimate the exclusion limit (2σ) and discovery potential (5σ), we utilize the test statistics [40] where L(x|n) = e −x x n /n! is the likelihood function of Poisson probabilities with n the observed number of events, and x is either the number of events predicted by the background-only hypothesis b, or signal plus background hypothesis s + b. For exclusion (s + b hypothesis) we demand Z(s + b|b) ≥ 2 for 2σ, while for discovery (b hypothesis) Z(b|s + b) ≥ 5 for 5σ. Utilizing the signal cross sections for the reference ρ tu = 1 value in Table I  and the background cross sections in Table II, we find the exclusion and discovery contours in m H -ρ tu plane (with m A = m H +50 GeV) for different integrated luminosities in the left and right panels of Fig. 3, respectively, where we have interpolated the contours for m H values other than the ones given in Table I Fig. 3 as blue and red lines, respectively, for the three different integrated luminosities of 150 (dotted), 300 (solid) and 3000 fb −1 (dashed). The 150 fb −1 data size reflects the target luminosity for Run 2, but the contours are generated with √ s = 14 TeV rather than 13 TeV. We find that, with 150 (300) fb −1 one could exclude |ρ tu | 0. 13

B. Distinguishing ρtu and ρtc Effects
Unless the final state charm can be efficiently tagged (which is not the case), the cg → tH/tA → ttc processes also give rise to the same-sign top signature for nonzero ρ tc . In this subsection, we outline a procedure to distinguish same-sign top signatures induced by ρ tu vs ρ tc .
The valence u-quark induced ug → tH/tA → ttū process has much larger cross section compared toūg → tH/tA →ttu. So one expects the former to be considerably larger than the latter. To understand the relative significance of ug → tH/tA → ttū, we take a benchmark point with ρ tu = 0.13, m H , m A = 300, 350 GeV that is still allowed by Fig. 3. To distinguish between the signature induced by ρ tu vs ρ tc , we separate positively charged vs negatively charged same-sign dileptons. Following the same analysis as in the previous subsection, we find the signal (background) cross sections at √ s = 14 TeV for the ++ and −− charged dileptons to be 0.5 fb and 0.06 fb (∼ 2.35 fb and ∼ 1.38 fb), respectively. We then find the significance for dileptons with ++ charge to be ∼ 4.1σ (∼ 13σ) with 300 (3000) fb −1 , while the corresponding significance for −− charged dileptons is at ∼ 1σ (∼ 2.7σ). Note that the former (latter) arises from the ug → tH/tA → ttū (ūg → tH/tA → ttū) process. Thus, separating the ++ from −− same-sign dilepton events, one expects the ++ dileptons to emerge earlier than the −−. We have again assumed the non-prompt background to be ∼ 1.5 times the ttW background, while Q-flip background is assumed at half the value given in Table II for the respective signatures.
In comparison, the c-quark induced cg → tH/tA → ttc andc-quark inducedcg →tH/tA →ttc processes should have similar cross sections. Assuming all ρ ij = 0 except ρ tc we find, for example, that ρ tc = 0.36 is allowed at 2σ by CRW of CMS 4t search for m H , m A = 300, 350 GeV. Following the same cut based analysis for these parameter values, we find the cross sections at √ s = 14 TeV for ++ and −− charged dilepton processes at 0.074 and 0.081 fb, respectively, which translates to ∼ 2.7σ (∼ 8.4σ) and ∼ 3.8σ (∼ 11.9σ) with 300 (3000) fb −1 integrated luminosity. That is, both ++ and −− samesign dilepton events are at similar level, which contrasts with the ρ tu -induced same-sign dilepton events.
So far we have discussed scenarios when either ρ tu or ρ tc is nonzero. Recasting the results from Ref. [54], it was found [55] that |ρ * tu ρ tc | 0.02 is excluded by D-D mixing for m H ≈ m A ≈ m H ± 500 GeV, which would be even more stringent for lighter exotic scalars. This gives the ballpark of the constraint when both ρ tu and ρ tc are nonzero. A detailed analysis treating both ρ tu and ρ tc nonzero would be studied elsewhere.

IV. DISCUSSION AND OUTLOOK
Let us comment on the impact of turning on ρ tt . As ρ tt = 0 would induce H/A → tt decays, the 4t search constraints from CRW of CMS and CRttW2 of ATLAS would weaken for m H (m A ) > 2m t due to B(H/A → tt) = 0. In particular, ρ tt = 0.5 is still allowed for m H , m A , m H ± ∼ 200-600 GeV [17]. For ρ tu = 0.15 and ρ tt = 0.5, B(H/A → tū +tu) would be suppressed by ∼ 70%-90% for 400 GeV m H 600 GeV, weakening the limits from CRW of CMS 4t search. Nonzero ρ tu and ρ tt may also induce ug → tH/tA → ttt (triple-top) and ug → bH + → btb signatures, where the latter process may even emerge from Run 2 data [17]. Such final states can also arise from ρ tc coupling. However, separating ++ and −− same-sign dileptons can in principle differentiate between ρ tu and ρ tc couplings. Scenarios when ρ tu , ρ tc and ρ tt are all nonzero would receive multiple constraints, in particular from flavor physics. A study involving all three couplings is beyond the scope of this work. However, based on the extensive work on ug, cg → tH/tA → ttū, ttc, ttt processes reported or cited here, we advertise a public twiki page [56] where interested LHC workers could use to join the quest.
At this point, it is useful to recall that ρ tt provides a robust driver [57] for electroweak baryogenesis (EWBG) in g2HDM, even for |ρ tt | values at the percent level, which provides strong motivation. If ρ tt is sizable, it would make probing nonzero ρ tu more challenging at the LHC. However, we have emphasized our current experimental knowledge, and such knowledge on ρ tu comes primarily from the LHC at present. Even if one takes EWBG into consideration, we note a second, backup mechanism [57]: ρ tc at O(1) with near maximal phase can also drive EWBG if ρ tt accidentally vanishes in g2HDM. However, it would still make probing ρ tu = 0 rather challenging, and the LHC experiments would have to try their best at the HL-LHC, as we have tried to illustrate. This is especially so if B(B → µν)/B(B → µν) is found by Belle II to differ from SM expectation. On the other hand, baryogenesis may not occur through g2HDM, hence one should exploit the full potential of the LHC.
In summary, we pose the question: "If the flavor chang-ing neutral Higgs coupling ρ tu is nonzero, how can one check this at the LHC?" With only ρ tu = 0, we show that it is possible with HL-LHC, by comparing the significance of positively vs negatively charged same-sign dilepton events. Nonzero ρ tc can mimic ρ tu -induced events, while ρ tt = 0 would further dilute the sensitivity to finite ρ tu . The issue would become important if the ratio of B → µν decay rate to B → τν is found by Belle II to deviate from Standard Model expectation.