Top FCNC induced by a Z' boson

We consider a Z' model where the Z' boson couples only to up and top quarks. As a simple setup, one can consider a family-dependent $U(1)'$ symmetry, under which only right-handed up-type quarks are charged. After symmetry breaking, the right-handed quarks mix with each other and top flavor-changing neutral currents (FCNCs) mediated by the Z' boson can be generated at tree level. We take into account several processes to probe the top FCNCs, and find that the same sign top quark pair production or the triple top quark production is the most capable of testing the top FCNCs. We also consider a few non-FCNC processes to probe the Z' boson, for example, dijet production at the LHC.


Introduction
The Standard Model (SM) has passed through stringent tests in various experiments. It is, however, widely believed that the SM is not the end of the story and new physics beyond the SM will be probed at experiments, e.g., the LHC. So far, a lot of models have been suggested for new physics. One of the simple extensions of the SM is to consider a new U (1) gauge symmetry. Such a U (1) gauge symmetry could be effectively induced at low energy from high-scale models like E 6 GUT models (see, for example, Ref. [1,2]). One of the distinctive signatures of U (1) models would be the production of a Z boson, which is a massive gauge boson of U (1) symmetry.
One of the typical searches for the Z boson proceeds through production of the dilepton or dijet at the Z resonance. That Z boson decays into a dilepton or dijet, and it usually has flavor-independent couplings to fermions up to the differences among U (1) charges. It is apparent that any processes in which the Z boson is involved could be a probe of the U (1) model. In this work, we take into account a more general case in which couplings of the Z boson to fermions are flavor-dependent, and flavor-changing neutral currents (FCNCs) are induced by the Z boson at tree level after electroweak and U (1) symmetry breaking. Then, we note that several processes via FCNCs could provide opportunities to probe the U (1) model. Depending on U (1) charge assignment and the strength of FCNCs, flavor-violating processes may be more efficient to probe the Z model in comparison with the conventional dilepton or dijet production.
In the SM, FCNCs exist at the loop level owing to the Glashow-Iliopoulos-Maiani (GIM) mechanism [3]. In addition to the loop suppression factor, most of the FCNC observables usually have another suppression factor due to the GIM mechanism. Therefore, FCNCs are very rare in the SM and have been paid great attention to in the search for new physics beyond the SM. In particular, mixing between a meson and antimeson is very sensitive to Z interaction with FCNCs relevant to quarks. Such FCNCs can contribute to the mixing at tree level or via box diagrams depending on underlying models describing interactions.
In the case of top FCNCs, they can contribute to D 0 -D 0 mixing through a loop-level diagram, but both t-c-X and t-u-X FCNCs, where X is a scalar or vector boson, are required for the mixing. Hence, if only one of t-u-X and t-c-X FCNCs exists, it must be searched for through FCNC top decays or direct production of top quarks. So far several observables have been proposed to study top FCNCs at colliders. The search for top FCNCs in which the top quark couples to SM particles would provide the most stringent bound on FCNC couplings in the top quark decay because the top quark is the heaviest particle in the SM and a lot of top quarks are produced at the LHC. For example, current bounds on the branching ratio of the top quark decay into a u quark and Z boson (or Higgs boson, H) at 95% confidence level (CL) are BR(t → Zu) < 2.4 (1.7) × 10 −4 and BR(t → hu) < 4.7 (2.4) × 10 −3 as given by the CMS (ATLAS) Collaboration [4][5][6][7].
However, if top FCNCs are generated from a new particle heavier than the top quark, then top quark decays through FCNCs would be highly suppressed. Thus, one has to investigate the FCNCs through the production of the top quark. In the case of top-Higgs FCNCs via t-q-H (q = u or c) couplings, Atwood et al. [8] discussed three processes, pp → tt(tt), pp → tj(tj), and pp → tjH(tjH), where j denotes a jet, and found that the last process is the most capable of yielding the best upper bound on the top FCNCs.
In this work, we consider top FCNCs, which are generated by a new gauge boson, Z , heavier than the top quark. In the search for FCNCs, we take into account four FCNC processes: same-sign top quark pair production, pp → tt (tt); single top quark production, pp → tj(tj); radiative Z production, pp → tZ j(tZ j); and triple top quark production, pp → ttt (ttt). We calculate cross sections for the four FCNC processes and compare them with each other. We search for the best FCNC process that yields the strongest upper bound on the top FCNC coupling. Since the FCNC couplings are related to non-FCNC couplings, such as t-t-Z coupling, we also consider non-FCNC processes, dijet production, pp → jj, top quark pair production, pp → tt, and four top quark production, pp → tttt, to compare them with FCNC processes. Furthermore, since the triple top quark production involves both FCNC and non-FCNC couplings, the constraints from the non-FCNC processes must be considered [9]. This paper is organized as follows. In Sec. 2, we introduce an effective model which provides FCNCs through a Z exchange. For an example of the UV complete model, we consider a U (1) model in which only right-handed up-type quarks are charged. In Sec. 3, we perform numerical analyses for the flavor-violating processes as well as the flavorconserving processes at the parton level. Section 4 is devoted to the numerical analysis at detector level for the same-sign top quark pair production and triple top quark production. Finally, we summarize our results in Sec. 5.

Simple model
In this section, we introduce a model that predicts FCNCs through a Z exchange. The Z boson would be a gauge boson of an additional U (1) gauge symmetry or, in a more general case, any larger gauge symmetry. We assume that only right-handed up and top quarks couple to the Z boson in the quark mass eigenstates after electroweak and U (1) symmetry breaking. Otherwise the models would be strongly constrained by Then the relevant Lagrangian for the Z interaction with right-handed up-type quarks is given by where g ij (i, j = u or t) are Z couplings to right-handed up-type quarks and can be expressed in terms of the coupling constant g of U (1) symmetry and mixing angles among right-handed up-type quarks [10][11][12][13]. For a simple example of the UV complete model, one may consider the chiral U (1) model, which was originally proposed to resolve the anomaly in the top quark forward-backward asymmetry at the Tevatron [10][11][12][13]. In this class of models, only right-handed up-type quarks are charged under U (1) symmetry in a flavor-dependent manner, while other fermion fields are uncharged or charged in a flavor-universal manner. For one of possible scenarios, one can assume that only right-handed top quark is charged under U (1) symmetry, i.e., (u R , c R , t R ) = (0, 0, 1), in the interaction basis. * After symmetry breaking, three right-handed quarks mix with each other, and in the mass eigenstates, FCNCs mediated by the Z boson may be generated. If we consider mixing between only right-handed up and top quarks, the Z couplings can be obtained as g uu = g sin 2 α, g ut = −g tu = g sin α cos α, g tt = g cos 2 α, where α is the mixing between u and t quarks. For a small mixing angle, sin α 1, one obtains the following hierarchy among couplings: g tt g ut g uu . In the case of sin α ∼ cos α, all three couplings become comparable to one another. For charge assignment on right-handed up-type quarks, (u R , c R , t R ) = (1, 0, 0), the hierarchy among couplings could be g uu g ut g tt when the mixing angle is small. In general, one must consider mixing among three right-handed up-type quarks. If we assume Eq. (2), we obtain the relation among couplings as where only two of the four couplings are independent. However, this relation is valid only to mixing between two quarks in Eq. (2). If we consider the mixing among three right-handed up-type quarks, the relation is invalid, and one must extend Eq. (2) to the more general case including g uc , g ct , and g cc as shown in Ref. [11]. In general, the coupling g ij is expressed in terms of the rotation matirx, R u , which diagonalizes the mass matrix of the right-handed up-type quarks: where u k is the U (1) charge assignment on the right-handed up-type quarks. We note that g ij could be complex in principle, but we shall assume all g ij are real for simplicity in this paper.
For g uu g ut , we can ignore Z uū interaction so that the Z boson might not be searched for from dijet signals at colliders. Then, the FCNC processes via Z ut (Z tū) interaction may be the most sensitive to probe this class of Z model, whereas for g ut g uu , dijet production through uū → Z → uū would become a more sensitive process.
Before the study of the detailed phenomenology, let us discuss how to satisfy the anomaly-free conditions. In our setup, the U (1) gauge symmetry is chiral, so extra fermions charged under U (1) are required to achieve the anomaly-free conditions. When the U (1) charge assignment is generic, some simple matter contents are discussed in Refs. [10,11]. In the case with (u R , c R , t R ) = (0, 0, 1), the anomaly-free U (1) gauge symmetry can be achieved, introducing a SM-vectorlike fermion, T , whose charge of the SM gauge symmetry is the same as that of the right-handed top quark. Defining the U (1) charges of left-handed and right-handed T as 1 and 0, respectively, we obtain an anomaly-free U (1) model.
In addition, the flavor-dependent U (1) gauge symmetry often requires extra scalars, since the symmetry forbids some Yukawa couplings and disturbs realization of the realistic mass matrices for quarks. As is discussed in Refs. [10][11][12][13], a simple way is to introduce extra Higgs doublets charged under U (1) . The extra Higgs fields deviate the ρ parameter from one at tree level, while they could relax the bound on the same-sign top signals in the collider experiments [10][11][12] and could explain the deviation in the semileptonic B decay [13].
In this paper, we do not discuss the detail of the Higgs sector and the extra fermions, simply assuming that Z dominates over the Higgs contribution to the flavor-violating processes as well as flavor-conserving processes. We also assume that the extra fermions are heavy, and Z does not decay to the extra fermions and the extra fermions are irrelevant to Z physics.

Numerical analysis at parton level
In this section, we study several processes at the parton level with which one can search for couplings in the Lagrangian (1). We fix the center-of-momentum energy as √ s = 13 TeV. The integrated luminosity L dt = 137 fb −1 is taken into account in numerical analysis. We perform the numerical analysis at leading order in α s by making use of MADGRAPH [14,15] after implementing the simple model given in Eq. (1) into the generator.
For a reference value, we take the Z mass, m Z = 2 TeV, but we scan m Z from 1 to 5 TeV in numerical analysis. In top quark pair production and four top quark production, we also consider lower m Z mass than 1 TeV. Since m Z > 2m t in the overall region, the Z boson can decay into top quarks. In this model, the decay channels to quarks are Z → uū, tt, ut, and tū. In principle, there could exist small mixing between the Z boson and SM neutral bosons through kinetic mixing or one-loop corrections, but it is ignored by assuming that the mixing is very small. Hence, the Z boson is leptophobic since it does not couple to the SM leptons. In a UV complete model, the Z boson can also decay into two scalar bosons or SM gauge bosons, and the decay pattern depends on a given scalar potential. Therefore, the decay width, Γ Z , of the Z boson depends on parameters in the scalar potential and corresponding kinematics among scalar and gauge bosons. Here, we take Γ Z /m Z = 0.1 for simplicity.
Since the Z boson couples to up and top quarks, we consider four processes which could play important roles to probe top FCNCs: (a) the same-sign top quark pair production, pp → tt(tt); (b) the single top quark production, pp → tj u (tj u ), where j u = u orū; (c) the radiative Z production, pp → tj u Z (tj u Z ); and (d) the triple top quark production, pp → ttt(ttt). We also consider three flavor-conserving processes, which may play important roles in searching for couplings in Eq. (1): (a) the dijet production, pp → Z → uū; (b) the top quark pair production, pp → tt; and (c) the four top quark production, pp → tttt. All processes are mutually connected and must be considered simultaneously to find more detailed properties of the Z boson. In Ref. [16], some FCNC processes were taken into account simultaneously, but there are two different points from our analysis. The model in that paper is based on an effective theory in which the scale of new interactions is beyond the reach of near-future colliders. Therefore, new interactions at current colliders are expressed in terms of dimension-6 operators. Another point which one must keep in mind is that CP -conserving interactions could be generated by renormalization group evolution even though they do not exist at the new interaction scale. Then, for a more complete discussion, CP -conserving processes have to be considered.

Same-sign top quark pair production
In this subsection, we consider the same-sign top quark pair production induced by the Z boson, which proceeds through pp → tt(tt). The same-sign top quark pair production has been paid attention to as a sensitive process searching for new physics because the SM background is very small [17][18][19]. In particular, it could reject a lot of new models, which might be a resolution of the anomaly in the top forward-backward asymmetry at the Tevatron [20]. In our model, this process can occur in the t channel through a Z exchange with uu orūū initial partons. Thus, only the g ut (= −g tu ) coupling is involved in production amplitudes. For a reference value m Z = 2 TeV, we obtain cross sections for the same-sign top quark pair production at the leading order in α s : σ(pp → tt) = 1.67 g 4 ut pb, σ(pp →tt) = 0.028 g 4 ut pb.
(4) Note that production cross sections are proportional to g 4 ut . For g ut = 0.5, the sum of production cross sections is about 0.11 pb. Because the SM backgrounds are very small in these processes, one can get a strong constraint on g ut [21]. The upper bound on the cross section for the same-sign top quark pair production is obtained as σ(pp → tt(tt)) < 1.2 pb by the CMS Collaboration at 95% CL with the integrated luminosity 35.9 fb −1 at √ s = 13 TeV [22], while the upper bound at ATLAS is 89 fb at 95% CL with the integrated luminosity 36.1 fb −1 at √ s = 13 TeV [24]. Then the FCNC coupling is constrained as g ut < 0.92 (0.48) for m Z = 2 TeV by the CMS (ATLAS) bound.
In Fig. 1, we show the results for the same-sign top quark pair production. The left panel represents the total cross section for pp → tt (red line) and pp →tt (blue line) in units of pb divided by g 4 ut as a function of the Z mass in units of TeV at √ s = 13 TeV. The black line is the sum of them. The horizontal lines indicate the upper bounds for the same-sign top quark pair production at CMS (black dotted line) and ATLAS (gray dashed line). For g ut = 1, the region of m Z > 4.7 TeV is allowed by the ATLAS bound and for m Z = 1 TeV, g ut > 0.28 is excluded. Since the same-sign top quark pair production is forbidden at tree level in the SM, we find the region where the signal events do not exceed 1 [8]. The right panel of Fig. 1 represents the contour plot of g ut vs m Z (in TeV) for the number of signals, S = 1, for an integrated luminosity L dt = 137 fb −1 . Therefore, we expect that g ut ∼ 0.046 might be searched for through the same-sign top quark pair production for an integrated luminosity of 137 fb −1 when m Z = 2 TeV. Up to m Z ∼ 5 TeV, the g ut 0.1 region could be ruled out. However, this result is based on the analysis at the parton level. If one considers more realistic analyses or experimental uncertainties, the bound could be approximately doubled [8] or greater [23,24].

Single top quark production
In this subsection, we consider the single top quark production in association with a u orū quark. We note that four processes contribute to the single top quark production mediated by the Z boson: pp → tu(tū,tu,tū).
In the production of tu ortū, there exists only one t-channel diagram mediated by the Z boson, while both s-and t-channel diagrams via a Z exchange contribute to the tū or tu production. In each process, one of the vertices with which the Z boson is involved contains g ut (or g tu ), while the other vertex contains g uu . Therefore, the cross section for the single top quark production is proportional to g 2 uu g 2 ut . For m Z = 2 TeV, we calculate the cross sections for each process at √ s = 13 TeV, where we implement the cuts on the associated u (orū) quark as its transverse momentum p T ≥ 30 GeV and rapidity |η| < 2.7. Because the parton density of the u quark is greater than that of theū quark inside the proton, the cross section for the tu production, whose parton process is uu → tu, is the greatest among the four processes in Eqs. (5a)-(5d). The small difference between the production cross sections for tū andtu is due to the kinematic cuts on jets. Since an incident u quark would be more energetic than an TeV. The red, blue, green, and purple lines correspond to tu, tū,tu, andtū production, respectively, and the black line is the sum of them.
incidentū quark, the u quark in the final state would be faster than theū quark in the final state statistically. Then, more u quarks would be removed by the cuts on jets and the tū production would have a slightly larger cross section than thetu production. For m Z = 2 TeV and g ut = g uu = 0.5, the sum of the cross sections for the single top quark production via a Z exchange is about 0.46 pb.
In Fig. 2, we draw the cross sections for the single top quark production (in pb) divided by g 2 ut g 2 uu as a function of m Z (in TeV) at √ s = 13 TeV. The red, blue, green, and purple lines correspond to tu, tū,tu, andtū production, respectively. Their sum is represented by the black line. Thetū production is suppressed in the overall region due to small incident parton density of theū quark in comparison with that of the u quark. For m Z ≈ 1 TeV, the production cross sections for tū ortu are comparable to that of tu. However, for a larger value of m Z , the tu production process dominates over the other single top quark production processes.
Note that the cross section for the same-sign top quark pair production is proportional to g 4 ut , while that for the single top quark production is to g 2 ut g 2 uu . The cross section for the single top quark production strongly depends on the value of g uu as well as g ut . For larger g uu , the single top quark production may be more sensitive to probe the top FCNCs, while the same-sign top quark pair production could be more sensitive for relatively smaller g uu .
To calculate the sensitivity, we need to calculate the SM backgrounds for the single top quark production. We take into account pp → tj,tj, tjj,tjj, tb, andtb processes with the cuts on jets: p T ≥ 30 GeV and |η| < 2.7. The bounds on the couplings g uu and g ut are determined by the ratio of signal to SM backgrounds, S/ √ B, at 2σ level. The b-jet identification efficiency is assumed to be 50 %.
In Fig. 3, we depict the contour plots of g ut vs m Z for signal significance S/ √ B = 2 at the production level for L dt = 137 fb −1 at √ s = 13 TeV. The red, blue, and purple lines correspond to the cases of g uu = 0.5, g uu = 0.1, and g ut = g uu , respectively. The contour plots of g ut vs m Z for signal significance S/ √ B = 2 at the production level for L dt = 137 fb −1 at √ s = 13 TeV. The red, blue, and purple lines correspond to the cases of g uu = 0.5, g uu = 0.1, and g ut = g uu , respectively.
Here, we compare the bounds from the single top quark pair production with those from the same-sign top quark pair production. As shown in Fig. 1, the bound on g ut could reach about 0.046 for m Z = 2 TeV with the integrated luminosity L dt = 137 fb −1 in the same-sign top quark pair production. For m Z = 5 TeV, the bounds on g ut could be about 0.10, while the bounds on g ut from the single top quark production depend on g uu as we already mentioned. For g uu = 0.5 and m Z = 2 TeV, the bounds on g ut can reach 0.18 with the integrated luminosity L dt = 137 fb −1 in the single top quark production. However, for g uu = 0.1 and m Z = 2 TeV, the bounds on g ut could reach 0.90, which is about five times larger than the g uu = 0.5 case. To obtain bounds similar to those in the same-sign top quark pair production, g uu must be about 2.0, but this value is ruled out by the dijet production at the LHC, as we will show soon. Therefore, we conclude that the same-sign top quark pair production is more capable of yielding more stringent bounds on the top FCNC coupling than the single top quark production.

Radiative Z production
In this subsection, we take into account the Z production radiated from the u or t quark line, where relevant processes are pp → tūZ ortuZ [25,26]. These processes contain the FCNC coupling g ut . The emitted Z boson decays into uū (tt, ut, tū), a scalar and a gauge boson, or two gauge bosons.
Before the discussion on the Z production, we briefly compare the radiative Z production in this work with the radiative Higgs (H) production by the top-Higgs FCNC through pp → tūH ortuH [8]. The latter is dominated by the on-shell production of a tt pair, followed by t(t) → Hu(ū) decays. The cross section is enhanced by on-shell property of the intermediate t ort because the Higgs boson is lighter than the top quark. The Higgs boson is measured through its decays into bb, W W * , or ZZ * pairs, which are governed by the SM interactions. Therefore, the total cross section for the radiative H production is proportional to y 2 tu , where y tu is the Yukawa coupling responsible for the top-Higgs FCNC. In comparison with the radiative H production, the same-sign top quark pair production proceeds through a t-channel exchange of a Higgs boson in the parton process uu → tt orūū →tt. Then, the cross section for the same-sign top quark pair production becomes proportional to y 4 tu . Because of the intermediate on-shell top quark and dependence on the top-Higgs FCNC coupling, the radiative Higgs production can provide more stringent bounds on the top-Higgs FCNC than the same-sign top quark pair production [8].
The previous mechanism does not work in the radiative Z production with a t-u-Z FCNC coupling and a Z boson heavier than the top quark. Here, the Z boson is radiated from off-shell u(ū) or t(t) quarks so that production amplitudes are suppressed at least by a factor of O(m 2 t /ŝ), whereŝ is the center-of-momentum energy of parton processes. Another point which one must consider is that the produced Z boson decays into uū, tt, ut, tū, or a pair of bosons. Then, the cross section for the radiative Z production followed by the decay of the Z boson would be proportional to g 4 , which is of the same order as that for the same-sign top quark pair production. Therefore, we could not expect a dramatic increase of the sensitivity to the FCNC coupling in the radiative Z production.
The radiative Z production processes which we take into account are pp → tt(uū) followed by subsequent off-shell decays t(t) → Z u(ū) or u(ū) → Z t(t). The subsequent decays contain a FCNC coupling g ut , while tt or uū production amplitudes have a dependence on Z -involved couplings g 2 ij , where i, j = u or t, or do not contain Z interactions. Therefore, the total cross section can be expressed as the sum of terms proportional to g 2 ut , g 2 ut g 2 ij , and g 2 ut g 4 ij . In Fig. 4, we show the cross sections for the radiative Z production, pp → tjZ (in pb) for g ij = 1 (left) and g ij = 0.5 (right) as functions of m Z (in TeV) at √ s = 13 TeV. The red, blue, green, and purple lines correspond to tuZ , tūZ , tuZ , andtūZ , respectively, and the black line is the sum of them. As was discussed in the single top quark production, the cross section for the tuZ production is the largest because of the parton density of the incident u quark. The small difference between the tūZ andtuZ processes stems from the kinematic cuts (p T ≥ 30 GeV and |η| ≤ 2.7) as was discussed in the previous section. The sum of the cross sections for the radiative Z production is about 0.3 pb for m Z = 1 TeV and g ij = 1, but it decreases to 10 −2 fb for m Z = 5 TeV.
Since the cross sections for the radiative Z production are given in terms of g 2 ut , g 4 ut g 2 ij , g 2 ut g 4 ij , and g 6 ut , we cannot express them in a simple form like the same-sign top quark pair production or single top quark production. However, we find that the contribution of the g 2 ut terms to the total cross section is dominant over the other terms. For m Z ≥ 2 TeV, its contribution is more than 85 % and even for a lighter Z with m Z = 1 TeV, it becomes about 71 %. Hence, for m Z = 2 TeV, the cross sections can be approximated as where the sum of the cross sections is σ(pp →tjZ ) ∼ 10.7 g 2 ut fb.
If one considers the decay of the Z boson, for example, Z → uū, the cross section would be O(1) fb for g ut = 1. The dominant SM backgrounds for the radiative Z production followed by the Z decay into uū are pp → t(t)jjj and pp → tb(tb)jj, where the b-jet identification efficiency must be taken into account in the latter case. Without implementing any cut on the invariant mass of a pair of final jets, the cross sections of the SM backgrounds reach O(100) pb. If one implements the condition that the invariant mass of a pair of jets is between m Z − Γ Z /2 and m Z + Γ Z /2, the SM backgrounds could become negligible. The situation for the SM backgrounds with the cuts on the invariant mass of a pair of jets is similar to that for the same-sign top quark pair production. If we compare the radiative Z production with the same-sign top quark pair production, the latter has a larger cross section by a factor of O(10 2 ). Therefore, we conclude that the same-sign top quark pair production is more capable of yielding better bounds on the FCNC coupling g ut .

Triple top quark production
Another possible test for the top FCNCs is the triple top quark production via ug → tZ → ttt orūg →tZ → ttt. The production amplitudes are proportional to g ut g tt , where g tt can be constrained from the top quark pair production and/or four top quark production. In the case of g tt g uu,ut , the cross section for the triple top quark production would be negligible in comparison with the single top quark production whose production amplitude is proportional to g uu g ut . On the other hand, in the case of g uu g tt,ut , the single top quark production would be less sensitive to probe top FCNCs than the triplet top quark production. Therefore, the sensitivity of each process to the top FCNCs strongly depends on the couplings g uu and g tt .
For m Z = 2 TeV, the cross sections for the triple top quark production at √ s = 13 TeV are obtained as σ(pp → ttt) = 0.11 g 2 ut g 2 tt pb, (8) σ(pp → ttt) = 0.0054 g 2 ut g 2 tt pb, where the latter is much suppressed because of theū-quark parton density in a proton in comparison with that of the u quark. For g ut ∼ g tt ∼ 0.5, the sum of the triple top quark production cross sections is about 7.1 fb. In Fig. 5, we plot the cross sections for the triple top quark production (in pb) divided by g 2 ut g 2 tt as a function of m Z (in TeV) at √ s = 13 TeV. The red and blue lines correspond to ttt and ttt production, respectively, and the black line is the sum of them. For m Z = 1 TeV, the cross section reaches about O(1) pb, but it becomes O(1) fb for m Z = 5 TeV.
The SM backgrounds for the triple top quark production which we consider are pp → tttj, tttj, tttb, tttb, tttb, and tttb, where one must consider the b-jet identification efficiency for the last four processes. With the cuts on jets, p T ≥ 30 GeV and |η| < 2.7, we calculate the cross sections of the SM backgrounds and the ratio of signal to backgrounds at 2σ level. In Fig. 6, we depict the contour plots of g ut vs m Z for signal significance S/ √ B = 2 for the integrated luminosity L dt = 137 fb −1 at √ s = 13 TeV. The red, blue, and purple lines correspond to the cases of g tt = 0.5, g tt = 0.1, and g ut = g tt , respectively.
Here, we compare the bound on g ut from the triple top quark production with the one from the same-sign top quark pair production. The bound on g ut from the triple top quark production depends on the value of g tt as we already mentioned. For g tt = 0.5 and m Z = 2 TeV, the bound on g ut can reach 0.055 with the integrated luminosity L dt = 137 fb −1 in the triple top quark production, while for g tt = 0.1 and m Z = 2  Figure 6: The contour plots of g ut vs m Z for signal significance S/ √ B = 2 for L dt = 137 fb −1 at √ s = 13 TeV. The red, blue, and purple lines correspond to the cases of g tt = 0.5, g tt = 0.1, and g tt = g ut , respectively.
TeV, the bound on g ut reaches 0.27. To obtain bounds comparable to those in the samesign top quark pair production, g tt should be about 0.68. Therefore, we conclude that the same-sign top quark pair production tends to yield more stringent bounds on the top FCNC coupling than the triple top quark production for g tt 0.68. However, for g tt 0.68, the triple top quark production might provide more stringent bounds on g ut . We note that this comparison is based on the numerical analysis at the parton level, where the t ort quarks are completely reconstructed from the data. In the analysis at detector level, it would be difficult to reconstruct all the top quarks, in particular, in the multitop production because of the missing energy for the semileptonically decaying top quarks and combinatoric problems of jets of hadronically decaying top quarks. We will perform the numerical analysis at detector level in Sec. 4, and it will turn out that the bounds on the couplings in the triple top quark production could be much larger than the results in this subsection.

Dijet production
The dijet production is one of the best channels to probe an s-channel resonance like, in particular, a leptophobic Z boson. In this work, the Z boson is taken into account as a leptophobic Z boson by assuming that mixing between the SM neutral gauge bosons and the Z boson is negligible. The dijet production through the Z resonance proceeds in the parton process uū → Z → uū, whose amplitude is proportional to g 2 uu . Then, the cross section for the dijet production through the Z resonance for m Z = 2 TeV at √ s = 13 TeV is calculated as σ(pp → Z → jj) = 0.92 g 4 uu pb, where the K factor is taken to be 1.3 [27] and the cuts on jets are implemented. In Fig. 7, we depict the cross sections for the dijet production at √ s = 13 TeV. The red, blue, and green lines correspond to g uu = 0.5, 0.6, and 0.7, respectively. The black line is the observed 95 % CL upper limits with the integrated luminosity, L = 36 fb −1 , at CMS [27]. From the CMS data, the region g uu 0.5 is allowed for m Z ≥ 1 TeV. For m Z 3.3 TeV, g uu 0.7 is allowed. With more accumulated data, the upper bounds could be much improved. However, it might not be capable of probing the Z boson with the small coupling g uu 1 even with more data. Then, the search for the Z boson through the top FCNC processes, for example, the same top quark pair production, may be a more probable channel to probe the Z boson in such a case.

Top quark pair production
In this subsection, we consider the top quark pair production pp → tt. In principle, one can consider the top quark pair production via a Z resonance like the dijet production in the previous section. So far at the c.m. energy √ s = 13 TeV, there are no results for a resonance search in the top quark pair production except for the heavy Higgs boson search ranging from 400 to 750 GeV in CMS experiments [28]. Thus, in this subsection, we concentrate on the effects of the Z boson on the total cross section for the top quark pair production.
The parton process relevant to the top quark pair production is uū → tt, where the Z boson can be exchanged in an s-channel as well as in a t-channel. Then, three couplings g uu , g ut , and g tt are involved in the top quark pair production. The s channel exchange amplitude is proportional to g uu g tt , while the t channel one is to g 2 ut . In Fig. 8, we depict the total cross sections for the top quark pair production (in pb) as functions of m Z in TeV (left) and g tt (right) for various values of g uu at √ s = 13 TeV. Here, the K factor is taken to be 1.6. g ut is determined by g ut = √ g uu g tt , but we note that this relation is valid for the mixing between two right-handed quarks u R and t R . If one considers the mixing among three right-handed up-type quarks, g ut could be set to be a free parameter. It should be noted that the Z boson does not contribute to the top quark pair production for g uu ∼ g ut ∼ 0.
In Fig. 8, the horizontal black line is the SM prediction for the top quark pair production. In experiments, statistical uncertainties in the top quark pair production are well controlled and much smaller than the other ones, systematic and luminosity uncertainties [29]. It seems that the sum of the systematic and luminosity uncertainties in quadrature is about 5 %, regardless of the decay channels of the top quark [29], which is at least ten times larger than statistical uncertainties in the lepton+jet channel of the top decay. Thus we do not perform the analysis for the SM backgrounds in the top quark pair production. Instead, we consider 5 % deviation from the SM prediction, which is denoted by the gray line in Fig. 8. In the left panel of Fig. 8, we plot the total cross sections for several values of g ij . For g uu = g tt = 0.5, the total cross section cannot reach the 5 % uncertainty line for m Z 500 GeV. In the right panel of Fig. 8, the total cross sections are depicted as a function of g tt for m Z = 700 GeV (red) and 1 TeV (blue), respectively. For m Z = 700 GeV and g uu = 0.5, the cross section can reach the gray line for g tt ∼ 0.9. Therefore, we conclude that the light Z boson and large g tt couplings are required for the Z boson to be probed in the top quark pair production. It is worthwhile to mention that if g uu is negligible it might be difficult to probe the Z boson in the top quark pair production.

Four top quark production
The last process which we consider in this work is the four top quark production at the LHC. The Z boson can affect the four top quark production through a Z exchange between top quark pairs or between the incident uū pair and produced tt pair. Then, all three couplings, g uu , g ut , and g tt , are involved in the four top quark production. Since the amplitudes may involve one or two Z propagators, the total cross section can be expressed  Figure 9: The cross sections for the four top quark production (in pb) as functions of m Z in TeV (left) and g tt (right) at √ s = 13 TeV. The yellow band is the measured value at CMS within 95 % CL [31], and the black line is the SM prediction.
in terms of g k ij (k = 0, 2, 4, 6,8) if one also considers the SM interactions. One of the merits of the four top quark production is that, unlike the top quark pair production, the Z boson can contribute to the four top quark production even for g uu = g ut = 0. This can occur by the tt pair production, followed by radiation of a Z boson from t ort and its subsequent decay into another tt pair. Therefore, one may expect the Z signal even though one cannot observe it in the top quark pair production and same-sign top quark pair production, in which g uu and/or g ut are involved. On the other hand, the production mechanism of this process is similar to the radiative Z production discussed in Sec. 3.3. Thus, one could not expect a large deviation from the SM prediction for large m Z .
In Fig. 9, we depict the total cross sections for the four top quark production in pb as functions of m Z in units of TeV (left) and g tt (right) at √ s = 13 TeV. The black solid line is the SM prediction. Here, we set the K factor to be 1.3 [30]. The yellow band is the measured value at CMS with the integrated luminosity 137 fb −1 within 95% CL: σ = 12.6 +5.8 −5.2 fb [31], which is consistent with the SM prediction 12.0 +2.2 −2.5 fb [30]. The total cross sections are depicted for several combinations of g uu and g tt , and in the first four cases, g uu ≤ 0.5 are chosen by considering the bound on g uu from the dijet production. g ut is determined by g ut = √ g uu g tt . In the case of mixing of three right-handed up-type quarks, g ut could be chosen freely, as we have argued in Sec. 2.
As shown in Fig. 9, the cross section could be enhanced for a small Z mass, for example, in the region of m Z 1 TeV. From the right panel of Fig. 9, we also require a large g tt coupling to get a large deviation from the SM prediction. As an example, g tt 0.7 is required for m Z = 0.7 TeV and g uu = 0.5. For a small m Z , the four top quark production may probe the Z boson. However, for the Z mass larger than about 1.2 TeV, this process would be difficult to probe the Z boson, even though more data are accumulated at the LHC.

Numerical analysis at detector level
In this section, we perform numerical analyses at detector level for the same-sign top qurk pair production and triple top quark production, which might be the best candidate for the proof of the top FCNC coupling g ut . In the former case, the result of the detector simulation is not very different from that of the parton-level analysis because the SM backgrounds are very small in comparison with generated signals. However, for the triple top quark production, we find that the bound for the coupling obtained in the previous section is much smaller than that from the detector simulation in this section. The main difference between two analyses comes from the estimation of the SM backgrounds, which we will discuss later.
In the same-sign top quark pair production, we consider the process pp → tt(tt) as well as pp → ttū(ttu), where the latter occurs from a ug(ūg) collision. For leptonical decays, t → bl + ν l andt →bl −ν l (l = e, µ), the final signals contain two same-sign leptons and two b jets with missing transverse energy. We produce the signal events by making use of MADGRAPH [14,15], Pythia [32], and Delphes [33] for parton-level event generation, parton shower, hadronization, and detector simulation at the 13 TeV LHC. We follow the analysis methods for the search for new physics signals with the signature of a same-sign lepton pair and b jet by the ATLAS group [24] and require following conditions: (i) exactly one same-sign lepton pair with each lepton p T larger than 28 GeV, (ii) at least one b-tagged jet with the b-jet p T larger than 25 GeV, (iii) missing transverse energy ( / E T ) lager than 40 GeV, (iv) H T larger than 750 GeV, (v) the azimuthal angle separation between two leptons larger than 2.5.
The H T is the scalar sum of the transverse momentum of all jets in an event. We find that about 0.3% of the signal events passes the cuts listed above and the ATLAS Collaboration shows that 23.9 events of the SM background survive after the selection for L dt = 36.1 fb −1 . We simply rescale the expected number of SM background events n b = 23.9 × ( L dt/36.1) for other integrated luminosities L dt [16].
In Fig. 10, we depict the contour plots of g ut vs m Z (in TeV) for signal significance S/ √ B = 2 for L dt = 137 fb −1 (red) and L dt = 3000 fb −1 (blue) at √ s = 13 TeV, respectively. We find that the upper bound on the FCNC coupling g ut could reach about 0.6(1.3) for m Z = 2(5) TeV and L dt = 137 fb −1 , respectively. For L dt = 3000 fb −1 , the upper bound could be reduced by a factor of about 0.6.
In the triple top quark production, it is difficult to reconstruct all top quarks from the decay products. Therefore, in experiments, signal events are chosen by taking into account the decay channels, where tt (tt) decay semileptonically and the remainingt (t) decays hadronically [34]. Then, the signals contain a pair of same-sign leptons and several b or light quark jets. We simulate the signal events by taking into account the top quark decays, ttt → bl + ν l bl + ν lb jj and ttt →bl −ν lb l −ν l bjj (l = e, µ), at the 13 TeV LHC. We follow the ATLAS Collaboration for the optimal signal region (Rpc2L1bH) [34] and require following conditions: (i) at least one same-sign lepton pair with each lepton p T larger than 20 GeV, (ii) at least six jets with the jet p T larger than 20 GeV, (iii) at least one b-tagged jet with the b-jet p T larger than 20 GeV, (iv) / E T lager than 250 GeV, We find that about 0.3% of the signal events survives after requiring cuts listed above and the ATLAS Collaboration shows that 9.8 events of the SM background remained after the selection for L dt = 36.1 fb −1 . For other integrated luminosities L dt, we simply rescale the expected number of SM background events n b = 9.8 × ( L dt/36.1).
In Fig. 11, we depict the contour plots of g ut vs m Z (in TeV) for signal significance S/ √ B = 2 at production level for L dt = 137 fb −1 (left) and L dt = 3000 fb −1 (right) at √ s = 13 TeV, respectively. The red, blue, and purple lines correspond to the cases of g tt = 0.5, g tt = 0.1, and g ut = g tt , respectively. As shown in Fig. 11, the upper bounds on g ut are greater than O(1) over the whole range of m Z ≥ 1 TeV. For m Z = 2 TeV and g tt = 0.5, the bound could reach 5.92 (2.74) for the integrated luminosity L dt = 137 (3000) fb −1 . The obtained bounds on couplings at the detector level are much larger than those at the parton level in the triple top quark production. The difference stems from the estimation of SM backgrounds. Since the top quarks cannot be fully reconstructed in the multitop production, the signal events are selected by a pair of same-sign leptons with at least one b-tagged jet and a few jets in the triple top quark production [34]. Then, the  Figure 11: The contour plots of g ut vs m Z (in TeV) for signal significance S/ √ B = 2 at the production level for L dt = 137 fb −1 (left) and L dt = 3000 fb −1 (right) at √ s = 13 TeV. The red, blue, and purple lines correspond to the cases of g tt = 0.5, g tt = 0.1, and g ut = g tt , respectively. main SM backgrounds are the tt + W (Z, γ, H) channels, which have much larger cross sections than the rare multitop quark production.
Comparing the result for the upper bound on g ut at the production level, we find that the same-sign top quark pair production could provide a much more stringent bound than the triple top quark production over the whole range of m Z ≥ 1 TeV. This conclusion is apparently distinguished from the expectation at the parton-level analysis. The discrepancy is originated mainly from the estimation of the SM backgrounds. In the same-sign top quark pair production, the SM backgrounds are not so different for both parton-and detector-level analyses. However, in the triple top quark production, the SM backgrounds at the detector level are much larger.

Summary
In this work, we have considered a Z model in which the Z boson couples only to u and t quarks. The model could be constructed with an extra U (1) symmetry, under which only right-handed up-type quarks are charged. The Z boson can be leptophobic if the mixing between the Z boson and SM gauge bosons is negligible. After symmetry breaking, the right-handed up-type quarks mix with each other, and the top FCNCs mediated by the Z boson can be generated at tree level.
The Z boson can be searched for through FCNC processes. In this work, we have considered several top FCNC processes at the LHC: the same-sign top quark pair production, single top quark production, radiative Z production, and triple top quark production. The first process depends on the coupling g ut , while g uu and/or g tt as well as g ut are involved in the other processes. Therefore, such processes cannot be available for the Z search if g uu and/or g tt are negligible. We find that among the top FCNC processes the same-sign top quark pair production could be the most capable of yielding the best bound on the Z boson and top FCNCs induced by the Z boson for g tt ≤ 0.68 if only statistical uncertainties at the parton level are taken into account. However, for g tt ≥ 0.68, the triple top quark production might provide stronger bounds on the top FCNC coupling than the same-sign top quark pair production.
On the other hand, the Z boson can be searched for through non-FCNC processes: the dijet production, top quark pair production, and four top quark production. Only g uu is involved in the dijet production, whereas g ut and g tt are also involved in the other two processes. In the dijet production, we find that the region g uu 0.5 is allowed for m Z ≥ 1 TeV and the bound could be enlarged to 0.7 for m Z 3.3 TeV. In the top quark pair production and four top quark production, small Z mass and large g tt coupling are required to probe the Z boson. We note that in the case of g uu 1 it may be impossible to probe the Z boson in the dijet or top quark pair production. In this case, the same-sign top quark pair production or triple top quark production might be the best probe to the Z boson for m Z ≥ 1 TeV.
Finally, we performed numerical analyses at detector level for the same-sign top quark pair production and triple top quark production by taking into account the results at the parton level. We find that, unlike the analysis at the parton level, the same-sign top quark pair production provides much more stringent bound on the FCNC coupling g ut . The bound could reach about 0.6(1.3) for m Z = 2(5) TeV and L dt = 137 fb −1 . To get complementary bound on g ut in the triple top quark production, it would be necessary to reconstruct multitop quarks and remove the main SM backgrounds in current experiments.