Search for the vectorlike leptons in the U(1)$_X$ model inspired by the $B$-meson decay anomalies

We consider the U(1)$_X$ model, which can induce the flavor violating couplings through vectorlike fermions and address the observed rare $B$-meson decay anomalies. To be consistent with all the other observations, both the associated gauge boson mass and the vectorlike lepton mass are bounded from above. We argue that the search for new vectorlike leptons is promising and provides a complement to the $Z'$ search. A detailed collider analysis shows that the model with the vectorlike lepton mass up to 1000 GeV could be tested at the future LHC.

A heavy Z boson with flavor-changing couplings to quarks and non-universal couplings to leptons [10] is an obvious candidate contributing to b → s anomalies. In the literature, the Z boson is either associated with a horizontal gauge symmetry [11][12][13][14], embedded in the 3-3-1 model [15,16], or has generic couplings to quarks and leptons [17][18][19]. Some models also employ vector-like particles to generate required couplings [11,12,20,21]. In this paper, the approach that introduces both vector-like quarks and vector-like leptons [20] is adopted. Through mixings of the U(1) X charged vector-like particles with their SM counterparts, LFU is violated in a manner similar to the SM with flavor-dependent Yukawa couplings.
There have been a number of works on studying the collider phenomenology of possible new physics (NP) that could address the B anomalies, many of which [22][23][24][25][26] focus on the signature of Z decaying into dimuon. In our setup, if the relevant gauge and Yukawa couplings are assumed to be less than unity, then several experiments imply an upper bound on the Z mass and a tighter constraint on the vector-like lepton mass. The prospect of detecting these two particles at a future collider will be discussed in detail. Searches for them are complementary to each other in some regions of parameter space. Their combined sensitivity at the high luminosity LHC will be able to cover a broader range of parameter space that can explain the B anomalies. This paper is organized as follows. In Sec. II, the model is presented. In Sec. III, various constraints imposed on the parameters by several measurements are examined. Sec. IV is devoted to LHC phenomenology of the model and the strategy to test it. In Sec. V we summarize our work.

II. THE MODEL
To accommodate the Z boson and vector-like particles, the SM is extended to incorporate a U(1) X gauge group. Under SU(3) C × SU(2) L × U(1) Y × U(1) X , the newly introduced fields are in the following representations [20], where Q = (U, D) and L = (N, E) are vector-like fermions and φ is a scalar which will develop a vacuum expectation value (VEV) φ = 1 √ 2 v φ that breaks U(1) X and gives mass to the Z boson, m Z = g X v φ . These vector-like fermions are charged under the SM gauge group in the same way as their left-handed SM counterparts. Their mixings with righthanded SM parts are highly suppressed except for the top quark. As a consequence, the gauge interactions of SM fermions are almost intact and electroweak precision measurements put few constraints on the mixing angles. We can write down Dirac mass terms for the vector-like fermions and their Yukawa couplings to SM counterparts, where i and q i represent the SM left-handed lepton and quark doublets respectively, λ 1,2,3 L and λ 1,2,3 Q will be denoted by λ e,µ,τ L and λ d,s,b Q below. Since the terms involving λ µ L and λ b,s Q are sufficient to provide the couplings that lead to b → s anomalies, λ e,τ L and λ d Q are set to be zero. This eliminates all lepton flavor violating (LFV) processes mediated by the Z boson and simplifies the model substantially yet still gives rise to rich phenomenology.
After the new scalar field acquires a VEV, the mass matrices for charged leptons and down-type quarks can be written as where λ µ,s,b are SM Yukawa couplings and v is the Higgs VEV. The up-type quark mass matrix is similar to M D . In the limit of m Q λ µ,s,b v, the masses for the new lepton and quark are The effective couplings of the Z boson to SM fermions is induced by mixings between vector-like and SM fermions, with The remaining nonzero couplings g νµνµ , g cc , g tt and g tc (g ct ) are not independent: g νµνµ = g µµ , g cc = g ss with m c being neglected, and g tt = g bb , g tc = g bs provided that m Q m t . Note that g bb , g ss and g bs are correlated, there is always g bb g ss = |g bs | 2 .

III. LOW ENERGY CONSTRAINTS
The measurements of R K ( * ) supplemented by a few other low energy measurements mentioned below imply a correlation between the mass of Z and its couplings to the SM fermions.
The effective Hamiltonian describing b → s transitions is conventionally written as [27] with the following four-fermion interactions: The primed Wilson coefficients do not receive significant SM contributions while the unprimed can be split into two parts, the SM and the NP ones: Many groups have performed global fits to the data on b → s transitions [27][28][29][30][31]. One of the favored scenarios can be exactly implemented in our model, where new particles only couple to left-handed quarks and left-handed muons, i.e., C µ,NP while all other coefficients of NP remain zero. To be specific, the effective Hamiltonian for anomalous b → s transitions is The best fit point that takes into account only LFU observables instead of all available data requires C µ,NP 9 = −C µ,NP 10 = −0.63 [27], which translates into There are two relevant constraints in the parameter space around the best fit point, one from B s −B s mixing [32] and the other from the neutrino trident production [33]. The former puts a bound on m Z over g bs while the latter pertains to m Z over g µµ : Combining these constraints with the relations in Eq. (10) will give Moreover, it is reasonable to assume that the NP couplings |λ µ L | and g X are less than unity, which then indicates that the masses of the Z boson and the vector-like lepton are bounded from above 1 m Z 3.9 TeV , m E 2.7 TeV .
Note that after mixing with the up-type vector-like quark U , the top quark mass will be slightly smaller than 1 √ 2 λ t v h , but will still lie within the uncertainty of current measurements.
D g X and g µµ < g X into Eq. (16), we obtain another bound: m D 22 TeV.

IV. LHC PHENOMENOLOGY
The aforementioned constraints can be used to restrict σ(pp → Z ) × BR(Z → µ + µ − ) which is the true observable concerning a direct collider search. Larger g f i f j with f i = c, s, t, b typically mean larger σ(pp → Z ) but lower BR(Z → µ + µ − ) and vice versa. Each of the processes cc → Z , ss → Z , bb → Z and bs → Z contributes a certain fraction of σ(pp → Z ), and σ×BR reaches the lower bound approximately when g f i f j are adjusted accordingly to minimize σ(pp → Z ) while BR(Z → µ + µ − ) 50%, because couplings of Z to quarks are negligible compared to g µµ at this point. The upper bound does not necessarily matter because that region of parameter space has already been ruled out by current searches. The boundaries of parameter space consistent with the constraints discussed in Sec. III are presented in the σ×BR vs m Z plane in Fig. 1. The search for Z in the dimuon final state has been performed by both the ATLAS [34] and CMS [35] collaborations at the LHC. The expected exclusion limits by CMS using 36 fb −1 of data collected at 13 TeV is shown by the dashed black curve in Fig. 1. The region above the curve covering half of the viable parameter space is excluded. Further constraints on g f i f j can be derived from the exclusion limits as follows. Combining Eqs. (16) and (17), we obtain both the upper and lower bounds on g µµ and g bs . With g µµ saturating the lower bounds, larger g bb or g ss would cause σ×BR to exceed the exclusion limits, hence the upper bounds on g bb and g ss . Considering that g bb , g ss and g bs are correlated, i.e., g bb g ss = g 2 bs , the lower bounds on g bb and g ss are straightforward. The results are collected in Fig. 2. These couplings are directly related to the branching ratio for Z → µ + µ − 2 whose upper and lower bounds are plotted in Fig. 2 as well. Because g µµ g bs and g bb g ss = g 2 bs , the 50% upper bound is trivial and will be reached when g ss,bb,bs are of the same order. Z will decay dominantly to leptons if g 2 µµ 3 max (g 2 bb , g 2 ss ). In fact, the leptonic branching ratio is at least 90% for m Z under 1.5 TeV.

FIG. 2:
The bounds on the couplings of Z to the SM fermions (left y-axis) and the branching ratio for Z → µ + µ − (right y-axis), taking into account the exclusion limits on Z in the dimuon channel at the LHC. Magenta, orange, green, blue, and cyan correspond to g µµ , g bb , g ss , g bs , and BR(Z → µ + µ − ) respectively. The solid and dashed lines represent upper and lower bounds respectively. It should be noted that g bb , g ss , and g bs are not independent, in other words, they can not reach the upper or lower bounds at the same time.
To show the prospect for a discovery in the dimuon channel, we extrapolate the exclusion limits to the 14 TeV LHC with an integrated luminosity of 3000 fb −1 using a method dedicated to resonance searches [26,36]. The sensitivity to σ×BR can be improved by one order of magnitude, but there is still plenty of parameter space left, even a very light Z with m Z ∼ O(100) GeV may possibly escape the resonance search. So it demands other strategies to test this model, and we will show that the search for the new charged lepton E ± provides a more effective probe under certain conditions. Searches for vector-like leptons have been studied within various theoretical frameworks [37][38][39]. In our model, the interactions between E ± and SM gauge bosons are given by where λ µ is the muon Yukawa coupling and v the SM Higgs VEV. As a consequence, the E ± → µ ± Z (∝ λ µ ) and E ± → νW ± (∝ λ 2 µ ) decay channels are highly suppressed. Moreover, the E ± → N W ± decay channel is kinematically impossible as the mass difference between E and N is of order m µ , There are only two major decay channels left, 3 E ± → µ ± H and E ± → µ ± Z ( * ) with either an on-shell or an off-shell Z , which subsequently decays into a pair of muons at least 45% of the time for m Z < 1.5 TeV as governed by Eq. (20). Through the E ± → µ ± Z ( * ) channel, 6-muon final states can be produced at the collider. In contrast to the production of a single Z , which are bounded by very small NP couplings, g ss , g bs , and g bb , the E ± pair production is practically governed by SM gauge couplings the same as those of e/µ to γ and Z. More importantly, the 6-muon signature is almost free from the SM background at hadron colliders. The region of parameter space that predicts O(10) events can be probed with a high significance. The total number of 6-muon events at a hadron collider is determined by m E , m Z , g f i f j (mostly g µµ since the others are negligible in the considered region), and the mixing angle between the Higgs and φ, θ H−φ which will diminish BR(E ± → µ ± Z ( * ) ). On top of the boundaries depicted in fig 2, g µµ is subject to the bound: The upper (solid contours) and lower (dashed contours) limits on the number of 6-muon events at the LHC at a centre-of-mass energy of 14 TeV with an integrated luminosity of 3000 fb −1 are plotted in the top panels of Fig. 3, where MG5 aMC@NLO 2.6.0 [40] has been used to evaluate the production cross section of the E ± pair. Needless to say, both the upper and lower limits are highly dependent on m E . One remarkable feature of these plots is that the number of events hardly changes with m Z as long as Z is on-shell but drops abruptly when Z goes off-shell, especially the lower limits. The charged vector-like lepton as heavy as 1000 − 1400 GeV can be probed in the parameter space where m Z < m E .
The upper and lower limits of BR(E ± → µ ± H) are plotted in the bottom panels of Fig. 3. The two limits almost coincide with each other in the region where m Z < m E and E ± → µ ± Z is dominant. The branching ratio increases with increasing m Z /m E and sin θ H−φ . The current Higgs precision measurements still allow |sin θ H−φ | 0.3 [41]. With a sizeable scalar mixing (sin θ H−φ > 0.1) and an off-shell Z , i.e., m Z /m E > 1, the E ± → µ ± H decay mode will dominate over the E ± → µ ± Z * mode. Even with sin θ H−φ = 0.05 and m Z /m E = 1, BR(E ± → µ ± H) could still be as large as 50%. In the following subsections, we will study its collider phenomenology in detail. The signal to be considered is illustrated in Fig. 4. The contribution from s-channel Z exchange is ignored which is justified by the fact that only the second and third generation quarks couple to Z and their couplings are negligible compared with those to γ and Z except for a very heavy Z . 3 Due to the extra scalar boson φ in our model, there will be another decay mode E ± → µ ± φ if kinematically allowed. The reason why we do not consider it in this work is that the mass of φ is a free parameter in the model, a light scalar field φ that mixes with the SM Higgs is stringently constrained, it can only give rise to signatures similar to those of the E ± → µ ± H channel. Owing to the fact that all g f i f j except g µµ are negligible in the displayed region, basically the number of 6-muon events reaches its upper limits while BR(E ± → µ ± H) reaches its lower limits and vice versa.

A. Simulated samples and object reconstruction
Our signal and background events are generated with MG5 aMC@NLO 2.6.0 [40], in which MadSpin [42] is used for the decays of the vector-like leptons and SM Higgs boson, and Pythia8 [43] is used to implement parton shower, hadronization and decay of hadrons. The detector effects are simulated by Delphes 3.4.0 [44] with ATLAS configuration card, where the b-tagging efficiency has been set to 70% [45], and mis-tagging rates for the charmand light-flavor jets are 0.15 and 0.008, respectively. The jet reconstruction is handled by FastJet 3.2.1 [46]. The signal benchmark points are chosen as m E ∈ [150, 1500] GeV with step size of 25 GeV. The dominant SM background processes for this signal are tt, ttbb, ttH and ttZ. Their estimated production cross sections at 14 TeV proton-proton collider [47][48][49][50] are given in Tab. I.
In our analysis, the Higgs boson are reconstructed with two different methods, and each is suitable for a certain phase space. In the first method, all jets in the final state are  reconstructed with anti-k t algorithm [51] with radius parameter R = 0.4. Among them, we require at least three b-tagged jets for Higgs reconstruction. The combination of the three b-tagged jets with the fourth jet that minimizes the mass asymmetry defines two Higgs. We will denote them by normal Higgs (NOR Higgs) in the following. The second method is devoted to tagging more energetic Higgs which forms a single jet in the detector. In this case, the jets in the final state are reconstructed by Cambridge-Aachen (CA) algorithm [52] with cone size parameter R = 1.4. The CA jets that fulfill the massdrop tagger [53] as well as contain at least one b-tagged subjet are identified as Higgs jets. They will be denoted by substructure Higgs (SUB Higgs). Each of the reconstructed Higgs is then combined with one of the two muons in the final state to form a vector-like lepton. Same as above, the combination that minimizes the asymmetry is chosen. Due to the relatively low efficiency of reconstructing the vector-like leptons, especially when they are light, we find the stransverse mass of the dimuon system [54] m T 2 (µ 1 , µ 2 ) ≡ min outperforms the invariant mass of the reconstructed vector-like lepton in signal and background discrimination. Here the transverse mass m 2 2 with m H =125 GeV and index j in Eq. (26) runs over all Higgs constituents. 4 The distributions of the leading Higgs invariant mass, the dimuon stranverse mass and the transverse momenta of the leading Higgs and leading vector-like leptons in the SUB Higgs reconstruction method are presented in Fig. 5 for illustration. We can see that the Higgs boson of the signal with relatively heavy E ± can be effectively reconstructed by the SUB method. The invariant masses of fake Higgs jets in the background processes are typically below the true Higgs boson mass. The m T 2 (µ 1 , µ 2 ) variable is always larger than m H . In background processes, the hardest constituents are given by the top quark mass, so the distribution of m T 2 (µ 1 , µ 2 ) is cut off at m H + m t . While in signal processes, the upper bounds on m T 2 (µ 1 , µ 2 ) are given by the masses of the vector-like leptons, which can be much higher than the top quark mass. These features make m T 2 (µ 1 , µ 2 ) very efficient in signal and background discrimination. Moreover, for a pair of relatively heavy vector-like leptons, the energy scale of the signal process is much higher than those of background processes, leading to a harder spectrum in the distributions of transverse momenta of Higgs and vector-like leptons.

B. Event selection and signal significance
The pair production of vector-like leptons in the model is dominantly given by the schannel Z/γ exchanges through electroweak interaction. The cross section is below ∼ O(1) fb for m E 1 TeV at the 14 TeV LHC. A high integrated luminosity would be required to probe the vector-like leptons with mass around O(1) TeV scale.
Our event selections proceed as follows. The preselection requires at least four jets and two opposite sign (OS) muons in the final state. Here, the jets are reconstructed by anti-k T algorithm with radius parameter R = 0.4, p T > 20 GeV and |η| < 2.5. Three signal regions are defined for selecting the OS dimuon, as given in Tab. II. Each is suitable for some vector-like lepton masses. The muons should be within the pseudorapidity region |η| < 2.5.  Then, we apply a cut on the dimuon stransverse mass, m T 2 (µ 1 , µ 2 ) > 300 GeV. Furthermore, we require that there are exactly two reconstructed Higgs (either NOR Higgs or SUB Higgs), both of which satisfy 90 GeV ≤ M H ≤ 130 GeV. The two Higgs should contain at least three b-tagged subjet in total. 5 In addition, two signal regions are defined for Higgs p T , and they are denoted by SRij, where i = 1 − 3 stands for signal regions for selecting the OS dimuon, and j = 1, 2 stands for signal regions for selecting the Higgs transverse momentum.   Finally, the SRij are further divided according to the reconstructed Higgs type and vector-like lepton mass, as given in Tab. IV. This gives a total of 24 signal regions in our analysis, i.e., SRijk, i = 1 − 3, j = 1, 2, k = 1 − 4.
Once we obtain the numbers of signal (s) and background (b) events in each signal region, the signal significance of that signal region can be calculated by [55]   IV: Four signal regions based on the reconstructed Higgs type and vector-like lepton mass. The cut is applied on the heavier one of the two reconstructed vector-like leptons. The index i runs over 1, 2, 3, corresponding to Tab. II, and the index j runs over 1, 2, corresponding to Tab. III.
For the signal process with given m E , the signal region that provides the highest signal significance is chosen. In Tab. V, we show the cut flow in the chosen signal regions for three benchmark points. The most sensitive signal regions for m E = 500 GeV, 700 GeV and 1000 GeV are SR212, SR322, and SR323, respectively.  The cut flow of our analysis for signals (with three representative vector-like lepton masses 500 GeV, 750 GeV, and 1000 GeV) and background. The numbers correspond to the production cross sections (in fb) after cuts at the 14 TeV LHC. We have assumed BR(E → µH) = 100%.
In Fig. 6, we present the highest signal significance among signal regions with varying vector-like lepton mass m E ∈ [150, 1500] GeV and different branching ratios BR(E → µH) ∈ [60, 100]%. It can be seen that the vector-like lepton mass below ∼ [800, 1000] GeV can be probed at 2-σ level at the 14 TeV LHC with an integrated luminosity of 3000 fb −1 .

V. SUMMARY
The observed rare B-meson decay anomalies might indicate the existence of a Z boson which has flavor-changing couplings to quarks and non-universal couplings to leptons. We considered a U(1) X extension of the SM gauge group such that the desired types of couplings can be naturally generated by introducing extra vector-like fermions. Taking into account the constraints from B s −B s mixing and the neutrino trident production, assuming the NP gauge and Yukawa couplings to be less than unity, the observed B-meson anomalies require the masses of new particles in the model to be bounded from above: m Z 3.9 TeV, m N,E 2.7 TeV, and m U,D 22 TeV.
The search for Z in the dimuon final state at the LHC by CMS covered only part of the parameter space. Nonetheless, the couplings of Z to muon/muon neutrino, the second and third generation quarks are further constrained by the exclusion limits on Z . In terms of branching ratio, for m Z < 1.5 TeV, the Z boson decays into a muon pair or a muon neutrino pair at least 90% of the time. By extrapolating current exclusion limits to the 14 TeV LHC with an integrated luminosity of 3000 fb −1 , we found it impractical for the direct search to provide enough sensitivity to a Z boson with mass ∼ O(100) GeV.
On the other hand, the search for the relatively light vector-like lepton is complementary to that for the Z boson, because its production at the LHC is almost entirely controlled by the SM gauge couplings. In the parameter space where m E > m Z , the vector-like lepton decays dominantly into a muon and a Z boson which subsequently decays into two muons with a certain branching ratio. This will give as much as 6 muons in the final state. The 6-muon signature is essentially background free, so that a number of events over O(10) would allow high confidence level signal/exclusion. Our study showed that the 6-muon signature can probe vector-like lepton mass up to 1000 − 1400 GeV at the future LHC with an integrated luminosity of 3000fb −1 . If the vector-like lepton mass is less than the Z mass, the E → µH channel will become competitive with or even dominant over the E → µZ * (→ µµ) channel, especially when the mixing between the SM Higgs and the new scalar field φ is sizeable. We performed a detailed search for the signature of dimuon plus two boosted Higgs boson from vector-like lepton pair production. The future LHC is sensitive to the vector-like lepton with mass below ∼ [800, 1000] GeV for BR(E → µH) ∈ [60, 100]%.