Mutually elusive: vectorlike antileptons and leptoquarks

We study the properties of vectorlike fermions that have the same gauge charges as the Standard Model lepton doublets, but opposite lepton number. These antileptons undergo decays mediated by heavier scalar leptoquarks, while the symmetries of this renormalizable model protect the vectorlike fermions and the leptoquarks from standard decays probed so far at colliders. If the new particles couple predominantly to second-generation quarks, then their collider probes involve multiple jets and two taus or neutrinos, and are hampered by large backgrounds. If couplings to third-generation quarks are large, then the collider signals involve top quarks, and can be probed more efficiently at the LHC. Even in that case, both the vectorlike fermion doublet and the leptoquarks remain more elusive than in models with standard decays.


I. INTRODUCTION
Chirality is an essential characteristic of all Standard Model (SM) fermions.The left-and right-handed components of the SM fermions transform differently under the electroweak gauge group, and thus the quarks and leptons acquire masses by coupling to the Higgs field.Vectorlike fermions are distinct from chiral fermions in that the transformation properties under the SM gauge group are identical for both chiralities.A Dirac mass for a vectorlike fermion does not violate any SM symmetry.Therefore, the size of their masses is not tied to the Higgs doublet.If elementary vectorlike fermions exist, they represent a new form of matter, and dedicated searches for these are a major goal for the LHC experiments.
A vectorlike fermion that is a color singlet and carries lepton number L = +1 is referred to as a vectorlike lepton.It is typically assumed that the vectorlike leptons have the same charges under the SU (3) c ⊗ SU (2) W ⊗ U (1) Y gauge group as a SM lepton, namely (1, 2, −1/2) for a weak doublet, or (1, 1, −1) for a weak singlet.In that case, a Yukawa coupling to the SM Higgs doublet and a SM lepton field is allowed by the gauge symmetry, and induces a mass mixing between the vectorlike lepton and the SM lepton [1][2][3]).As a result the vectorlike lepton decays into a SM lepton and a W , Z, or Higgs boson.Searches for electroweak production of vectorlike leptons that decay under these hypotheses have been performed by the CMS collaboration [4,5], and a particularly stringent lower mass limit, of 1.05 TeV, has been set on the vectorlike lepton doublet.
In this article, we study the properties of a vectorlike fermion ψ that transforms as (1, 2, −1/2) under the SM gauge group, but which carries lepton number L = −1.
As this is the opposite of the lepton number carried by the SM leptons, we generically refer to ψ as a Vectorlike AntiLepton (VAL).We assume that ψ has a Yukawa interaction to a heavier leptoquark scalar (ξ) and a SM quark, and that all renormalizable interactions of ψ preserve lepton number.Consequently, the coupling of ψ to any SM lepton and the Higgs doublet is forbidden.Thus, the current searches for vectorlike leptons do not apply to ψ, as there are no standard decays (i.e., into a SM lepton and a SM boson).Instead, electroweak production of a ψ pair is followed, in our model, by 3-body decays mediated by the leptoquark.
The reverse is also true: instead of decaying into a SM lepton and a quark (for a review, see [6]), the leptoquark predominantly decays into a VAL and a SM quark, leading to a cascade decay.As a result, the VAL and the scalar leptoquark mutually elude present searches for vectorlike leptons or leptoquarks at the LHC, and motivate new dedicated searches.
Certain elements of the renormalizable model proposed here have been encountered in other contexts.Threebody decays of vectorlike quarks via a heavy scalar leptoquark have been analyzed in [7].Three-body decays of a vectorlike lepton through a leptoquark of spin-1 have been studied in [8][9][10][11], leading to τ b b final states searched for at CMS [12]; we will show that this is different than the final states obtained in the case of the VAL.Nonstandard decays of a vectorlike lepton, into a SM lepton and a light pseudoscalar, are discussed in [13] (see also [14]); in that model, though, the standard decays remain important.
On a more general note, vectorlike fermions appear in various frameworks beyond the SM, including bound states in models of quark and lepton compositeness [15], or higher multiplets of grand unified groups [16], so if they exist, vectorlike fermions would likely provide a window towards deeper principles.Thus, it is important to analyze a wide range of possible properties of vectorlike fermions, and to search for a variety of signals in experiments.The VALs studied here are as motivated as the vectorlike leptons that have mass mixing with the SM leptons, and thus are worth exploring in more detail.
In Section II we present the interactions of the VAL and of the leptoquark, analyze constraints from flavor processes, and compute the decay length of ψ.In Section III we study the LHC phenomenology of these particles, focusing on signals involving quarks of either second or third generation.We summarize our results in Section IV, where we also list the most important LHC searches that can test this model.

II. VECTORLIKE ANTILEPTON DOUBLET
Let us consider an extension of the SM that includes a weak-doublet vectorlike fermion ψ = (ψ 0 , ψ − ) ⊤ that carries lepton number L = −1 and transforms under the Since the SM lepton doublets ℓ j L (j = 1, 2, 3 is the generation index) have the same gauge charges as ψ, but opposite lepton number (by convention the electron has lepton number +1), it is appropriate to refer to ψ as an antilepton.A Dirac mass term for the ψ doublet (−m ψ ψ L ψ R + H.c.) is allowed both by the SM gauge symmetry (since the left-and right-handed components have the same charges) and by lepton number conservation.
We assume that the interactions of ψ preserve lepton number.Thus, any mixing between ψ and the SM leptons can be safely neglected, because it is forbidden by lepton number conservation.Note that the LEP searches for new charged particles imply m ψ ≳ 100 GeV [17,18].Even if the SM neutrinos acquire Majorana masses (which is not necessary, as the neutrinos may have Dirac masses), the lepton number breaking effects are suppressed by powers of m ν /m ψ ≲ 10 −12 .
Although both the left-and right-handed components of ψ have L = −1, one could use instead their chargeconjugate fields (ψ c L and ψ c R ) which have L = +1.Thus, ψ c may be viewed as a vectorlike lepton of wrong-sign hypercharge (compared to the SM lepton doublets), so the mixing between ψ c and the SM leptons is still forbidden.
In the absence of additional fields or of higherdimensional operators, the lightest particle of the ψ doublet remains stable.We introduce, however, a leptoquark scalar ξ that transforms as (3, 2, +1/6) under Gauge and Lorentz invariances allow the following Yukawa interaction of ξ with ψ and with the SM up-type quark singlets (labeled u j R in the gauge eigenstate basis): Here λ j ψ are three dimensionless couplings.If the leptoquark mass satisfies M ξ > M ψ , then the above Yukawa interactions still do not allow the decay of the lightest particle of the ψ doublet.This is due to a global U (1) ψ symmetry under which both ψ and ξ carry charge +1, while the SM fields are singlets.Renormalizable gauge-invariant terms that explicitly break U (1) ψ can be written as We assume that the above U (1) ψ breaking is suppressed, so the Yukawa couplings satisfy λ jτ ≪ λ j ψ .The Yukawa interaction (2) involves only the third-generation lepton doublet because we impose that τ -lepton number (L τ ) is conserved in this model, assigning L τ = −1 to both ψ and ξ.Focusing on the interaction with ℓ 3 L = (ν τ L , τ L ) is motivated in part by the larger τ Yukawa coupling to the Higgs doublet in the SM, which suggests that new physics might also have larger couplings to τ than to µ or e.A consequence of this coupling is that we will focus on collider signatures containing taus, which have larger backgrounds than those with final states containing muons or electrons.
Assigning baryon number B = 1/3 to ξ, and B = 0 to ψ, ensures that B is preserved by all the interactions introduced here.We point out that scalars that transform as (3, 2, +1/6) under the SM gauge group have been studied in various contexts, such as squarks (labeled q) in supersymmetric extensions of the SM, or leptoquark doublets (sometimes labeled R2 [6] or Φ 2 [19]).We use here the notation ξ to emphasize that the scalar participates in the peculiar interaction (1).
Besides the gauge interactions of ψ and ξ, and the Yukawa interactions discussed above, the model includes a single renormalizable interaction, namely λ H ξ † ξH † H.This Higgs portal coupling for the leptoquark changes both the Higgs production via gluon-fusion and the Higgs decay widths to γγ and γZ (see [6] for a review).However, the ensuing constraints on the λ H coupling remain relatively mild for now [20].

A. Constraints from flavor processes
In the mass eigenstate basis for the up-type quarks, the Yukawa interactions (1) take the form Here and in what follows it is implicit that all the interaction terms are accompanied by their Hermitian conjugates in the Lagrangian.Note that there is a linear relation between the Yukawa couplings introduced in (1) and (3) through the 3 × 3 matrix U R that relates the physical and gauge eigenstate bases for the right-handed up-type quarks.As U R is not determined by any existing measurements, we focus on the physical couplings λ t , λ c , λ u instead of λ j ψ .The terms involving the up and charm quarks give rise to FCNC processes involving D mesons.In particular, there are contributions to D 0 − D 0 meson mixing from box diagrams with ψ and ξ are running in the loop, such as the one shown in Figure 1.The ∆C = 2 operator generated by the box diagrams is given by where α, β are color indices, and f (m ψ /M ξ ) is a function with values roughly of order one.A discussion of the above operator, labeled Q 3 , can be found in [21,22].The comparison of the measured D 0 − D 0 mixing with the SM prediction (which suffers from large QCD uncertainties related to the charm quark) imposes that the coefficient of the (c L u R ) 2 operator has a real part smaller than 6 × 10 −8 TeV −2 and an imaginary part smaller by an extra factor of about 5 [23].Based on that constraint, we estimate 10 −3 M ξ 1 TeV . ( In what follows we assume |λ u | ≪ |λ c |, so we neglect any effects due to λ u .Thus, we do not need a more precise estimate of the above constraint.
Searches for rare top-quark decays can in principle constrain the product |λ t λ c |.For example, the t → cg and t → cγ decays would be induced by ψ and ξ running in a loop.This constraint is weak given the current sensitivity of LHC experiments, because the new t branching fractions are suppressed by both the loop factor and the large ξ mass.
The Yukawa interactions of ξ to the quarks of charge −1/3, shown in (2), can be written in the mass eigenstate basis for the down-type quarks as Any of the three pairs of down-type quark flavors in these interactions is constrained by flavor-changing neutral processes.For example, there are contributions to K 0 − K 0 meson mixing from box diagrams (one of them shown in Figure 2) with ξ = (ξ u , ξ d ) ⊤ and the SM lepton doublet ℓ 3 L running in the loop.This process imposes constraints on the real and imaginary parts of λ s λ * to ( 5), but stronger [24]: Another set of flavor constraints arises from the treelevel s → d ν τ ν τ transition mediated by the leptoquark of electric charge −1/3 (i.e., ξ d ), as shown in Figure 3.This contributes to the K + → π + ν ν decay, and imposes limits [24] on the real and imaginary parts of λ s λ * d that have a different M ξ dependence than the limits from K 0 − K 0 mixing.Conservatively the K + → π + ν ν limits can be written as We assume that |λ d | ≪ |λ s |, so that the constraints from kaon measurements are satisfied without imposing an upper limit on |λ s |.Combining this assumption with the earlier one made in the discussion of the bounds from D 0 − D 0 , we will neglect the couplings of ξ to firstgeneration quarks.
Even when the leptoquark does not couple to firstgeneration quarks, there are nontrivial constraints on the λ b λ * s product of couplings.These arise from comparing the measurements of the B s − B s meson mixing with the computation [25] of the box diagrams with ξ and ℓ 3 in the loop (replace the s and d fields by b and s in Figure 2), which gives Here we used the fit result from [26] for the B s mass difference: ∆m(B s )/∆m(B s ) SM = 1.14 ± 0.08.Other constraints on λ b λ * s are set by the measurement of the B + → K + ν ν branching fraction (B K + ) [27], and by the existing limit on the B 0 → K ⋆0 ν ν branching fraction (B K * 0 ) [28], which get contributions from the tree-level b → s ν τ ν τ transition [25,29] (as in Figure 3 with appropriate quark replacements).The recent measurement B K + = (23 ± 7) × 10 −6 performed by Belle II [27] is 2.7σ higher than the SM prediction.Thus, one can derive both an upper limit and a lower limit on |λ b λ s | at the 95% CL.However, the lower limit needs further experimental confirmation of potential new physics contributions to B + → K + ν ν, and thus we will not consider it as a constraint on this model. 1 Using input parameters from [30], we find that the upper limit imposed by the B K * 0 limit is stronger than the one from the B K + measurement, and is given by Due to the different dependence on M ξ , this constraint is weaker than (9) for M ξ > 950 GeV.Given that we assume small U (1) ψ breaking, Yukawa couplings of ξ to b or s quarks are expected to satisfy 2), so the constraints ( 9) and ( 10) are rather mild.

B. Decays of the antileptons
For m ψ ≪ M ξ we can integrate out the leptoquark from ( 3) and ( 6) to obtain the following 4-fermion interactions that violate ψ number but preserve lepton number: The Lorentz indices here are contracted inside the square brackets, while the SU (3) × SU (2) W indices are contracted between the two square brackets.These operators induce 3-body decays of the VALs.There are four final states that involve a charm antiquark, with partial widths given at leading order by 1 Reconciling the measurement for B K + and the experimental limit on B K * 0 may eventually require new physics coupling to right-handed b and s quarks, generating a right-handed vector interaction, see e.g.[30].We will not explore here the range of parameters in our model capable of reconciling these experimental results.
where small corrections due to the mass-splitting between the two components of ψ are neglected, and we defined For m ψ ≳ m t + m τ ≈ 175 GeV, decays that involve a top antiquark are kinematically allowed: (14) with the suppression due to the top mass taken into account through a function of fermion masses: For m ψ ≳ m t +m b +m τ ≈ 180 GeV, there are also decays that involve both a top antiquark and a b quark: where If |λ c | ≪ |λ t |, the decay modes of ψ that involve its Yukawa coupling to the top quark may dominate even for m ψ < m t .In that case, the top quark is off-shell, and the ψ VALs have 4-body decays: Let us now consider the limit |λ t | ≪ |λ c |, which implies that the total widths of ψ − and ψ 0 are dominated by the partial widths (12).The ensuing decay length in the rest frame for the ψ VALs is Even after taking into account the typical boost of the ψ pair produced at the LHC, the ψ VALs decay promptly into visible final states provided c τ ψ ≲ 10 −2 cm.We focus on this scenario in what follows.
A small mass-splitting between the two components of the ψ doublet is induced at one loop.The contributions from the Z boson or photon running in the loop have been computed [31], and give δm ψ = m ψ − − m ψ 0 > 0. This mass-splitting is only logarithmically dependent on m ψ , and continuously grows from 250 MeV to 350 MeV when m ψ grows from 100 GeV to 1 TeV.This masssplitting allows a charged-current decay, ψ − → ψ 0 π − , with a width where f π ≈ 130 MeV is the pion decay constant, and G F is the Fermi constant.The decay length in the rest frame associated with ψ − → ψ 0 π − is above 0.6 cm, and thus much longer than the one given in (18).Consequently, the branching fraction for ψ − → ψ 0 π − is small enough so we can neglect this process.An implication of the expression for the decay length in (18) is that the couplings λ s and λ b , although expected to be small due to the U (1) ψ symmetry, cannot both be arbitrarily small if the ψ VALs decay promptly in the detector.In Figure 4, the unshaded region corresponds to prompt ψ decays (c τ ψ < 10 −2 cm) and is allowed by the B meson constraints (9) and (10).The mass ratio is fixed there at M ξ /m ψ = 5, which is near the lower end that allows 2 us to integrate out ξ and use the 4-fermion operators (11).For larger M ξ /m ψ ≡ r m , all the lines in Figure 4 move up by the following factors: r m /5 for the upper solid line, r m /5 for the dashed line, and (r m /5) 2 for the lower solid and dotted lines.
The Yukawa coupling of the b quark to the ξ leptoquark is fixed in Figure 4  .If instead of the |λ t /λ c | ≪ 1 limit assumed in Figure 4, we consider the opposite limit of dominant λ t , then the region of prompt decays is reduced.This is because these decay widths decrease due to the additional phase-space suppression for final states with a top quark.
For m ψ < m t , the top quark can decay through an off-shell ξ into 3-body final states, such as t → ψ 0 b τ + and t → ψ + b ν τ .The branching fractions of these processes are proportional to |λ t λ b | 2 , and are highly phasespace suppressed.Nevertheless, it is worth searching for peculiar cascade decays of the top quark, for example t → ψ 0 b τ + → (c b τ − ) b τ + .

III. COLLIDER PHENOMENOLOGY
The global U (1) ψ symmetry of (1) implies that direct production of ψ VALs at colliders can proceed only in pairs, through electroweak processes.Although the explicit U (1) ψ breaking allows in principle single leptoquark production, followed by ξ → ψ c or ξ → ψ t, the small couplings in (6) and the need for a τ in the initial state effectively suppress this channel.Effects of a ξ u exchanged in the t-channel [32] lead to ss → τ + τ − and b b → τ + τ − partonic processes (the latter is further PDF suppressed), but these are suppressed It is also possible to pair produce the leptoquarks, with one of them decaying to a ψ and the other one decaying through the U (1) ψ breaking couplings (6).This would lead to a final state with a single ψ, but we will neglect it in what follows as the associated branching fractions are likely to be very small.
In this Section, we will study pair production of the new particles, followed by decays via couplings involving SM quarks of either the second or third generation.We will not address here the case where the couplings to the second-and third-generation quarks are comparable, which would imply sizable branching fractions for final states involving t and s quarks, or b and c quarks.
A. Pair production of fermion doublet at the LHC The ψ 0 and ψ − fermions, which carry lepton number L = −1, have antiparticles ψ 0 and ψ + fermions that carry lepton number L = +1.There are four main mechanisms for ψ production at hadron colliders: NLO cross sections for pair production of a weak-doublet fermion through s-channel off-shell electroweak bosons at the 13.6 TeV LHC.The sum of the cross sections for pp → W + * → ψ + ψ 0 and pp → W − * → ψ − ψ 0 is given by the dashed red line.The cross section for pp → Z * → ψ 0 ψ 0 is given by the solid blue line, and is almost equal to the cross section for ψ + ψ − production (see Figure 6).
Each of the above s-channel electroweak boson is offshell, and only the dominant partonic processes are shown here.
To compute the cross sections for production of each of the four fermion pairs in proton-proton collisions, we generated model files using FeynRules [34] and run simulations using MadGraph aMC@NLO [35].The PDF set used here is NNPDF4.0NLO [36].The renormalization and factorization scales were set dynamically by Mad-Graph aMC@NLO.
In Figure 5 we show the ψ pair production cross sections, at the next-to-leading order (NLO) in α s , as a function of the ψ mass, for pp collisions at √ s = 13.6 TeV, which is the center of mass energy of the current Run 3 at the LHC.The sum of the cross sections for the pp → W + * → ψ + ψ 0 and pp → W − * → ψ − ψ 0 processes4 is shown as the dashed red line, while the cross section for pp → Z * → ψ 0 ψ 0 is shown as the solid blue line.The cross sections for the pp → γ * , Z * → ψ + ψ − and pp → Z * → ψ 0 ψ 0 processes differ by less than 10% for m ψ in the 0.1-1 TeV range.Furthermore, the cross sections for pp → W − * → ψ − ψ 0 and pp → Z * → ψ 0 ψ 0 differ by less than 15% for m ψ > 0.6 TeV.These differences are accidentally small due to the interplay between the PDFs and the electroweak couplings of the quarks and ψ.In Figure 6 we show the following ratios of cross sections: σ(pp → γ * , Z * → ψ + ψ − )/σ(pp → ψ 0 ψ 0 ) as a solid red line, σ(pp → W − * → ψ − ψ 0 )/σ(pp → ψ 0 ψ 0 ) as a dashed blue line, and σ(pp → W + * → ψ + ψ 0 )/σ(pp → ψ 0 ψ 0 ) as a purple dash-dotted line.Note that the ratio of the cross sections for the processes mediated by W + and W − grows from about 1.7 to 2.9 when m ψ is increased from 0.1 to 1 TeV because the ratio of the u and d PDFs is approximately 2 at low parton momentum fraction (x) where the valence quarks of the protons are counted.
The effective couplings (11) allow four decay modes for each of the ψ fermions.To reduce the number of final states, we now adopt a flavor ansatz for the coupling of the VALs and leptoquarks to quarks: ξ and ψ couple predominantly to one generation of SM quarks at a time.

B. Couplings to second-generation quarks
In this section, we assume that the couplings to secondgeneration quarks dominate, i.

Signals of the ψ antileptons with taus and jets
The main decay modes of the VALs in the case of dominant couplings to second-generation quarks are The process of the largest cross section involving ψ VALs at proton-proton colliders is then as can be seen in the first diagram of Figure 7.The signal is thus τ + 4j + / E, where j stands for any hadronic jet that is not b-tagged.Two of the four jets can in principle be tagged as c-jets, but that would reduce the Representative diagrams for the production and decay of the VALs.The blobs represent the effective interactions mediated by a heavy ξ leptoquark.The charged-current process leads to a τ + + 4j + / E signal, while the process mediated by an off-shell Z leads to a τ + τ − + 4j signal.
signal rate because currently the efficiency of c-tagging is relatively low.The missing transverse energy (/ E) due to the neutrino is roughly m ψ /3, so it provides limited means of rejecting the background.
An irreducible background to the τ + 4j + / E signal is SM W + 4j production followed by W → τ ν, which has an inclusive cross section5 times branching fraction of the order of 70 pb [37] when the transverse momentum of each jet is above GeV.Since this is at least an order of magnitude larger than the signal even for m ψ near 100 GeV, and there are additional challenging backgrounds (such as pure QCD in the case of hadronic tau decays), these charged-current channels can be probed only if the experimental collaborations would perform advanced, dedicated searches.
Another channel is the neutral-current process which is shown in the second diagram of Figure 7.This τ + τ − + 4j signal has a smaller cross section (see Figures 5), but the backgrounds are also significantly smaller.Most notable backgrounds include SM production of Z +4j followed by Z → τ + τ − , or W + W − +4j followed by each W boson decaying to τ ν.The small cross sections of these backgrounds, and the additional opportunities at rejecting the background, make τ + τ − +4j the most sensitive channel.
A CMS search for a τ + τ − + 4b signal [12], arising from pair production of a vectorlike lepton doublet that decays through an off-shell vector leptoquark [8][9][10][11], has set a lower mass limit of 500 GeV.That CMS search, based on Run 2 data, relied on at least three b-tags to reduce the backgrounds.Given that our τ + τ − + 4j signal does not include b jets, the highest m ψ that a dedicated search with Run 2 data can explore is substantially below 500 GeV.
The remaining ψ pair production channel, leads to a difficult 4j + / E signal.Searches at the LHC in similar channels are sensitive to cross sections [39] larger by an order of magnitude compared to the one for ψ + ψ − given in Figures 6 and Figures 5. Nevertheless, future dedicated searches for the VALs may combine this chanwith τ + τ − + 4j and τ + 4j + / E to increase the sensitivity.
We will not attempt to predict the range of m ψ that the ATLAS and CMS collaborations can explore, but we can safely conclude that the mass reach in the ψ pair production channels is much smaller than in the case of a vectorlike lepton doublet that decays into two SM particles via mass mixing with the tau (the current lower mass limit in that case is 1.05 TeV [5]).
The cross section for pair production of VALs at a muon collider or an e + e − collider is large enough [40,41], and the backgrounds are much smaller than at hadron colliders.Therefore, a lepton collider with √ s ≥ 2m ψ would allow a thorough investigation of the VAL properties.

Signals of the ξ leptoquarks with taus and jets
Given that the breaking of the global U (1) ψ symmetry is small, and that the couplings to second-generation quarks are assumed to be dominant in this Section, the largest interaction of the leptoquark doublet ξ is provided by the λ c ξ † c c R ψ R term in (3).This implies that the dominant decay modes for the ξ u and ξ d leptoquarks are for M ξ > m ψ .Each ψ fermion produced in a ξ decay subsequently undergoes a 3-body decay into SM fermions, mediated by an off-shell ξ.As a result, leptoquark pair production is followed by the following cascade decays Thus, pair production of the leptoquark of electric charge +2/3 (ξ u ) leads to a τ − τ + + 6j signal with a topology shown in Figure 8. Existing searches for leptoquarks coupling to a tau and a quark assume a (τ b)(τ b) or (τ t)(τ t) signal, and set lower mass limits of 1.19 TeV [42] and 1.43 TeV [5,43], respectively.Compared to the existing (τ b)(τ b) search, our τ − τ + + 6j signal does not include b jets and has much softer taus and jets due to the larger number of finalstate particles.Consequently, the backgrounds are much larger, so we expect that the mass reach of even a dedicated search for ξ u with the Run 2 data would be below 1 TeV.
Pair production of the leptoquark of electric charge −1/3 (ξ d ) leads to a 6j + / E signal.Current leptoquark in the jets + / E channel [44] assume a (νj)(νj) signal, and set a lower limit mass limit of 1.14 TeV.As our signal is plagued by much larger backgrounds, due to larger jet transverse momenta and more missing transverse energy, we again expect that M ξ can be substantially below 1 TeV.Thus, the ξ leptoquarks are clearly more elusive than the leptoquarks searched so far in LHC experiments.

C. Couplings to third-generation quarks
Let us now consider the case where the couplings to third-generation quarks dominate, i.e., the Yukawa couplings in Eqs. ( 3) and ( 6 In this case the main signal at proton-proton colliders is τ ± W + W − + 4b + / E, and arises from the diagram with 9. Pair production of VALs through an off-shell W + , followed by decays induced by couplings to third-generation quarks.This process leads to a τ + W + W − + 4b + / E signal. an off-shell W + shown in Figure 9 and the analogous diagram with an off-shell W − .Tagging three or four of the b-jets substantially reduces several backgrounds (e.g., t t plus a W or a Z, and W Zb b) without significantly depleting the signal.For a b-tagging efficiency ϵ b in the 0.8 − 0.9 range [45,46], the combined efficiency of at least three b-tags, ϵ 3b = ϵ 3 b (4 − 3ϵ b ), is in the 82% − 95% range.
Nevertheless, irreducible backgrounds with four b jets exist in the SM, including t th 0 or W W + 4b.The presence in the final state of two muons or electrons of same charge, due to leptonic decays of the τ or W bosons, suppresses these backgrounds, but at the cost of also reducing the signal.The combined branching fractions for the τ ± W + W − + 4b + / E final state to include two samesign charged leptons (ℓ ± ℓ ± where ℓ stands for an electron or a muon), or three charged leptons (ℓ ± ℓ + ℓ − ), are where B τ ℓ is the average of the τ branching fractions into µν ν and eν ν, and lepton universality is taken as a good approximation in the W branching fractions [47].
Even these ℓ ± ℓ ± + (≥ 3)b and 3ℓ + (≥ 3)b signals have SM backgrounds, especially from t t t t and t t t productions.There are also important backgrounds due to leptons produced in meson decays or in photon conversions, and due to the misidentification of the lepton charge or of the b tags.
Despite all these complicated backgrounds, we could have a rough estimate for the Run 3 reach in ψ pair production by comparing to the pp → t t t t process, which has recently been observed by the ATLAS and CMS collaborations [48,49].With 138-139 fb −1 of data at √ s = 13 TeV, the measured t t t t cross section (σ 4t ) has a central value of about 18 fb for CMS and 24 fb for AT-LAS, and are within 1σ of each other and consistent with the SM prediction [50].The combined leptonic branching fractions for ψ signals, given in (28), are smaller than the analogous ones for t t t t (B ℓ ± ℓ ± ≈ 11%, B 3ℓ ≈ 4.3%).Furthermore, the backgrounds are larger at √ s = 13.6 TeV compared to 13 TeV, so even with a larger integrated luminosity of 300 fb −1 the cross section sensitivity to ψ pair production cannot be smaller than roughly 20 fb (i.e., about the measured σ 4t in Run 2).This corresponds to a mass reach m ψ ≲ 500 GeV, as can be seen in Figure 5.
The reach in m ψ can be increased by adding the pp → Z * → ψ 0 ψ 0 → τ + τ − W + W − + 4b process, which has a smaller production cross section (by a factor of about 4, see Figures 5 and 6) but larger leptonic branching fractions (B ℓ ± ℓ ± ≈ 17%, B 3ℓ ≈ 10%).This process also leads to final states with hadronic τ decays that would further increase the mass reach.Even though the pp → ψ + ψ − → t t b b + / E process does not produce two same-sign leptons, it can be useful due to the missing transverse energy, which suppresses the t t b b background.
Combining all these channels and optimizing the event selections would improve the sensitivity, so that the largest m ψ that can be probed in Run 3 is likely in the 600-700 GeV range.Note that this is again significantly below the lower mass limit of 1.05 TeV set by Run 2 searches [5] for vectorlike lepton doublets undergoing standard decays.

Signals the ξ leptoquarks with top quarks
Consider next the ξ leptoquark signals in the case of ψ decays involving third-generation quarks shown in (27).Pair production of the charge 2/3 leptoquark followed by cascade decays through the ψ VALs leads, as shown in the diagram from Figure 10, to a τ + τ − b b + 4t final state.Thus, the signal is τ + τ − + 6b + 4W , and includes a large number (up to 6) of charged leptons in addition to the six b jets.These provide various ways of eliminating the backgrounds.On the other hand, requiring many leptons implies small combined branching fractions which would reduce the signal too much.Moreover, the large mul-tiplicity of particles in the final state implies that most of them are relatively soft.Following the discussion in Section III C 1, we expect that the reach in ξ pair production cross section is of the same order of magnitude as for 4t, so around 20 fb.This corresponds to M ξ ≲ 0.9 TeV [33,51,52], which is substantially below the Run 2 lower mass limits (in the 1.2-1.4TeV range [5,42,43,53]) for leptoquarks that have 2-body decays into SM particles.Finally, if m ψ is in the ∼ 100 − 180 GeV range, then the ψ decays shown in (27), which involve a top quark, are kinematically closed.In that case, ψ 0 and ψ − may undergo 4-body decays through an off-shell top quark (ψ 0 → W − b b τ + and ψ − → W − b b ν τ ).If these are the main decay modes, then the additional phase-space suppression is likely to make the decay length long enough so that a highly-ionizing charged track due to ψ ± can be observed.Another possibility is that |λ c | is large enough (while satisfying |λ c | 2 ≪ |λ t | 2 so that the couplings to third-generation quarks dominate), such that the 3-body decays involving a charm quark, have larger widths than the 4-body decays, and are prompt.We focus on this case in the remainder of this Section.Pair production of ψ then includes the following signals at the LHC: Although the presence of b jets avoids some of the backgrounds discussed in Section III B, these signals remain challenging.The above charge-current process has very large irreducible backgrounds from t t and W bbjj productions.Even the neutral-current process may be hard to observe due to t tjj and W + jets backgrounds.
It may be that the discovery modes for the VALs in this case are via the decays of the leptoquarks.Note that the latter decays involve top quarks even when ψ is below the m t threshold.The most interesting process for m ψ below the top threshold is The signal includes four b jets and up to four charged leptons, so if M ξ is roughly in the 0.5 − 1 TeV range, then it is possible that a dedicated search in this channel would discover both ξ u and ψ 0 in Run 3 of the LHC.
IV. CONCLUSIONS extension of the SM proposed here includes two new fields, which are weak doublets: a vectorlike fermion ψ transforming as (1, 2, −1/2) under the SM gauge group, and a scalar leptoquark ξ transforming as (3, 2, +1/6).Gauge invariance allows a single type of renormalizable interaction involving both these fields, namely a Yukawa coupling to the SM up-type weak-singlet quarks, as displayed in (3).Likewise, there is a single type of interaction involving only ξ and SM fields: a Yukawa interaction of a down-type weak-singlet quark with a SM lepton doublet (assumed here to be the third-generation one) and ξ, of the form shown in (6).Both these types of Yukawa interactions conserve lepton number provided both ψ and ξ carry lepton number L = −1.Since the ψ doublet has the gauge charges of an often studied vectorlike lepton, but opposite lepton number, we use the term vectorlike antilepton, or VAL.
Interactions (3) preserve a global U (1) ψ symmetry under which ψ and ξ carry the same charge, so it is natural to expect that the interactions (6), which break that symmetry, have smaller couplings.This coupling hierarchy implies that the ξ scalars elude direct searches for leptoquark decays into 2-body SM final states, while allowing the 3-body decays of the VALs to be prompt for a range of parameters, as shown in Figure 4.
Constraints from flavor-changing processes, such as D 0 − D 0 mixing, K 0 − K 0 mixing, and K + → π + ν ν, suggest that the Yukawa couplings of the new particles to first-generation quarks are suppressed.Even the couplings to second-and third-generation quarks are constrained, but only mildly (see Figure 4), by flavor-changing processes including B s − B s mixing, B 0 → K * 0 ν ν, and B + → K + ν ν.
The electroweak production of vectorlike fermion pairs at the LHC, computed at NLO in α s for √ s = 13.6 TeV and displayed in Figure 5, applies to a VAL doublet, but also to a vectorlike lepton doublet.While the current lower mass limit is already above 1 TeV for a vectorlike lepton [5], it is substantially less stringent for a VAL.If the couplings of the new particles are predominantly to the second-generation quarks, then the main LHC signals are τ + τ − + 4j and τ + 4j + / E for VALs, and τ + τ − + 6j for leptoquarks.These signals with taus and jets are difficult to observe due to large backgrounds, but with advanced experimental techniques can be probed in the current Run 3 of the LHC.
The main LHC signals in the case of dominant couplings to third-generation quarks are τ ± W + W − + 4b + / E and τ + τ − W + W − + 4b for VALs, and τ + τ − + 6b + 4W for leptoquarks.The presence of several charged leptons from τ or W decays makes these signals easier to detect than the ones with just taus and jets.Nevertheless, the large multiplicities of these signals imply that most objects in the final states are relatively soft.Thus, even in the case of third-generation couplings, the vectorlike fermions and leptoquarks make each other more elusive than in the case of standard 2-body decays.

FIG. 1 .
FIG. 1. Box diagram contributing to D 0 − D 0 meson mixing, with the ψ VAL and ξ scalar running in the loop.

FIG. 4 .
FIG.4.Constraints on |λs| as a of the ψ mass, where λs is the Yukawa coupling of the s quark to the ξ leptoquark.Here we fix the b quark coupling λ = λs.The unshaded region is allowed and corresponds to prompt ψ decays, for M ξ = 5m ψ .The upper shaded (blue) region is ruled out by ξ contributions to Bs − Bs mixing (above the solid blue line) and B 0 → K ⋆0 ν ν (above the dashed blue line).The lower shaded (orange) region corresponds to ψ displaced vertices (rest-frame decay length c τ ψ < 10 −2 cm) for a c-quark coupling λc = 0.2 and |λt/λc| ≪ 1.The edge of that region moves to the dotted red line when λc = 0.6.
at |λ b | = |λ s |.For a different λ b , the dashed and upper solid lines scale as |λ s /λ b | 1/2 , while the dotted and lower solid lines scale as √ 2/(1+|λ b /λ s | 2 ) 1/2 FIG. 5.NLO cross sections for pair production of a weak-doublet fermion through s-channel off-shell electroweak bosons at the 13.6 TeV LHC.The sum of the cross sections for pp → W + * → ψ + ψ 0 and pp → W − * → ψ − ψ 0 is given by the dashed red line.The cross section for pp → Z * → ψ 0 ψ 0 is given by the solid blue line, and is almost equal to the cross section for ψ + ψ − production (see Figure6).
) satisfy |λ s | 2 ≪ |λ b | 2 and |λ c | 2 ≪ |λ t | 2 .As this scenario leads to signals involving t and b quarks, the ψ and ξ particles are less elusive than in the case of couplings to second-generation quarks discussed in Section III B.

1 .
Signals of the ψ antileptons with top quarksIf the VALs are sufficiently heavy, m ψ > m t + m b + m τ ≈ 180 GeV, then their main decay modes are:

FIG. 10 .
FIG.10.Representative diagram for pair production of the ξu leptoquark followed by cascade decays via the ψ VALs, leading to a τ − τ + b b + 4t signal at hadron colliders.

3 .
Signals for m ψ in the 100 − 180 GeV range