Measuring Electroweak Quantum Numbers of Color Sextet Resonances at the LHC

We study the prospect of measuring the electroweak quantum numbers of beyond the Standard Model (SM) color sextet particles that decay into same-sign top quark pairs. Among these particles, the color sextet scalars give rise to top quarks with the same chirality, while the top quarks coming from the color sextet vector would have opposite chirality. This difference gets encoded in the angular distributions of the bottom quarks and leptons originating from the decays of the top quarks. We utilize this feature and the energy distributions of the final state jets and leptons to distinguish among the three possible color sextet resonances, taking into account various SM background processes at the $13$ TeV LHC.


I. INTRODUCTION
A variety of beyond the Standard Model (BSM) scenarios, especially those addressing the Higgs hierarchy problem, e.g., supersymmetry or composite Higgs, predict new physics around the TeV scale (see Ref. [1] for reviews).
The search for such BSM states has been actively going on at the Large Hadron Collider (LHC) and will continue through its high-luminosity (HL-LHC) phase, in conjunction with other indirect probes.
Given the absence of new physics at the LHC so far, we can ask a bottom-up and purely group theoretic question as follows. Noting that the SM is based on the gauge group SU(3) c × SU(2) L × U(1) Y , we can ask which possible BSM scalar or vector particles can have direct, treelevel couplings to SM fermions. A priori, there are a number of such BSM states [2,3]. However, a subset of them would couple to both leptons and quarks so as to mediate proton decay at tree level, unless that is forbidden by some other global symmetry, and therefore are ruled out [4] for TeV-scale masses. Among the remaining states, those coupling to two top quarks are particularly interesting from an experimental perspective. As we will discuss in the following, two such scalar and one such vector resonance have the SM quantum numbers, Φ 1 ∼ (6, 1, 4/3), Φ 3 ∼ (6, 3, 1/3), Φ µ 2 ∼ (6, 2, 5/6). (1) At the LHC, these states can be produced via their couplings to the first-generation quarks, and subsequently, they can decay into a like-sign top quark pair. The top quarks can then decay into a pair of b-jets, a pair of like-sign leptons, and neutrinos when the intermediate W bosons decay leptonically. The phenomenology and ultraviolet origin of such color sextet diquarks have been discussed extensively in the literature, see, e.g., [2][3][4][5][6][7][8][9][10][11][12][13][14][15][16].
The question we would like to ask in the present work is the following: suppose a discovery of a color sextet particle is made at the (HL-)LHC. Then purely based on the lab-frame observables, can we extract the quantum numbers of the discovered sextet state and distinguish among * Electronic address: soubhik@berkeley.edu † Electronic address: rafiqulrahaman@hri.res.in ‡ Electronic address: ritesh.singh@iiserkol.ac.in the three possible sets of quantum numbers of a generic color sextet particle, as mentioned in Eq. (1)?
To answer this, we first note that the fermionic couplings of interest are inherently chiral in nature. Therefore, the top quarks coming from the sextet decay would carry definite polarization and this plays a crucial role in extracting the quantum numbers of the sextet states [12,14]. For example, the rest-frame angular distributions of leptons and b-jets are different depending on the polarization of the parent top quarks, as we describe explicitly below. However, since in such processes, the top quarks decay leptonically, there is missing energy in the form of neutrinos in the final state, and reconstructing the rest frame of a top quark is not immediate. While it is possible to use the M T 2 observable [17,18] to address this issue [19,20], it is still useful to construct observables that do not rely on any such rest frame reconstruction. To this end, we construct some new lab-frame observables that can be used to investigate and isolate the quantum numbers of the various sextet states.
Some lab-frame observables for processes involving missing energy have been discussed in the literature. One example is the visible energy fraction of the leptons [21,22] Here E l i and E b i are respectively the energies of the lepton and the associated b-quark originating from the decay of the same top quark. This observable is sensitive to the polarization of the top quark. However, in the signal of our interest, we have two b-jets and two leptons, and the effectiveness of z i inherently depends on correctly pairing a bquark with the associated lepton. This motivates us to look for additional lab-frame observables which are independent of such pairing. We will show that such observables can be constructed based on the azimuthal distribution of final state visible particles. The rest of this work is organized as follows. In Sec. II, we describe the couplings of the sextet states to the SM fermions and the existing experimental constraints on such states. In Sec. III, we construct the lab-frame observables of interest and explain how top quark polarization and spin correlation plays an important role in this context through a parton-level analysis at the 13 TeV LHC. In Sec. IV, we present our results to distinguish among the color sextet arXiv:2302.11355v2 [hep-ph] 20 Sep 2023 states through a detector-level simulation, taking into account possible SM backgrounds. We conclude in Sec. V.

II. MODEL
In this work, we focus on color sextet BSM resonances that couple to two top quarks. Given the quantum numbers of the SM quark doublet q L = u L d L ∼ (3, 2, +1/6), and the right-handed up-type quark u R ∼ (3, 1, +2/3), the quantum numbers of the BSM scalar and vector resonances are fixed as in (1). We first consider the two scalars, Φ 1 and Φ 3 . We can write the Yukawa coupling of Φ 1 as (see, e.g., [10]) where the superscript c denotes charge conjugation operation. The matricesK a i j are determined by the Clebsch-Gordon coefficients for the sextet representation of SU(3). In both Eq. (3) and Eq. (4) below, the indices a and i, j correspond to color indices and they run over 1 · · · 6 and 1 · · · 3, respectively.
The Yukawa coupling of Φ 3 is given in an analogous manner, except that it couples to the symmetric combination of two copies of q L . The coupling to u L is given by Here we have focused on the part of the isospin triplet field that couples only to the up-type quarks since that can decay into a pair of top quarks, which is the signal of our interest. Both in (3) and (4), we have suppressed the generation indices on the Yukawa couplings, λ 1 and λ 3 . Finally, the coupling of the vector resonance is described by, Here we have focused on the up-quark coupling for the same reason as above. We will be agnostic about how Φ µ 2 gets its mass. For a general Yukawa coupling, there would be large flavor changing neutral current processes mediated by the above sextet states. To avoid the stringent experimental constraints from those, we assume that the matrices λ 1 , λ 2 , and λ 3 are flavor diagonal.
With this assumption, the next stringent constraints on λ 1 come from the measurements of D 0 −D 0 mixing [23], where x = ∆m D /Γ D is a D 0 −D 0 mixing parameter. Assuming CP conservation and taking x ≈ 4 × 10 −3 [24], we get Re(λ 1,cc λ * 1,uu ) < 3 × 10 −7 for m φ = 1 TeV. In the following, we will assume a hierarchy between λ 1,cc and λ 1,uu with |λ 1,cc | ≪ |λ 1,uu | so as to satisfy the above bound. The dominant constraints on λ 2 and λ 3 also come from D 0 −D 0 mixing [12], which can again be suppressed by taking the couplings to second generation quarks to be small as we mentioned above.
We note that Φ 1 and Φ 3 decay to a pair of right-handed and left-handed top quarks, respectively. This implies that we would be able to distinguish between the quantum numbers of these two sextet scalars using the polarization properties of the top quarks. In particular, in its rest frame, a right-handed top quark would decay into leptons whose average distribution would peak in the same direction as the top quark spin, while the associated b quark distribution would be peaked in the opposite direction. Thus after boosting to the lab frame, the angle between the two leptons, each coming from the two daughter top quarks from Φ 1 decay, will be peaked around ∆φ = π, while the angle between the two b quarks will be more broadly distributed around ∆φ = π. The situation with Φ 3 is exactly the opposite of this. While the vector BSM resonance Φ µ 2 decays into a same-sign top pair as well, the top quarks would have opposite chirality. These features can then be used to distinguish among Φ 1 , Φ µ 2 , and Φ 3 , as we will see below. We will refer to these particles as Singlet, Doublet, and Triplet, respectively, based on their SU(2) L quantum numbers. For numerical simulations, we choose the benchmark: m φ = 1 TeV, λ uu = λ tt = 0.003 for all the three sextet particles. We note that there is an upper bound of 1.2 pb on the tt production cross-section given by CMS coming from the same-sign di-lepton, missing energy, and jets search [25]. However, the tt production cross-sections in our models are much smaller compared to the CMS limit. For example, σ tt ≃ 0.02 pb for the Singlet case in our chosen benchmark, which is only 1% of the CMS limit.

III. OBSERVABLE
The polarization of a top quark is preserved in its decay products and this plays an important role in our analysis. The double angular distribution of daughter fermions from the two top quarks can be expressed as [26] 1 σ in terms of the polarization of two top quarks (p t 1 , p t 2 ) and their spin correlation (pp tt ). Here, θ i are the decay angles of the daughter fermions at their respective top quark's rest frame. The quantity α i is called the analyzing power which depends on the daughter fermion. For example, for a lepton, α l = 1, whereas for a b-quark, α b = −0.4 [27]. The polarization p t is positive (negative) for the right (left)polarized top quark. The above formula in Eq. (7) is valid only in the rest frame of the top quark which requires a complete reconstruction of the missing neutrinos [28][29][30][31].
The polarization of the top quark can also be obtained from the distribution of the ratio of energies of the daughter bquark (E b ) and the top quark (E t ), E b /E t . This again requires a complete reconstruction of the top quark momenta and hence that of the neutrinos.
vs. ∆φ (b 2 , p miss ) (middle-column), and ∆φ (l 1 , p miss ) vs. ∆φ (l 2 , p miss ) (right-column) for the signals with parton level events. The red points represent maximum density, and light blue represents minimum density for the number of events. two b-quarks with the corresponding leptons. This can be achieved using the fact that in the lab frame, a sibling lepton and a b-quark have a smaller ∆R than the ∆R between a cousin lepton and a b-quark. Here, ∆R = ∆φ 2 + ∆η 2 where φ is the azimuthal angle and η is the pseudorapidity of a particle. We label the two leptons as l 1 and l 2 , ordered according to their transverse momenta (p T ), i.e., with p T (l 1 ) > p T (l 2 ). The two b-quarks can pair with the leptons in two ways. If the correct pairs are such that l 1 and b 1 come from one top quark, while l 2 and b 2 come from another top quark, then we expect to have . This criterion is satisfied for more than 96 % of times for the chosen mass of 1 TeV. For a heavier scalar, the top quarks will be more boosted which will lead their decay products to collimate further, and this increases the efficiency of pairing the lepton and the b-quark correctly.
We study the distributions for the energy fraction variables z 1 and z 2 defined in Eq. (2), pairing the leptons and b quarks as described above using parton-level events at √ s = 13 TeV to see how these variables can separate the three signals, namely Singlet, Doublet, and Triplet. The normalized distributions for z 1 and z 2 are shown in Fig. 1. Both z 1 and z 2 peak on the lower side for Triplet (red lines) as the produced top quarks are left-handed in this case. This means the final state leptons are less boosted as we go from the top rest frame to the lab frame, and thus z i 's peak at smaller values. For the Singlet (blue lines), the variables z i sharply drop near 0.2, having more fractional events with z i > 0.5, compared to the Triplet. For the Doublet (green lines), however, the variables peak below z i = 0.5, but the asymmetry with respect to z i = 0.5 is smaller than the Triplet since the Doublet decay gives rise to both left-handed as well as right-handed top quarks. With realistic detector effects the strengths of these variables might diminish, but the qualitative result would remain the same, as we will see below. We define net asymmetries for the visible energy fractions (z i ) as with N(x > c) being the number of events with x > c, and N tot is the total number of events. We only consider the tt production process containing 2ℓ + final state and do not include the charge conjugation process because σtt /σ tt ≈ 0.03 owing to the small parton distribution functions (PDFs) at the LHC. Similar to the polarization parameters, one can construct variables to capture the spin correlation between the two top quarks using the lab frame angular separation between the final state particles in the transverse plane. We study two-dimensional (2D) correlations between ∆φ (l 1 , l 2 ) vs. ∆φ (b 1 , b 2 ), ∆φ (b 1 , p miss ) vs. ∆φ (b 2 , p miss ), and ∆φ (l 1 , p miss ) vs. ∆φ (l 2 , p miss ). The normalized 2D correlations are shown in Fig. 2 with parton-level events. The correlations are not symmetric with respect to the diagonal axes, and they are different for different signals. In the first column in Fig. 2, the distributions are more peaked in ∆φ (l 1 , l 2 ) compared to ∆φ (b 1 , b 2 ) for the Singlet, while the opposite is true for the Triplet. For the Doublet, however, the distributions are less peaked and more symmetric around the line. This can be explained in the following way. As the two top quarks are produced from a heavy resonance, ∆φ (l 1 , l 2 ) and ∆φ (b 1 , b 2 ) are expected to peak around π in the lab frame due to relativistic focusing. But due to the positive polarization of top quarks from the Singlet, the leptons (b-quarks) are emitted primarily in the same (opposite) direction as the top quark spin in the top rest frame. As a result, after boosting to the lab frame, the peak of ∆φ (l 1 , l 2 ) becomes sharper, while the peak of ∆φ (b 1 , b 2 ) becomes broader. For the Triplet with negatively polarized top quarks, the opposite happens, i.e., the peak of ∆φ (b 1 , b 2 ) becomes sharper, while the peak of ∆φ (l 1 , l 2 ) becomes broader. Finally for the Doublet, with a mix of positively and negatively polarized top quarks, the peaks of ∆φ (l 1 , l 2 ) and ∆φ (b 1 , b 2 ) have similar broadening.
The ∆φ (l/b, p miss ) peak near zero and π, as shown in the second and third column of Fig. 2. The distribution of ∆φ (l/b, p miss ) can be explained in a similar way as that of ∆φ (l 1 /b 1 , l 2 /b 2 ), after noting thatν is distributed the same way as b quarks around the top quark spin in the top rest frame. The peak of ∆φ (l, p miss ) is sharper than that of ∆φ (b, p miss ) for the Singlet, and the opposite is true for the Triplet. For the Doublet, ∆φ (l, p miss ) and ∆φ (b, p miss ) behave roughly in the same way. Given this, we propose the following variables based on these 2D correlations: x lb = ∆φ (l 1 , l 2 )/π − ∆φ (b 1 , b 2 )/π, x ll = ∆φ (l 1 , p miss )/π + ∆φ (l 2 , p miss )/π, x bb = ∆φ (b 1 , p miss )/π + ∆φ (b 2 , p miss )/π, to extract the spin correlation of the two top quarks. The asymmetries for the correlations are defined as, Note that these ∆φ variables are independent of pairing the leptons and the b quarks, the pairing is required only for the variables z i . To maximize the asymmetries defined in Eqs.  Table I for the three signals. It is clear that with the magnitudes and signs of the asymmetries, one can easily identify and separate the signals. However, we need to extract these asymmetries with realistic detector effects, including possible backgrounds. Therefore, in the next section, we analyze the signals with detector-level events in the presence of SM backgrounds.

IV. RESULTS
In this section, we discuss how to distinguish the three types of signals in the presence of background events, with a simplified detector-level simulation. The backgrounds that can mimic our signal topology of two positively charged leptons (2l + ), two b-jets, and missing energy are [25,[32][33][34] ttW + , ttZ, tth, ttW + W − , W + W + j j, ZZ j j, ZZW + , and W + W − Z. In addition, the tt+jets process, which has a large cross-section, can fake our signal if the lepton charges get misidentified [35][36][37]. We generate the signal (Singlet, Doublet, and Triplet) and background events in MadGraph5 aMC@NLOv2.7.3 [38] at leading order (LO) in QCD, without cuts on the final state particles, and with a dynamic choice of factorization scale given by ∑ i M T i /2, where M T i is the transverse mass of the i-th final state particle. We use nn23lo1 [39] for the PDFs. Background events are generated in MadGraph5 with the final states that can give rise to 2l + 2b + E T at the detector level. Events are then passed to PYTHIA8.2 [40] for showering and hadronization, followed by fast detector simulation in Delphes v3.4.2 [41]. Events are selected at detector level with at least two b-tagged jets, two positively charged leptons (2l + ), and missing transverse energy with the following selection level cuts p T (l) > 10 GeV, p T (b) > 20 GeV, E T > 10 GeV, With the selected background (BKG) and signal events, we study the one-dimensional and two-dimensional normalized distributions for the variables (z i and ∆φ correlation, respectively) that we introduced in the previous section. These are shown in Figs. 3 (z i ) and 4 (∆φ ). The qualitative features of the distributions at the detector level remain the same as the parton level distributions studied in the previous section. The z i distributions make clear distinctions between the Singlet and Triplet peaking in the right and left, respectively. Distributions are roughly symmetric around z i = 0.5 for the Doublet. For the BKG, the z 1 distribution is symmetric around z 1 = 0.5 as well, while the z 2 distribution is asymmetric. In the case of all three signal benchmarks, the distributions for the z 1 and z 2 are identical owing to the similar kinematics. However, this not true for the background because the paired lepton and b quark do not necessarily originate from a top quark. Additionally, z 2 peaks at lower values owing to the p T ordering of leptons. In the 2D ∆φ distributions, the background events are distributed quite differently than the signals. Therefore, it is possible to distinguish the signals based on these distributions, even with the detector-level events. However, the total cross-section for the background is larger than that for the signal, and therefore we now discuss some kinematic cuts that are needed to reduce backgrounds and increase the signal sensitivity. We study various kinematic distributions, such as the transverse momenta (p T ) of the final state particles, total hadronic energy (H T ), and visible mass (m l + 1 l + 2 b 1 b 2 ). We find that visible mass, the normalized distribution shown in Fig. 5, can be used to suppress the background and enhance the significance of all three signals. In particular, a cut on the visible mass of (14) maximizes the signal significance. The signal and background processes are summarized in Table II, showing their cross sections at the generation level and the expected number of events for an integrated luminosity of L = 300 fb −1 after the visible mass cut. The total number of background events is expected to be 208 in the 2l + 2b E T final state. The signal significance expected for the three signals are given in Table III for integrated luminosities of L = 150 fb −1 (currently available), 300 fb −1 (next phase of LHC), and 3000 fb −1 (high luminosity phase). We calculate the signal significance [42] using where s and b stand for the total number of signal and background events surviving after cuts. For the chosen coupling of λ = 0.003 and m φ = 1 TeV, the significance for the Triplet and Doublet signals are higher than the Singlet signal. A luminosity of about 1300 fb −1 is required for discovery of Singlet, whereas the Doublet and Triplet can be discovered with about 790 fb −1 and 930 fb −1 of luminosity, respectively with 5σ significance.
Having discussed the discovery potential for the signals, we now turn to our primary goal of this analysis, which is to distinguish among the three signals with the help of the observables discussed in Sec. III. To this end, we first calculate the asymmetries for all the variables and summarize them in Table IV for all three signals and backgrounds separately, as well as, signals in the presence of backgrounds. The numbers in the first three rows show that all three signals can be identified by looking at the value and the sign of the five asymmetries. However, these asymmetries are affected by the background events, as shown in the last three rows. We calculate the differences between any two signals in the presence of backgrounds using all five asymmetries, given in Table IV, in terms of the χ 2 function Here is the statistical uncertainty due to the SM backgrounds with σ (BKG) being the total background cross section. Signal j ( j = 1, 2) denote any two among the Singlet, Doublet, and Triplet signals. We estimate the luminosity required for 2σ , 3σ , and 5σ C.L. separability among the signals using the above χ 2 functions and quote them in Table V. The Singlet and the Triplet are quite different since the former (latter) decays only into right-(left-) chiral top quarks; a luminosity of about 628 fb −1 is required to achieve separability at 5σ C.L., although neither of them can be discovered for that luminosity with the expected number of events, see Table III. On the other hand, we require about 2842 fb −1 and 2061 fb −1 of luminosity for 5σ separation between the Singlet versus Doublet and Doublet versus Triplet, respectively, as the two top quarks have both chiralities originating from the Doublet. Thus, to make distinctions between the signals involving the Doublet,     (14). For all the signals, we choose λ = 0.003 and m φ = 1 TeV.

V. CONCLUSION
In this work, we have discussed how to measure the electroweak quantum numbers of BSM color sextet scalar and vector particles. While all the sextet particles that we consider decay into a like-sign top quark pair, the top quarks have identical chirality for the two sextet scalars and opposite chirality for the sextet vector. Furthermore, one of the scalars give rise to left-handed top quarks while the other decays to right-handed ones. These features can be captured by several kinematic variables that rely only on visible final states. One such variable is the well known visible energy fraction (z i ) of final state leptons, Eq. (2). Here we construct three additional variables, defined in Eq. (9), that depend on the angular correlation between the final state leptons and b-jets. All of these variables are sensitive to the polarization of the top quarks, and in a combined fashion can distinguish among the three possible sextet states.
Through a parton level analysis, we first demonstrate the  utility of the visible energy fraction variables and the angular correlation variables. We study their distributions in Figs. 1 and 2, and compute a set of asymmetries in Table I to identify the differences among the three signals. These show that the asymmetries can fully distinguish, as well as, identify the quantum numbers of the sextet states, in principle. We then implement a simplified detector level simulation, taking into account possible SM backgrounds, to verify how well we can differentiate among the three types of signals. We find that while the three signals can be distinguished among themselves even at the detector level, the inclusion of SM background reduces the difference among the signals. Nonetheless, with sufficient statistics within the reach of the high-luminosity phase of the LHC, the three types of signals can still be distinguished, as demonstrated in Table V. We find higher luminosities are required to make all these distinctions than the luminosities required to discover them. To summarize, if discovered at the LHC, it is possible to measure the electroweak quantum numbers of BSM color sextet particles using top quark polarization and spin correlation observables.