Non-standard signatures of vector-like quarks in a leptophobic 221 model

We consider vector-like quarks in a leptophobic 221 model characterized by the gauge group $SU(2)_L\times SU(2)_2 \times U(1)_X$, where the $SU(2)_2$ is leptophobic in nature. We discuss about the pattern of mixing between Standard Model quarks and vector-like quarks and how we prevent tree level flavour-changing interactions in the model. The model also predicts tauphilic scalars decaying mostly to tau leptons. We consider a typical signal of the model in the form of pair production of top-type vector-like quarks which decays to the tauphilic scalars and a third generation quark. We analyze the resulting final state signal for the 13 TeV LHC, containing $\geq 3j(1b) \, + \geq 2\tau \, + \geq 1l$ and discuss the discovery prospects of such vector-like quarks with non-standard decay modes.


I. INTRODUCTION
As the Large Hadron Collider (LHC) churns out more and more data and gets ready for an energy upgrade, lack of new physics signal at the high energy frontier only makes us  35 TeV given by ATLAS [5]. Since the single production of VLQs depend on the mixing between VLQs and SM quarks, based on the searches for single-production of VLQs both CMS and ATLAS collaborations have given exclusion limits for the product of production cross section and branching fraction for different mass values [8][9][10]. Extensive phenomenological studies on VLQs in standard decay scenarios exist in literature [11][12][13][14][15][16][17].
Exotic fermions are a necessary ingredient in some gauge extended models for anomaly cancellation, e.g., exotic quarks in leptophobic 221 model [18,19], exotic leptons in hadrophobic 221 model [18] and exotic fermions in almost all U (1) extensions [20]. These exotic fermions become vector-like once the full symmetry group breaks down to the SM gauge group. These fermions are quasi-chiral in nature, i.e., they are vector-like under the SM gauge group but chiral under the extra gauge group [20].
VLQs in gauge extended models can have interesting collider signatures because the rich spectrum of the model opens up non-standard decay modes for the VLQs. In this work we have considered the collider signatures of certain non-standard decay modes of top-like VLQ in a leptophobic 221 model characterized by the gauge group SU (2) 1 × SU (2) 2 × U (1) X .
Because of the presence of the non-standard decay modes the existing constraints on the masses of VLQs will get significantly relaxed and the VLQ can lie at the sub-TeV scale.
In general, the mixing between SM quarks and VLQs which play an important role in the phenomenology of VLQs also generate tree level flavour-changing neutral current (FCNC) interactions of SM quarks with the Z boson. Several studies on the mixing of VLQs with SM quarks are available in the literature [16,[38][39][40][41][42][43] which take into account constraints from flavour physics and electroweak precision measurements . Mixings are strongly constrained from FCNC processes. Even in the presence of mixings with VLQs the possibility of avoiding tree level FCNC interactions is possible for judiciously chosen mixing patterns [44]. We discuss such a mixing pattern for the leptophobic 221 model which avoids tree level flavourchanging interactions with the Z and Z . For general mixing scenarios between VLQs and SM quarks, all the neutral scalars present in the model have flavour-changing interactions, thus making it difficult to get a 125 GeV neutral Higgs and simultaneously satisfying constraints from FCNC processes. We show that it is possible to avoid flavour-changing interactions for certain neutral scalars (non-FCNH scalars), which can then lie at sub-TeV scale. In this work we study the collider signatures of the third generation top-type VLQ decaying to a final state with one of these non-FCNH scalars (other than the 125 GeV Higgs) and a third generation SM quark and the scalars then decay dominantly to tau leptons.
The paper is organized as follows. In Section II we discuss our model. In Section III we discuss about the interaction of the VLQs with the SM gauge bosons. In Section IV we discuss the pattern of mixing between the SM quarks and the VLQs for which FCNC interactions of SM quarks with Z-boson and the FCNH interactions with the SM-Higgs boson is zero at the tree level. In Section V we discuss the possible phenomenology of VLQs in the model and explore the possible collider signatures in Section.VI. Finally we conclude and summarize in Section VII.

II. THE MODEL
The SU (2) extensions of the SM characterized by the gauge group SU (3) C × SU (2) 1 × SU (2) 2 × U (1) X are generally called 221 models in the literature. Depending on the way the SM fermions transform under the gauge groups, different versions of 221 models are possible, viz. leptophobic, hadrophobic and left-right symmetric etc. The model is called leptophobic when the SM right-chiral leptons are singlets under SU (2) 2 , hadrophobic when the right-chiral SM quarks are singlets under SU (2) 2 and left-right symmetric when both the SM right-chiral quarks and the right-chiral leptons (with the addition of right-chiral neutrinos to the model) form doublets under SU (2) 2 . The most popular among all 221 models is the left-right symmetric model [45][46][47][48][49][50]. A general classification of all 221 models has been done in [18] based on two types of symmetry breaking patterns of the gauge group.
The two patterns are as follows : • Type-I : SU (2) 1 is identified with SU (2) L of SM. The first stage of symmetry breaking Certain type of 221 models need exotic fermions for the cancellation of anomalies. For example, the leptophobic 221 model needs exotic quarks and the hadrophobic 221 model requires exotic leptons. These exotic fermions become vector-like after the breaking of the full symmetry group down to SM gauge group.
In this work we consider the leptophobic 221 model, which follows the type-I symmetry breaking pattern described above and the SM leptons are singlets under the SU (2) 2 gauge group 1 . We will denote SU (2) 1 as SU (2) L throughout the article. The scalar sector of the model contains two scalar doublets represented by H 1 and H 2 , and a bi-doublet scalar represented by Φ. The symmetry breaking of the full gauge group to U (1) em by the scalars occurs in two stages : (1) The first stage of symmetry breaking of the full gauge group to the SM gauge group   Note that the exotic quarks present in the model are chiral in nature under the full unbroken gauge group. However, after the stage-I symmetry breaking SU (2) 2 × U (1) X → U (1) Y , the exotic quarks become vector-like under the SU (2) L × U (1) Y , which is the SM gauge group. This feature can be realized by following the definition of hypercharge quantum number Y after the first stage of symmetry breaking which is given by where T 2 3 denote the diagonal generator of SU (2) 2 and Q X represents the charge for the gauge group U (1) X . The hypercharge quantum number for the exotic quarks are given by i.e. they are vector-like with respect to the SU (2) L × U (1) Y gauge group.
After the stage II of symmetry breaking, the electric charge for a field is given by where T 1 3 denote the diagonal generator of SU (2) 1 . Following the above definition the electric charges for the VLQs are given by Q(xu 0 i ) = + 2 3 and Q(xd 0 i ) = − 1 3 .
The scalar fields in component form can be written as where The structure of VEVs for the scalar fields are The set of tadpole equations ∂V i , χ 0 i , χ 0 i ) basis and for the charged scalars (M 2 C ) in the (φ + 1 , φ + 2 , χ + , χ + ) basis are given in appendix B 1, B 2 and B 3 respectively. There are four physical CP even scalars in the model. Two of the four CP odd scalars will be physical and the other two are massless Goldstone bosons which become part of the two massive neutral gauge bosons. Similarly there will be two physical charged scalars and the other two Goldstone bosons become part of the two massive charged gauge bosons. Hence, after the spontaneous symmetry breaking followed by the Higgs mechanism, the physical spectrum of the scalar sector consists of four neutral CP even scalars, two neutral CP odd scalars and two charged scalars (and their antiparticles).

C. Gauge Boson Sector
The gauge couplings corresponding to the gauge groups SU (2) L , SU (2) 2 and U (1) X are respectively represented by g 1 , g 2 and g X . Based on the two stages of symmetry breaking pattern we define two mixing angles φ and θ W in terms of which the gauge couplings are given by After the stage I of symmetry breaking SU (2) 2 × U (1) X → U (1) Y the gauge group becomes the Standard Model gauge group SU (2) L × U (1) Y . The SM hypercharge gauge coupling g Y for the gauge group U (1) Y is given by the relation 1 g 2 With the stage II of symmetry breaking SU (2) 1 × U (1) Y → U (1) em the electromagnetic gauge coupling constant e is defined by Here the angle θ W denotes the weak mixing angle in the SM.
At the end of the two stages of symmetry breaking the electromagnetic charge for any field in the model is defined by Q = T 1 3 + T 2 3 + Q X .
The gauge bosons corresponding to the gauge groups are denoted by : The mass square matrix for the charged gauge boson sector in the (W ± 1,µ , W ± 2,µ ) basis is given by Since the vacuum expectation value v 3 gives masses to the charged leptons, for simplicity we consider the situation where v 2 1 + v 2 2 >> v 2 3 and to have the stage II breaking at a higher scale we consider u >> v 1 , v 2 , v 3 . Based on this we define a small parameter which is given The mass eigenstates for the charged gauge boson sector in terms of the gauge eigenstates are given by with the mixing angle given up to order by the relation The mass eigenstate W denotes the observed SM W boson while the new W is heavy with mass at TeV scale. The mass squared matrix for the neutral gauge boson sector in the basis (W 3 1 , W 3 2 , X) is and the mass eigenstates are given by  Among the mass eigenstates, A denotes massless photon, Z denotes the observed neutral heavy SM weak gauge boson and Z , the heavier neutral gauge boson that has mass at TeV scale. Up to order the Z − Z mixing angle is given by cos θ zz 1 and sin θ zz sin φ cos 3 φ sin θ W .

III. INTERACTIONS OF VECTOR LIKE QUARKS WITH GAUGE BOSONS
The covariant derivative for a field determines its nature of interaction with the gauge bosons. To observe the vector-like nature of the interaction of exotic quarks with the SM gauge bosons we write those terms from the covariant derivative which contains neutral gauge bosons and that is given by By observing the term with Z µ from Eq. 18 we can conclude that the interaction strength of the left chiral fields xu 0 i L (xd 0 i L ) with Z boson differs from the interaction strength for the right chiral fields xu 0 i R (xd 0 i R ) by a term proportional to . This is because the left chiral and the right chiral fields do not have same T 2 3 value. Hence the interactions of the BSM quarks with the Z boson are vector-like in the limit → 0. The origin of the term is due to the Z − Z mixing. Since we are interested in the situation where u v and hence = v 2 we have identified the BSM exotic quarks (xu 0 i and xd 0 i ) as VLQs throughout the article. Since the exotic quarks are singlets under SU (2) L the interaction strength of the exotic quarks with the W boson will be very small for small values of W − W mixing angle.

IV. FCNC IN THE PRESENCE OF VECTOR LIKE QUARKS
In SM there is no flavour-changing neutral current (FCNC) interactions of quarks with the Z boson at the tree level because the quarks with same electric charge have universal charge assignments under the SM gauge group. But in models with VLQs this scenario of having universal charges under the gauge group of the model having the same electric charge breaks down. Hence the mixing between the SM quarks and the VLQs can generate tree level FCNC interactions for the SM quarks.
In our model this mixing is generated by the terms proportional to Y qxd ij , Y qxu ij and µ ij in the Yukawa Lagrangian in Eq. 4. For models with vector like quarks the possibility of restricting tree level FCNC interactions exists for special choice of mixing patterns between the quarks and VLQs [44]. It has been shown in [44] that if one linear combination of VLQs mix with only one SM quark mass eigenstate then there will be no Z-boson mediated FCNC interaction at tree level. This would imply that each VLQ will have a corresponding SM quark (mass eigenstate) with which it mixes. We use this formalism [44] and discuss the scenario in which the FCNC interactions vanish for our model.
The 6 × 6 dimensional mass matrices for the up-quark sector and for the down-quark sector are respectively given by The matrices Y qxu , Y qxd , Y xqxd , µ are 3 × 3 dimensional whose components are formed out of the Yukawa couplings in Eq. 4. The 3 × 3 matrices M u and M d are given by The quark gauge eigenstates ( U L/R , D L/R ) and the mass eigenstates (U L/R , D L/R ) are represented by The matrices M u and M d from Eq. 19 will be diagonalized by biunitary transformations and are given by where the S u L and S d L are 6 × 6 unitary matrices and can be represented by The matrices S u R and S d R can be obtained from Eq. 23 by replacing L → R. The matrices A, E, F, G are 3 × 3 dimensional and where E and F connect the SM quarks with the VLQs.
To avoid FCNC at the tree level we choose the mixing pattern such that the matrices for the left chiral sector take the form where then, for the left chiral down-type quarks in a similar way as above with up-types changed with down-type quarks : where The mixing matrices and the mass eigenstates for the right handed fields can be obtained by replacing L → R in Eqs. 24-28. Note that both Eq. 26 and its right handed counterpart show that in the absence of mixing between the SM quarks and the VLQs, the unitary matrices A u L and A u R are the matrices which diagonalize the mass matrix for the up-quark sector of the SM. The same can be concluded for the down-quark sector also.

A. CKM Matrix in presence of Vector Like Quarks
The interaction of the SM left-chiral gauge eigenstates with the W boson is given by Based on the interactions of the SM quark mass eigenstates with the W boson the CKM matrix is defined as V CKM L corresponds to the measured CKM matrix. It can be noted that in the presence of mixing between SM quarks and VLQs the matrix V CKM L is not unitary. The deviation from unitarity of the measured CKM matrix will put constraints on the mixing angles contained in the matrices C u L and C d L . Similarly the interaction term of the SM right-chiral quark gauge eigenstates with the W gauge boson is given by We define a right-handed CKM matrix given by The flavour diagonal nature can be seen by writing U 0 L γ µ U 0 L in terms of mass eigenstates by using the Eqs. 21-24 and is given by Since C u L is a diagonal matrix, the first term in the right hand side of the above equation is diagonal in the mass eigenstates u L , c L and t L . Similarly, with the equivalent form of Eq.34 for U 0 R , XU 0 L/R , D 0 L/R and XD 0 L/R , and since S u L is also a diagonal matrix, we find that there is no flavour-changing interactions of the SM quarks with the Z boson. Again, by expanding the interaction terms for Z in Eq. 18 we find that the Z also does not have any flavour-changing interactions with the SM quarks for the chosen mixing pattern.

C. FCNH interaction with Higgs Bosons
The unitary matrices in Eq. 22 diagonalize the quark mass matrices, i.e., where And From Eq. 35 the Yukawa couplings with the bi-doublet in terms of mixing matrices are given by Solving Eq. 38 for Y q and Y qC we get From the Yukawa Lagrangian in Eq. 4 the interaction of the neutral components of the bi-doublet Φ with the SM quarks in the gauge basis is given by Using Eq. 39 and Eqs. 22-28 the interactions of the scalars φ 0 1 and φ 0 2 with the SM up type quarks in the mass basis can be written as where the two orthogonal fields φ 0 − and φ 0 + are given by [52] φ 0 And v 2 ± = |v 1 | 2 ± |v 2 | 2 . Similarly the interaction terms for the SM down type quarks in the mass basis is given by From Eq. 41 and Eq. 43 it can be concluded that the interactions φ 0 − is flavour-diagonal but the interactions of φ 0 + is flavour-changing. φ 0 + interactions are flavour-changing because the matrix V CKM L which is the measured CKM matrix is not diagonal. Although the matrix V CKM R can be non-diagonal there is no experimental constraint which forces it to be nondiagonal and hence V CKM R can be taken to be diagonal by proper choice of Yukawa couplings.
In left-right symmetric model the field φ 0 − is always flavour-conserving in nature [52]. But in the 221 model we are discussing φ 0 − can also have flavour-violating interactions for general mixing patterns between VLQs and the SM quarks. It is the form of the mixing matrices in Eq. 24 and Eq. 27 which ensures that φ 0 − have flavour-conserving interactions. In general φ 0 + is not a mass eigenstate and when both v 1 and v 2 are nonzero all the neutral mass eigenstates will contain φ 0 + . And hence all of the neutral scalars have to be heavy to avoid constraints from FCNC interactions. Therefore when both v 1 and v 2 take nonzero values it will be impossible to get a light mass eigenstate at 125 GeV which do not have flavour violating interactions. This scenario has been extensively discussed in the context of left-right symmetric model in [53].
Hence to have a flavour-conserving Higgs at 125 GeV we made the choice v 2 = 0, which The VEV v 1 has been considered as a real parameter. As we will show it is possible to choose the parameters of the scalar potential in the model such that φ 0 2 do not mix with any other scalars and can be made heavy to avoid large neutral flavour-changing interactions. φ 0 1 will be a part of the observed 125 GeV Higgs . For v 2 = 0 the interactions of flavour conserving Higgs φ 0 1 with the SM type quarks are given by And the interactions for the flavour-violating Higgs φ 0 2 are given by To study the nature of interactions of the neutral components from the doublets H 1 and H 2 we list the Yukawa couplings in terms of the mixing matrices as below (for mixing matrices of the form taken in Eq. 24 and v 2 = 0) Lagrangian. These interactions are given by The superset sign (⊃) has been used in the above equations to highlight the terms containing only the fields χ 0 , χ 0 and the SM up-type mass eigenstate quarks (U L , U R ). From Eq. 48 it can be concluded that the scalars χ 0 and χ 0 do not have flavour changing interactions with the SM mass eigenstate quarks.
Based on the above discussions, we denote the field φ 0 2 as the FCNH (flavour-changing neutral Higgs) scalar and the fields φ 0 1 , χ 0 and χ 0 as the three non-FCNH scalars in the model. One linear combination of the three non-FCNH scalars will be the 125 GeV SM Higgs and the other linear combinations can lie at the sub-TeV scale.
The special cases for the matrices Y qxu and µ which will play an important role in the phenomenology of the VLQs are given by : and where i ∈ (u, c, t).
For simplicity we consider the scenario where the three matrices A d L , A u R and A d R are equal to 1. The choice for the Yukawa couplings which will lead to such a scenario can be made by using A d L = 1, A u R = 1 and A d R = 1 in Eq. 47. From Eq. 47 it can be observed that the complete determination of Yukawa couplings will depend on the SM quark masses, desired values of VLQ masses, desired values of mixing angles, VEVs and on the form of the matrix A u L . The measured CKM matrix will enter through the matrix A u L because, V. PHENOMENOLOGICAL ASPECTS So far we have discussed about the methodology to get rid of any tree level Z boson FCNC and to achieve a 125 GeV Higgs with no flavor changing interactions. Now we shall discuss some of the phenomenological implications of the model. To do that we will choose a representative mass spectrum for the exotic particles in our model and then discuss the possible collider signals which can be explored at the LHC.

A. Scalar and gauge boson mass spectrum
To relate the gauge eigenstates with the mass eigenstates for the scalar sector, we introduce three 4 × 4 matrices Z E , Z O and Z C for CP even sector, CP odd sector and charged sector respectively. The relations are given by Here h 1 , h 2 , h 3 , h 4 are CP even scalar mass eigenstates, A 1 and A 2 are CP odd scalar mass eigenstates and h + 1 , h + 2 are charged scalar mass eigenstates. G 1 , G 2 are neutral Goldstone bosons and G + 1 , G + 2 are the charged Goldstone bosons. As discussed in the previous section, the field φ 2 0 will have FCNH interactions while the fields φ 1 0 , χ 0 and χ 0 have no FCNH interactions. Therefore, any mass eigenstate formed out of the linear combinations of the three fields φ 1 0 , χ 0 and χ 0 can lie at sub-TeV scale. A typical example of the composition of the sub-TeV scalar states can be arranged through the following choice of the parameter values in our model: {M 2 , α 1 , α 2 , β 1 , β 2 , λ 2 , λ 3 , λ 4 , ρ 3 } = 0, u = 12 TeV, M 1 = −0.3 GeV, v 2 = 0, v 3 = 7 GeV, ρ 1 = 0.1, ρ 2 = 1, λ 1 = 0.13, β 3 = 1.4 and α 3 = 1.
For the above choice of parameters the mixing matrices are given by For the choice u = 12 TeV and for g 2 = g 1 , the mass values for W and Z are M W 4 TeV and M Z 4.7 TeV respectively, and both of them satisfies the current lower bounds obtained by the ATLAS collaboration [54,55]. Also the Z − Z mixing angle is small (θ zz 10 −4 ) and satisfies the constraint from the electroweak precision data [56]. Note that for v 2 = 0, the W − W mixing angle θ ww (see Eq. 14) is zero.

B. Production and decay of the Vector Like Quarks
As color triplets, the VLQs will be pair-produced at the LHC mostly through strong interactions. The pair production cross section at LHC with different center-of-mass energies and as a function of the mass of xu 3 has been shown in fig.1. We have used NN23LO1 [57] parton distribution function with default factorization and renormalization scale in Madgraph5_aMC@NLO for our estimates. for A 1 will be The possible final states that xu 3 can decay to are t Z, t h 1 , b W + , t h 2 , t A 1 and b h + 1 , since the scalars h 3 , h 4 , A 2 , h + 2 and the new gauge bosons Z , W are heavier compared to the VLQ xu 3 . For small mixing angles the "non-standard" decay modes (t h 2 , t A 1 , b h + 1 ) will mostly dominate over the standard decay modes (t Z, t h 1 , b W + ) because of the presence of direct Yukawa interaction term, Y qxu in the Lagrangian. The standard decay modes will The interaction terms including scalars are of the form φ K φ (C S φ + C P φ γ 5 ) xu 3 and including gauge bosons are of the form a µt K a γ µ (C V a + C A a γ 5 ) xu 3 . The interactions for the physical scalars can be obtained using the transformations given by Eq. 51. start dominating once Y qxu tends to zero. This feature is illustrated in Fig. 2 which shows the branching ratios for different decay modes for a 800 GeV xu 3 as function of mixing angles, where we have fixed sin θ t R = 10 −3 and varied sin θ t L accordingly. We have considered small mixing angles ( 10 −3 ) to avoid constraints from flavour sector and electroweak precision data [16,[38][39][40][41][42] and as an example, we have checked that the contribution of VLQs to the K −K oscillation parameter ∆m K is few orders of magnitude less compared to the SM value. We find that the non-standard decay modes dominate the interactions with the scalars h 2 , A 1 and h + 1 goes to zero. Consequently the decays of xu 3 to the SM particles enhances. In the next section we study the possible collider signatures for the scenario where the branching ratios for the VLQ xu 3 lie away from the standard mode dominated region such that after production xu 3 decays to one of the final states from t h 2 , t A 1 , b h + 1 . The collider signatures of xu 3 will eventually depend on the decay modes of the the scalars h 2 , A 1 and h + 1 . From Eq. 51 and Eq. 52 we note that the scalar h 2 is made up of a very small component (∼ 10 −2 ) of one of the real neutral part of the bi-doublet Higgs field (φ 0 1 r ) and a large component of the real neutral part of H 1 , i.e., χ 0 r . From the Yukawa terms in the Lagrangian it can be seen that H 1 gives mass to the charged leptons but there is no Yukawa interaction term involving SM quarks and H 1 . Hence the strength of the Yukawa interaction for the mass eigenstate h 2 with the SM quarks is negligible compared to the coupling strength with the leptons. Hence h 2 will mostly decay to leptons compared to the SM quarks. The same argument is also applicable for A 1 and h + 1 , because all of them are largely composed of H 1 . Note that the charged leptons get masses from the VEV of the SU (2) L doublet scalar H 1 which gets a small VEV, v 3 √ 2 ∼ 5 GeV. The Yukawa coupling strengths for the scalars h 2 , A 1 , h + 1 with the leptons from different generations follow the mass hierarchy and with more than 99% probability the scalar h 2 and A 1 will decay to τ + τ − whereas h + 1 will decay to τ + ν τ .

C. Benchmark points
For the collider analysis we have chosen three benchmark points based on xu 3 mass. The pair production cross section and the branching ratios for different benchmarks are given in table 3. For all the three cases the masses for the scalars h 2 , A 1 and h + 1 are kept fixed at 300 GeV, 300 GeV and 363 GeV respectively. Due to the smallness of the Yukawa couplings with the SM quarks, the scalars h 2 and A 2 can not be produced efficiently at the LHC via gluon fusion and the production cross sections of them are few tens of fb for 13 TeV center of mass energy. Hence, the experimental limits on on their masses are fairly weak. Note that the Yukawa couplings of these scalars with the VLQs are also very small and the VLQ loops contribute very less towards their production. The scenario is almost same as the leptonspecific two-Higgs doublet model where the limit on the massive states are of the order of 180-200 GeV [58]. Since the production cross-section falls off rapidly for higher masses we have used an integrated luminosity of 100 fb −1 for BP1 whereas 3000 fb −1 luminosity is used for the analysis of BP2 and BP3.
We would like to emphasise that the benchmark points chosen here are fairly general as xu 3 long as the mass of xu 3 is greater than the masses of the scalars h 2 , A 1 and h + 1 , such that the non-standard decay modes are kinematically allowed. From figure 2 it can be observed that the decay branchings depend mildly on the ratio of the mixing angles θ L and θ R , once we are away from the narrow peak region. Also the decay branchings of h 2 , A 1 and h + 1 to tau leptons depend on the yukawa couplings and are almost independent of the masses of the scalars.

VI. COLLIDER ANALYSIS
Now we consider the collider signatures of xu 3 for 13 TeV LHC in the scenario where the pair produced xu 3 will decay to the final states t h 2 , t A 1 , b h + 1 and the scalars h 2 , A 1 and h + 1 will further decay to tau leptons. Taking into account all possible decay chains of xu 3 , fig.3 shows all possible final states that can arise from the pair production of xu 3 . Since each final state contains at least two tau leptons and at least one b quark, we choose the final state with at least two τ -tagged jets, at least three non τ -tagged jets among which at least one is b-tagged, and at least one lepton (≥ 3j(1b) + ≥ 2τ + ≥ 1l) for the collider analysis.
The possible SM processes that can contribute as background to the above choice of final state are the following: • pp → tt + jets, • pp → tt l + l − + jets and pp → tt τ + τ − + jets, • pp → tt l + ν l + jets and pp → tt τ + ν τ + jets, • pp → ZZZ, Among all the above possible backgrounds the most dominant one is tt + jets. Although the cross section for the backgrounds W ± /Z + jets is large it is possible to get rid of this by a large / E T requirement which we have used in our analysis. The contribution of ZZZ will be negligible because of its small cross section. Hence we consider only the first three of the above list of backgrounds for the collider analysis in the context of 13 TeV LHC.
To study the collider phenomenology we implemented the model in the spectrumgenerator-generator SARAH [59]. The source code generated by SARAH for the spectrum generator SPheno [60] has been used in SPheno to study the spectrum of the model. The files generated by SARAH in the UFO format and the spectrum file generated by SPheno has been used in MadGraph 5 [61] for event generation for the signal. The background events have also been generated using Madgraph 5. For showering and hadronization we used Pythia 6 [62] interfaced in Madgraph 5. DELPHES 3 [63] within CMS environment has been used to take into account the detector effects and also for reconstruction of the final state objects. The anti-k T algorithm with cone size 0.5 have been used for the jet reconstruction. For the reconstruction of the jets, FastJet [64] embedded in DELPHES has been used.
MadAnalysis 5 [65] package has been used for the event-analysis using the event format ROOT and LHCO. The selection criteria for the final state objects in the reconstructed events are such that a non τ -tagged jet with p T (j) > 20 GeV and |η(j)| < 2.5 is considered in the event, an electron or a muon with p T (l) > 10 GeV and |η(l)| < 2.5 are considered in the event, a τ -tagged jet with p T (τ ) > 20 GeV and |η(τ )| < 2.5 is considered in the event. Note that here j denotes a non τ -tagged jet, τ denotes a τ -tagged jet. A non τ -tagged jet (j) is either a light jet or a b-tagged jet. The minimum angular separation between all final state objects satisfy ∆R > 0.4. The τ -tagging and mistagging efficiencies are incorporated in Delphes3 as reported by the ATLAS collaboration [66]. We operate our simulation on the Medium tag point for which the tagging efficiency of 1-prong (3-prong) τ decay is 70% (60%) and the corresponding mistagging rate is 1% (2%). For BP1 the production cross-section and different branching ratios are tabulated in table 3. For the background simulation we generated pp → tt events up to two additional jets at the leading order accuracy and used shower-K T matching scheme in Madgraph 5 to avoid the double counting between the partonic events and showered events. For the event analysis we used the cross section for 13 TeV at the NNLO accuracy for top quark pair production, i.e., 815.96 pb [67]. By following the same procedure we have generated both the ttl + l − and ttlν l events up to two additional jets at the leading order accuracy. The obtained leading order cross section at the parton level for ttl + l − + ≤ 2 jets is 96 fb for 13 TeV LHC and to accommodate the NLO effects we multiplied the cross section with a factor of 1.4 which is the NLO K-factor for ttZ. Similarly for ttlν l + ≤ 2 jets we multiply the leading order parton level cross section 166 fb for 13 TeV LHC with 1.4 which is the K-factor for ttW [61].
After the event selection we compare the phase space behavior of the signal events with the background and plot the normalized distributions of the transverse momentum (p T ) of the leading non τ -tagged jet and leading τ -tagged jet along with ( / E T ) in fig. 4. Due to the large mass separation between the VLQ and the scalars, the leading jet is expected to be quite hard as the figure shows. In addition, the τ which comes from the decay of the scalar which is around 300 GeV in mass is also quite hard in p T . Thus one can make a quite easy separation of the signal and background events using the distributions shown in 4. Based on the distributions and to further optimize the signal-to-background ratio, we apply the following kinematic cuts on the final state objects: With an integrated luminosity of 100 fb −1 the cut flow for the signal and background events is shown in table 4a. With the above cuts and with 100 fb −1 integrated luminosity, the statistical significance for BP1 is quite large (∼ 12σ). We have used 2((s + b) ln(1 + s b ) − s) to calculate the significance. Thus BP1 seems to be have VLQs in a mass range which would be very close to the current sensitivity of the LHC run if such a final state is analyzed for VLQs decaying in non-standard channels as in our model. For BP2 the branching ratios of xu 3 for the decay modes h 2 t, A 2 t and h + 2 t are around 25.5%, 25.5% and 49% respectively. The background events and their corresponding cross sections are same as in case of BP1 and we have used the same preselection criteria on the events (i.e. ≥ 3j(1b) + ≥ 2τ + ≥ 1l) for BP2. The differences between the mass of the VLQ xu 3 and the masses of scalars (h 2 , A 1 , h + 1 ) increase as we go higher in values of the mass of xu 3 while keeping the mass of the scalars fixed as before. It is worth pointing out here that even if the mass of the scalars are made larger, the decay probabilities of the VLQ do not change much. Therefore the event rates would remain the same, albeit the cut efficiencies would change due to new thresholds for the leading jet and tagged τ . Note that the b quark that will originate from the decay xu 3 → h + 1 b for a 1.5 TeV xu 3 will have a large p T compared to the b quark that originates from the decay of a 1 TeV xu 3 . Since the leading non τ -tagged jet is most likely the b jet coming from h + 1 b mode it will be in general harder in BP2 compared to BP1. Accordingly for BP2, we have applied the following selection criteria on the final state objects from the reconstructed events to optimize the significance : Notice after the cut on p T (j 1 ) to further improve the significance we have also applied cuts with higher values on p T (τ 1 ) and / E T compared to the scenario of BP1. For BP2 with  Finally for the last benchmark, we choose a very heavy mass of 2 TeV for the VLQ. Quite clearly the event rates would suffer from the very small production cross section and if we require two isolated τ -jets then the final events yield becomes extremely low even with an integrated luminosity of 3000 fb −1 . To counter the suppression due to small production cross section we modify our signal choice to a more inclusive channel given by : ≥ 3 non τ -tagged jets out of which one is b-tagged, at least one τ -tagged jet and at least one lepton in the final state.
As we go higher in the mass of xu 3 the probability for the jets and the tau leptons for the signal to have higher p T values is more compared to the backgrounds. Hence for BP3 with m xu 3 = 2 TeV to get a large statistics for the tt + jets background, we have generated pp → tt+2j events exclusively at the parton level for 13 TeV LHC with the following criteria : • For each event at least one top quark decays leptonically, because at the analysis level we have considered events with at least one lepton in the final state.
• All the jets and leptons satisfy |η| < 3.0 and the angular separation (∆R) between all pairs of final state particles are greater than 0.4 (except for leptons where they are separated from each other with minimum angular separation 0.2).
• All the final state objects satisfy p T > 10 GeV.
• The two leading jets in p T satisfy p T (j 1 ) > 250 GeV and p T (j 2 ) > 100 GeV.
With the above cuts the parton level cross section at the leading order accuracy for pp → tt + 2j with 13 TeV LHC is around 6.18 pb. The same events and cross sections as in case of BP1 has been used for the other two backgrounds ttl + l − + jets and ttlν l + jets. Note that with much stronger threshold requirements for the final state jets, we expect that the lesser-order processes involving tt and tt + 1j would not contribute much, where the extra jet comes from the showering. We then follow the usual procedure of using the Pythia showering and DELPHES 3 simulation to generate the final objects from the pp → tt + 2j process.
For our analysis, we further demand the following set of cuts on our final state events to improve the significance : Here the effective mass variable (M ef f ) is defined as the scalar sum of all the transverse momenta in an event and is given by The corresponding cut flow can be seen from the table 4c where we have achieved a signal significance of 2σ for BP3.
We plot the significance as a function of luminosity in figure 5 for all the benchmark points. It is evident that for BP1 significance of 5σ can be achieved with 17.3 fb −1 of data, hence with the already existing datasets of 36.1 fb −1 of data would be sensitive to this mass range. With high luminosity data the BP2 can be discovered while it is only possible to exclude a 2 TeV VLQ at 2σ. In fact, an upgrade in LHC energies and higher luminosities would be required to access VLQ signals in such models beyond VLQ mass of ∼ 1.8 TeV.
Definitive improvements in the sensitivity is also expected with more sophisticated analysis using boosted studies for the τ states coming from the decay of the heavily boosted scalars.

(A4)
Appendix B: Mass matrices for the scalar Sector

CP odd scalars
The components of the CP odd scalar sector mass square matrix(M 2 P ) in the (φ 0