Light charged Higgs boson with dominant decay to quarks and its search at LHC and future colliders

The possibility of a light charged Higgs boson $H^\pm$ that decays predominantly to quarks ($cs$ and/or $cb$) and with a mass in the range 80 GeV $\le m_{H^\pm} \le 90$ GeV is studied in the context of Three-Higgs-Doublet Models (3HDMs). At present the Large Hadron Collider (LHC) has little sensitivity to this scenario, and currently the best constraints are from LEP2 and Tevatron searches. The branching ratio of $H^\pm\to cb$ can be dominant in two of the five types of 3HDM, and we determine the parameter space where this occurs. The decay $H^\pm\to cb$ has recently been searched for at the LHC for the first time, and with increased integrated luminosity one would expect sensitivity to the region 80 GeV $\le m_{H^\pm} \le 90$ GeV due to the smaller backgrounds with respect to $H^\pm\to cs$ decays.


I. INTRODUCTION
In 2012, the ATLAS and CMS Collaborations of the Large Hadron Collider (LHC) announced the discovery of a new particle with a mass of around 125 GeV [1,2]. The current measurements of its properties are in very good agreement (within experimental error) with those of the Higgs boson of the standard model (SM), and measurements suggest that it has a spin of zero. Five decay channels (γγ, ZZ, WW, ττ, and bb) have now been observed with a statistical significance of greater than 5σ (e.g., see [3]). The measured branching ratios (BRs) are in agreement with those predicted for the SM Higgs boson. Moreover, the main four production mechanisms (gluon-gluon fusion, vector boson (W=Z) fusion, associated production with a vector boson, and associated production with top quarks) have been observed, with no significant deviation from the cross sections of the SM Higgs boson.
The simplest assumption is that the observed 125 GeV boson is the (solitary) Higgs boson of the SM. However, it is possible that it is the first scalar to be discovered from a nonminimal Higgs sector, which contains additional scalar isospin doublets or higher representations such as scalar isospin triplets. In such a scenario, future measurements of the BRs of the 125 GeV boson could show deviations from those of the SM Higgs boson. There is also the possibility of discovering additional neutral scalars, or physical charged scalars (H AE ). In the context of a two-Higgs-doublet model (2HDM), the lack of observation of an H AE at the LHC rules out parameter space of tan β (from the Yukawa coupling) and m H AE , where tan β ¼ v 2 =v 1 , and v 1 and v 2 are the vacuum expectation values (VEVs) of the two Higgs doublets, respectively (for reviews, see e.g., [4,5]).
In a three-Higgs-doublet model (3HDM), the Yukawa couplings of the two charged scalars depend on the four free parameters (tan β, tan γ, θ, and δ) of the unitary matrix that rotates the charged scalar fields in the weak eigenbasis to the physical charged scalar fields. As pointed out in previous works [6][7][8][9][10], in a 3HDM there is a phenomenologically attractive possibility of an H AE being light (m H AE < m t ) and having a large BR for the decay channel H AE → cb, a scenario which would not be expected in a 2HDM with natural flavor conservation (NFC) [11] due to the stringent bounds from the decay b → sγ. A search for H AE → cb decays originating from t → H AE b has recently been performed at the LHC [12]. The only study of the BRs of the two H AE s in 3HDMs (with NFC) as functions of the above four parameters was in Ref. [10]. However, this work did not fully study the dependence of the BRs on the parameter space. We perform the first comprehensive study of the BRs of the lightest H AE in the various 3HDMs (with NFC) as a function of the four parameters. We also study the dependence of the product BRðt → H AE bÞ × BRðH AE → cbÞ, which gives the number of events in the search in [12]. We give emphasis to the scenario of 80 GeV ≤ m H AE ≤ 90 GeV and a large BRðH AE → cs=cbÞ for which detection is currently challenging at the LHC, but prospects with the anticipated integrated luminosities are more promising.
This work is organized as follows. In Sec. II, we give an introduction to the phenomenology of the lightest H AE in 3HDMs with NFC. In Sec. III, the searches for H AE at past and present colliders that provide sensitivity to the region 80 GeV < m H AE < 90 GeV are summarized. In Sec. IV, our results are presented, and conclusions are contained in Sec. V.

II. THE THREE-HIGGS-DOUBLET MODEL (3HDM) WITH NFC
In this section, the fermionic couplings of the lightest H AE in the 3HDM as a function of the parameters of the scalar potential are presented. The constraints on the fermionic couplings are summarized, and explicit formulae for the BRs of the decay of H AE to fermions are given.

A. Fermionic couplings of H AE in the 3HDM
In a 2HDM, the Lagrangian that corresponds to the interactions of H AE with the fermions (the Yukawa couplings) can be written as follows: Here uðdÞ refers to the up(down)-type quarks, and l refers to the electron, muon and tau. The imposition of NFC, which eliminates tree-level flavor changing neutral currents (FCNCs) that are mediated by scalars, is achieved by requiring that the Yukawa couplings are invariant under certain discrete symmetries (e.g., see [10] for the charge assignments of the scalar and fermion fields under the discrete symmetries). This leads to four distinct 2HDMs [13]: Type I, Type II, lepton-specific, and flipped. In Table I, the couplings X, Y, and Z in the four distinct 2HDMs are given. The Lagrangian in Eq. (1) also applies to the lightest H AE of a 3HDM, with the X, Y, and Z couplings being functions of four parameters of a unitary matrix U. This matrix U connects the charged scalar fields in the weak eigenbasis (ϕ AE 1 , ϕ AE 2 , ϕ AE 3 ) with the physical scalar fields (H AE 1 , H AE 2 ) and the charged Goldstone boson G AE as follows: We take H AE 1 as the lighter of the two charged Higgs bosons, and from now on it is referred to as H AE with the following couplings [14]: The values of d, u, and l in these matrix elements are given in Table II and depend on which of the five distinct 3HDMs is under consideration. Taking d ¼ 1, u ¼ 2, and l ¼ 3 means that the down-type quarks receive their mass from the vacuum expectation value v 1 , the up-type quarks from v 2 , and the charged leptons from v 3 (this choice is called the "democratic 3HDM"). The other possible choices of d, u, and l in a 3HDM are given the same names as the four types of 2HDM. The couplings of the H AE 2 (i.e., the heavier charged scalar) are obtained from Eq. (3) by making the replacement 2 → 3 in the numerators of X, Y, and Z. We will not study these couplings for H AE 2 because our focus will be on H AE in the range 80 GeV < m H AE < 90 GeV.
The matrix U can be written explicitly as a function of four parameters tan β, tan γ, θ, and δ, where and v 1 , v 2 , and v 3 are the VEVs. The angle θ and phase δ can be written explicitly as functions of several parameters in the scalar potential [14]. The explicit form of U is: Here s and c denote the sine or cosine of the respective angle. Hence, the functional forms of X, Y, and Z in a 3HDM depend on four parameters. This is in contrast to the analogous couplings in the 2HDM for which tan β is the only free coupling parameter.
B. Constraints on the couplings X, Y, and Z The couplings X, Y, and Z (and their combinations) are constrained from various low-energy processes. Detailed studies in the context of the Aligned 2HDM (for which the couplings of H AE are also given by X, Y, and Z) can be found in Refs. [15,16]. These constraints can be applied to the lightest H AE of a 3HDM provided that the contribution to a given process from the H AE 2 is considerably smaller (e.g., if m H AE 2 ≫ m H AE ). In this work, we assume that any contribution from H AE 2 is sub-dominant and can be neglected to a good approximation. We summarize here the bounds (which are also summarized in [14]) that we will use in our numerical analysis.
The coupling Y is constrained from the process Z → bb from LEP data. For m H AE around 100 GeV (on which we focus) the constraint is roughly jYj < 1 (assuming jXj ≤ 50, so that the dominant contribution is from the Y coupling). The coupling X is also constrained from Z → bb, but the constraints from this process are weaker than those from t → H AE b (which will be studied later in this work).
From the rare decay b → sγ a constraint on the combination ReðXY Ã Þ is given by This constraint was derived in [16] for m H AE ¼ 100 GeV, and is an approximation for the case when i) the contribution from jYj 2 can be neglected (which is a fairly good approximation because jYj < 1) and ii) ImðXY Ã Þ is small (which is a good approximation, as shown shortly below). Detailed constraints on the H AE contribution to b → sγ in the Aligned 2HDM without this approximation can be found in [15]. Other works are usually in the context of the 2HDM with NFC [17][18][19][20][21][22].
In a 3HDM, one would have contributions to b → sγ from both H AE and H AE 2 . The only study of the prediction for BRðb → sγÞ in 3HDMs to next-to-leading order accuracy is in [10]. It was shown there that there exists parameter space for which H AE can be of the order of 80 GeV even for Type II and flipped structures (which would not be possible in the 2HDM with these structures). This is due to the additional presence of H AE 2 and the larger number of parameters in the couplings X and Y with respect to the 2HDM with NFC. In our numerical analysis for the BRs of H AE , we will use the allowed range given in Eq. (6) in order to find the regions of tan β, tan γ, θ, and δ that satisfy the b → sγ constraint. Although Eq. (6) neglects the contribution from H AE 2 we will take Eq. (6) as being representative of the b → sγ constraint in 3HDMs. The true region allowed by b → sγ (to next-to leading order accuracy, as done in [10]) would presumably be shifted somewhat from the regions allowed by Eq. (6). We argue later that we would not expect this to significantly alter our qualitative results.
The electric dipole moment of the neutron gives the following constraint on ImðXY Ã Þ [16]: This bound is for m H AE ¼ 100 GeV and is an order-ofmagnitude estimate. There are also the constraints jZj ≤ 40 and jXZj ≤ 1080, both for m H AE ¼ 100 GeV. In our numerical analysis, we will respect all these constraints.

C. The Branching Ratios of H AE
We will only consider the decays of H AE to fermions. If there exists a neutral scalar (e.g., a CP-even h 0 or a CP-odd A 0 ) that is lighter than H AE then the decay channel H AE → h 0 W Ã and/or H AE → A 0 W Ã would be open and can be sizeable (or even dominant) [9,[23][24][25][26][27][28][29]. We assume that these decays are negligible, and this is most easily achieved by taking m A 0 ; m h 0 > m H AE . In a 3HDM, the expressions for the partial widths of the decay modes to fermions of H AE are: In the expression for ΓðH AE → udÞ, the running quark masses should be evaluated at the scale of m H AE , and there are QCD vertex corrections which multiply the partial widths by (1 þ 17α s =ð3πÞ). A study of the BRs as a function of jXj, jYj, and jZj was first given in [7] and more recently in [9]. For jXj ≫ jYj, jZj the decay channel BRðH AE → cbÞ can dominate (which was first mentioned in [6]), reaching a maximum of ∼80%. In contrast, in a 2HDM with NFC the only model which contains a parameter space for a large BRðH AE → cbÞ with m H AE < m t is the flipped model (a possibility mentioned in [6,7] and studied in more detail in [30,31]), However, in this case the b → sγ constraint would require m H AE > 500 GeV [22] for which H AE → tb would dominate.
The first study of the dependence of the BRs of H AE in 3HDMs in terms of the parameters tan β, tan γ, θ, and δ was given in [10]. However, this work did not fully study the dependence of the BRs on the parameter space (i.e., δ ¼ 0, θ ¼ −π=4, and tan β ¼ 2ð5Þ was taken as a representative choice), and showed the BRs as a function of tan γ only. Moreover, in [10] the dependence of the BRs on the model parameters was carried out in the Higgs basis, and so the parameters tan β, tan γ, θ, and δ used in that work are not equivalent to the corresponding parameters in this work.
We now briefly mention other models in which a large BRðH AE → cbÞ is possible, although in this work we will just study the 3HDMs with NFC. The X, Y, and Z couplings of H AE in the Aligned 2HDM [32] (which does not have NFC, but instead eliminates scalar FCNCs at tree level by taking certain Yukawa matrices to be proportional to each other) are functions of five parameters. Consequently, jXj ≫ jYj; jZj can be realized and a large BRðH AE → cbÞ is possible [9]. In the 2HDM (Type III), in which fermions receive their masses from both VEVs (and scalar FCNCs are present at tree level), the Yukawa couplings of H AE depend on more parameters than in the Aligned 2HDM and thus a large BRðH AE → cbÞ can be obtained [33]. Similar comments apply to a four-Higgsdoublet model [14]. In models for which X, Y, and Z depend on several parameters, one expects some parameter space for a large BRðH AE → cbÞ for m H AE < m t , while satisfying the b → sγ constraint.
We focus on the scenario of H AE being lighter than the top quark. There have been searches for H AE in the region 80 GeV ≤ m H AE ≤ 90 GeV at LEP2, Tevatron and the LHC. However, the sensitivity to this mass region is often inferior to that for 90 GeV ≤ m H AE ≤ 160 GeV because of the large backgrounds from W decays. We pay particular attention to the region of 80 GeV ≤ m H AE ≤ 90 GeV, and in the following we discuss the searches for m H AE < m t at each of these colliders.

A. Tevatron searches
At the Fermilab Tevatron the production mechanism is pp → tt, where one top quark decays conventionally via t → Wb and the other top quark decays via t → H AE b. Taking jV tb j ¼ 1 one has the following expressions for the decays of a top quark to a W boson or an H AE : As can be seen from the above equations the BRðt → H AE bÞ depends on the magnitude of jXj and jYj. As discussed earlier, the BRs of H AE depend on the relative values of jXj; jYj and jZj. The search by the D0 Collaboration in [34] with 1 fb −1 of data obtained the following limit in the region 80 GeV ≤ m H AE ≤ 90 GeV: In the search strategy in [34], the presence of a large BRðH AE → cs=cbÞ in the decay t → H AE b would lead to a depletion in the expected number of events in the l þ jets, ll and lτ channels (l ¼ e or μ) compared to that expected from tt → W þ W − bb. Importantly, this "disappearance" search has sensitivity to the region 80 GeV ≤ m H AE ≤ 90 GeV and is thus an effective strategy when BRðH AE → cs=cbÞ is large and m H AE lies in the above region. The CDF Collaboration (with 2.2 fb −1 ) used a different search strategy [35] in which the signature of H AE → cs was searched for as a peak at m H AE in the invariant mass distribution of the quarks that it decays to (i.e., an "appearance" search for H AE → cs). This technique provides limits on BRðt → H AE bÞ that are competitive with those in [34] for values of m H AE that are not in the region 80 GeV ≤ m H AE ≤ 90 GeV. However the search provides no constraints for 80 GeV ≤ m H AE ≤ 90 GeV because the background from W → qq is too large. Up to now the LHC has only carried out appearance searches for H AE → cs=cb (see below).

B. LHC searches
The production mechanism at the LHC is pp → tt, where one top quark decays via t → H AE b (i.e., the same mechanism at the parton level as at the Tevatron). The LHC is expected to have accumulated around 150 fb −1 of integrated luminosity at ffiffi ffi s p ¼ 13 TeV by the end of the year 2018, at which point long shut down 2 will commence.
Various searches for the decay t → H AE b have been carried out at the LHC, and are summarized in Table III.

Decay H AE → τν
For the decay H AE → τν there are four basic signatures, which arise from the leptonic and hadronic decays of H AE and W AE . Searches for three of these signatures have been carried out with the 7 TeV data [36,39,40], which were then combined to give a limit on the product BRðt → H AE bÞ × BRðH AE → τνÞ for a given m H AE . Note that ATLAS used two different search strategies [39,40] that give comparable sensitivity. In [36,39,40], the limit is roughly ≥ 4% for m H AE ¼ 90 GeV, which strengthens with increasing m H AE to ≥ 1% for m H AE ¼ 160 GeV. Only the CMS search [36] presented limits (≥ 4%) for the mass range 80 GeV ≤ m H AE ≤ 90 GeV.
In the searches for H AE → τν with the 8 TeV data [41,43], both the τ and the W boson from t → W AE b decay were taken to decay hadronically. This signature (of the four) offers the greatest sensitivity at present. The transverse mass of H AE is calculated from its decay products of hadrons and missing energy. Both the ATLAS and CMS searches presented limits for the mass range 80 GeV ≤ m H AE ≤ 90 GeV. Limits on the product BRðt → H AE bÞ × BRðH AE → τνÞ were obtained, being around ≥ 1% for m H AE ¼ 80 GeV and strengthening with increasing m H AE to ≥ 0.2% for m H AE ¼ 160 GeV.
The CMS search [37] with 13 TeV data and 13 fb −1 also used the hadronic decay of the τ from H AE → τν, and selected the hadronic decay of the W AE . Similar limits to those in [41,43] were obtained, but are slightly weaker for the region 80 GeV ≤ m H AE ≤ 90 GeV. Recently a CMS search was carried out with 13 TeV data and 36 fb −1 [45], which combined separate searches for three of the four basic signatures (the case where both the W and τ decay leptonically was not searched for). Significantly improved limits on BRðt → H AE bÞ × BRðH AE → τνÞ were obtained, ranging from ≥ 0.36% for m H AE ¼ 80 GeV to ≥ 0.08% for m H AE ¼ 160 GeV.
There has been a search with the 13 TeV data [44] from the ATLAS Collaboration using 36 fb −1 , with limits similar to those in [45]. In contrast to the ATLAS search with 8 TeV data [41], both the leptonic and hadronic decays of the W AE boson were considered (the τ is still taken to decay hadronically). No limits are presented for the region 80 GeV ≤ m H AE ≤ 90 GeV, but the sensitivity to m H AE > 90 GeV has improved by a factor of approximately 5 to 10 e.g., for m H AE ¼ 90 GeV the limit on BRðt → H AE bÞ × BRðH AE → τνÞ is ≥ 0.3%, and with the 8 TeV data it is ≥ 1.2%.

Decay H AE → cs=cb
ATLAS carried out a search for H AE → cs [38] with 5 fb −1 of data at 7 TeV, while CMS [42] carried out a search for H AE → cs using 20 fb −1 of data at 8 TeV. The W boson is taken to decay leptonically. Two tagged b-quarks are required (which arise from the decay of the t-quarks), and the invariant mass distribution of the two quarks that are not b-tagged (i.e., the c and s quarks that originate from H AE ) is plotted. The signature of H AE would be a peak at m H AE in this invariant mass distribution. Limits on the product BRðt → H AE bÞ × BRðH AE → csÞ) are obtained, which range from around ≥ 5% for m H AE ¼ 90 GeV to 2% for m H AE ¼ 160 GeV. Note that these limits are weaker than those for H AE → τν decay for a given m H AE . In the invariant mass distribution, the dominant background from W → qq decays gives rise to a peak around 80 GeV. Hence, the expected sensitivity starts to weaken significantly with decreasing m H AE in the region 90 GeV ≤ m H AE ≤ 100 GeV, and there are no limits for the region 80 GeV ≤ m H AE ≤ 90 GeV.
CMS carried out a search [12] for H AE → cb decays (assuming a branching ratio of 100%) with the leptonic decay of W. Signal events will have three b-quarks, although one (or more) might not be tagged as a b-quark. Two event categories were defined: i) 3b þ e AE , and ii) 3b þ μ AE . A fitting procedure was carried out in order to correctly identify the tagged b-quark that arises from H AE → cb, which is then used (together with the non-btagged c quark) in the invariant mass distribution of H AE . Due to BRðW → cbÞ being very small, the background to H AE → cb decays is much smaller than that for H AE → cs.  future searches (e.g., with 150 −1 fb and ffiffi ffi s p ¼ 13 TeV) will be able to set limits on BRðt → H AE bÞ × BRðH AE → cbÞ in the region 80 GeV ≤ m H AE ≤ 90 GeV. Eventually, one would also expect some sensitivity in this region for the search with the 2b signature (which is sensitive to H AE → cs=cb decays) with 150 −1 fb and above. However, the limits would (most likely) be inferior to those in the 3b channel for a given luminosity. As mentioned earlier, the Tevatron strategy of a disappearance search for H AE → cs=cb has not yet been attempted at the LHC. A dedicated disappearance search at the LHC would be likely to improve on the Tevatron limit on BRðt → H AE bÞ × BRðH AE → cs=cbÞ of 20% [34] for 80 GeV ≤ m H AE ≤ 90 GeV. However, we are not aware of any LHC simulations, and so at present, it is not clear whether or not this strategy could give a sensitivity that is competitive with that for the appearance searches.

C. LEP2 searches and future e + e − colliders
The production mechanism at LEP2 was e þ e − → H þ H − . An important difference with the searches for H AE at hadron colliders is that the couplings X, Y, Z do not appear in the production cross section for e þ e − → H þ H − , which is instead a function of just one unknown parameter m H AE. Hence, this production mechanism at e þ e − colliders can produce H AE even with very small values of X, Y, Z, provided that 2m H AE < ffiffi ffi s p . The LEP working group combined the separate searches from the four LEP experiments [46]. These searches were carried out at energies in the range ffiffi ffi s p ¼ 183 GeV to ffiffi ffi s p ¼ 209 GeV, and with a total combined integrated luminosity of 2.6 fb −1 . In the searches for the fermionic decay modes of H AE , it is assumed that BRðH AE → csÞþ BRðH AE → τνÞ ¼ 1, but the actual experimental search for H AE → cs would be also be sensitive to H AE → cb and other light flavors of quark. Dedicated searches for the decay mode H AE → A 0 W Ã were also carried out in [46], but in this work we are assuming that this channel is absent or very suppressed. From the search for fermionic decays the excluded region at 95% confidence level (CL) in the plane ½m H AE ; BRðH AE → τνÞ is shown. For m H AE < 80 GeV the whole range 0 ≤ BRðH AE → τνÞ ≤ 100% is excluded. For 80 GeV ≤ m H AE < 90 GeV, most of the region is not excluded for BRðH AE → τνÞ < 80% (i.e., BRðH AE → csÞ > 20%). Notably, there is an excess of events of greater than 2σ significance around the point m H AE ¼ 89 GeV, BRðH AE → csÞ ¼ 65% and BRðH AE → τνÞ ¼ 35%, which could be readily accommodated in a 3HDM with appropriate choices of X, Y and Z. As mentioned in our earlier work [10] an excess like this is an example of a possible signal for H AE that was just out of the range of LEP2. Such an excess, if genuine, could be observed at the LHC provided that the values of jXj and jYj are large enough to ensure enough events of t → H AE b at a given integrated luminosity. Future LHC searches in the τν channel, which currently have sensitivity to the region 80 GeV ≤ m H AE ≤ 90 GeV, could then observe such an H AE . One could also expect a signal in the H AE → cs=cb channel provided that sensitivity to the region 80 GeV ≤ m H AE ≤ 90 GeV is obtained. If jXj and jYj are sufficiently small then such an H AE would escape detection at the LHC, but could be observed at future e þ e − colliders (see below).
The possibility of a future circular e þ e − collider operating at a variety of energies from ffiffi ffi s p ¼ m Z to ffiffi ffi s p ¼ 2m t is being discussed (FCC-ee at CERN and CEPC in China), and a future e þ e − Linear Collider (ILC) would also take data in this energy range (and higher energies). If such machines are approved, the earliest starting date of operation for CEPC (FCC-ee) would be the year 2030 (2040), with the ILC possibly starting between these two dates. The choice of ffiffi ffi s p ¼ 240 GeV would be optimal for detailed studies of the discovered 125 GeV neutral boson. This energy would also enable pair production of H AE up to a mass of 120 GeV. The integrated luminosity with ffiffi ffi s p ¼ 240 GeV at all three colliders is expected to be of the order of a few ab −1 , which is three orders of magnitude greater than the integrated luminosity (2.6 fb −1 ) used in the combined LEP search in [46]. Hence, an H AE with a mass in the region 80 GeV ≤ m H AE < 90 GeV would be discovered for any value of BRðH AE → cs=cbÞ, with a signal in at least one of the three channels H þ H − → jjjj, jjτν, τντν (where j signifies quarks lighter than the t quark). As mentioned earlier, the production mechanism e þ e − → H þ H − does not depend on the couplings to fermions. Hence, an H AE with 2m H AE < ffiffi ffi s p that escaped detection at the LHC due to small values of X, Y, and Z would be discovered at the above e þ e − colliders.

IV. RESULTS
We vary the four input parameters that determine X, Y, and Z in the following ranges (see e.g., [14]): We have checked that the phenomenological constraints on jXj, jYj, jZj, and jXZj from Sec. II B are respected but we do not show explicit plots for these parameters. The constraints on ReðXY Ã Þ and ImðXY Ã Þ rule out significant regions of parameter space, and these will be shown in specific plots. Taking δ ¼ 0 leads to real values for X, Y, and Z, and so in this case the constraint on ImðXY Ã Þ will be automatically respected. We only consider m H AE < m t , and results will be presented for the cases of m In our numerical analysis, we are only concerned with the four parameters in Eq. (12) and m H AE . These comprise five of the sixteen 1 free parameters in the scalar potential of the 3HDM [14]. There are theoretical constraints on these sixteen parameters from requiring the stability of the vacuum, the absence of charge breaking minima, and compliance with unitarity of scattering processes, etc. Such constraints are well known in the 2HDM (e.g., see [47] for a recent study) and have been discussed for the scalar potential of the 3HDM in [48,49].
In our analysis, we do not impose these constraints because they would only rule out certain regions of the parameter space of sixteen variables. As mentioned earlier, the phenomenology in the charged Higgs sector depends on only five parameters (which we take as unconstrained parameters in the above ranges). We assume that the freedom in the remaining eleven parameters can be used to comply with the above theoretical constraints while allowing the five parameters in the charged Higgs sector to vary in the above ranges. To justify this approach we note that the analogous constraints on the scalar potential in 2HDMs do not restrict the allowed ranges of the two parameters in the charged Higgs sector (m H AE and tan β) due to the freedom in the remaining four parameters (for the case of a 2HDM scalar potential with only soft breaking terms of a Z 2 symmetry). It is experimental data from processes involving H AE that constrain the ranges of the parameters of the charged Higgs sector in a 2HDM, and we carry this conclusion across to the charged Higgs sector of the 3HDM.
In the left panel of Fig. 1, we show contours of BRðt → H AE bÞ × BRðH AE → cbÞ in the plane [jXj, jYj] for jZj ¼ 0.1 and m H AE ¼ 130 GeV. This is an update of a figure in [9] in which the contours have been chosen to reflect the current and future sensitivity of the LHC. The region consistent with b → sγ lies below the curves of jXY Ã j ≤ 0.7 or jXY Ã j ≤ 1.1, depending on the sign of ReðXY Ã Þ in Eq. (6). In this figure, we take jXj and jYj as independent parameters and thus we do not consider them to be functions of the four parameters in Eq. (12) as in a 3HDM. As mentioned at the end of Sec. II C, in models such as the aligned 2HDM and a 4HDM the parameters X, Y, and Z would depend on more than four parameters. The results in the left panel of Fig. 1 are a model independent approach in which the allowed region of [jXj, jYj] (for a given jZj) are shown. For the chosen value of m H AE ¼ 130 GeV the current limit on BRðt → H AE bÞ × BRðH AE → cbÞ is ≤ 0.005 [12]. It can be seen from the left panel of Fig. 1 that the current limit is ruling out parameter space that is permitted by b → sγ. The contour with 0.001 will hopefully be approached with 150 fb −1 at ffiffi ffi s p ¼ 13 TeV, and such a search would further probe parameter space of [jXj, jYj], for a given jZj, that is still allowed by b → sγ. In the right panel of Fig. 1, we show contours of BRðt → H AE bÞ × BRðH AE → τνÞ in the plane [jYj, jZj] for jXj ¼ 5. This is also a model independent approach, and such a plot was not shown in [9]. In this case, the region allowed by b → sγ lies to the left of the perpendicular lines. For the chosen value of m H AE ¼ 130 GeV the current limit on BRðt → H AE bÞ × BRðH AE → τνÞ is ≤ 0.001 [44]. It can be seen from the right panel of Fig. 1 that the current limit is ruling out large regions of parameter space that is permitted by b → sγ. The contours with 0.005 and below will hopefully be approached with 150 fb −1 at ffiffi ffi s p ¼ 13 TeV, and such a search would further probe parameter space of [jYj, jZj], for a given jXj, that is still allowed by b → sγ. We now show our results for the flipped 3HDM and the democratic 3HDM. In the other 3HDMs (type I, type II, and Lepton-specific), we have checked that a large BRðH AE → cbÞ is not possible, and the maximum value is typically of the order of a few percent. In Fig. 2, we consider the flipped 3HDM with θ ¼ −π=3, δ ¼ 0, and m H AE ¼ 85 GeV. In the left panel of Fig. 2, we show contours of BRðH AE → cbÞ in the plane ½tan γ; tan β. It is evident that for tan γ ≥ 5 and tan β ≥ 5 one has BRðH AE → cbÞ ≥ 60%, and for tan γ ≥ 10 and tan β ≥ 10 the maximum value of around 80% is obtained. However, not all of this parameter space of ½tan γ; tan β survives the constraint from b → sγ. This can be seen in the right panel of Fig. 2 in which we show contours of ReðXY Ã Þ in the plane ½tan γ; tan β. The allowed parameter space lies below the contour of ReðXY Ã Þ ¼ 0.7, and roughly corresponds to the parameter space of tan γ > tan β. By comparing the left and right panels it is clear that a large parameter space for a dominant BRðH AE → cbÞ ≥ 60% (left panel) survives the b → sγ constraint (right panel). Taking a nonzero value of δ would only lead to slight modifications of BRðH AE → cbÞ, but would change the regions allowed by b → sγ due to X and Y both gaining an imaginary part. For δ ¼ 0 the constraint in Eq. (7) from the electric dipole moment of the neutron is automatically satisfied. For δ ≠ 0 this latter constraint would rule out parameter space, and we will consider this scenario later for the democratic 3HDM. In Fig. 3, we consider the flipped 3HDM with m H AE ¼ 85 GeV but now with tan β ¼ 10 and δ ¼ 0. In the left panel of Fig. 3, we show contours of BRðH AE → cbÞ in the plane ½tan γ; θ. In the right panel of Fig. 3, we show contours of ReðXY Ã Þ in the plane ½tan γ; θ. There is a large parameter space for a dominant BRðH AE → cbÞ which corresponds to large values of tan γ and less negative values of θ. In the right panel of Fig. 3, the parameter space allowed by b → sγ lies below the contour of ReðXY Ã Þ ¼ 0.7, and thus a large parameter space for a dominant BRðH AE → cbÞ ≥ 60% (left panel) survives the b → sγ constraint (right panel). In summary, from the results in Figs. 2 and 3, it is clear that a large part of the ½tan γ, tan β, θ parameter space (with δ ¼ 0) gives rise to a dominant BRðH AE → cbÞ while complying with constraints from b → sγ. As mentioned earlier, we consider the right panels of Figs. 2 and 3 to be representative of the true constraints on the planes ½tan γ; tan β and ½tan γ; θ from b → sγ. We expect that the true excluded region would be shifted somewhat from the excluded regions in Figs. 2 and 3, but it would not increase significantly in area. Given the large parameter space for a dominant BRðH AE → cbÞ in the flipped 3HDM we expect a sizeable region of large BR to survive. Taking a nonzero value of δ would only lead to slight modifications of the above plots for BRðH AE → cbÞ, but would have an effect on the plot for ReðXY Ã Þ. We will illustrate this when we consider the democratic 3HDM below.
In Fig. 4, we take the input parameters of Fig. 2 for the flipped 3HDM. In the left panel, we plot contours of BRðt → H AE bÞ × BRðH AE → cbÞ in the plane ½tan β; tan γ. This is the product that is being constrained by the CMS search at the LHC using three b-tags [12]. However, for m H AE ¼ 85 GeV [which is used in the Fig. 4] there is no limit on BRðt → H AE bÞ × BRðH AE → cbÞ from the LHC. The only limit is ≤ 20% from the Tevatron [34], using a strategy that was sensitive to any quark decay mode of H AE . We plot contours of BRðt → H AE bÞ × BRðH AE → cbÞ with values of 0.2 to 0.002. The region of the ½tan β; tan γ plane that is above the contour of 0.2 is ruled out, while the region below corresponds to a potential discovery of such an H AE . It is hoped that future searches of the LHC with ffiffi ffi s p ¼ 13 TeV and 150 fb −1 (or more) of data will have sensitivity to BRðt → H AE bÞ × BRðH AE → cbÞ of 0.02 or below. In the right panel of Fig. 4, contours of BRðt → H AE bÞ × BR½ðH AE → cbÞ þ BRðH AE → csÞ are plotted with m H AE ¼ 85 GeV. This product is the observable that is being constrained by the searches that use 2b tags [38,42], and the figure is very similar to the left panel of Fig. 4. However, to obtain sensitivity to a given contour we expect that the 2b search will require more integrated luminosity than the 3b search, because the latter has smaller backgrounds as discussed earlier in Sec. III B 2. Figure 5 is the same as Fig. 4 but with m H AE ¼ 130 GeV, and hence BRðt → H AE bÞ is reduced compared to the corresponding case with m H AE ¼ 85 GeV. However, in both panels in Fig. 5, the current excluded region is roughly above the contour of 0.02 (instead of 0.2) due to the LHC searches [12,38,42] having superior sensitivity to those of the Tevatron in the region 90 GeV ≤ m H AE ≤ 160 GeV. It can be seen that a sizeable area of the ½tan γ; tan β parameter space is ruled out, while the region below the 0.02 contour would provide a possible signal for H AE . It is hoped that future searches will have sensitivity to contours of 0.001 in both the 2b and 3b channels for 90 GeV ≤ m H AE ≤ 160 GeV.
We now show results in the democratic 3HDM. Taking δ ¼ 0 we find that large values of BRðH AE → cbÞ are possible in regions of parameter space, but these regions are essentially ruled out by the b → sγ constraint. However, by taking δ ≠ 0 there are regions that have a large BRðH AE → cbÞ while complying with the constraints from b → sγ and the electric dipole moment of the neutron. In Fig. 6, we take tan β ¼ 40, tan γ ¼ 10, and m H AE ¼ 85 GeV in the democratic 3HDM. In the left panel, contours of BRðH AE → cbÞ are plotted in the plane ½δ; θ. It can be seen that large values of BRðH AE → cbÞ are possible, but δ has almost no effect on its magnitude. In the right panel of Fig. 6, we plot contours of ReðXY Ã Þ in the plane ½δ; θ, and the allowed parameter space lies within the range −1.1 ≤ ReðXY Ã Þ ≤ 0.7. One can see that varying δ has a sizeable effect on ReðXY Ã Þ. By comparing the left and right panels it can be seen that the region 1 ≤ δ ≤ 5 and 0 ≥ θ ≥ −0.5 gives a large BRðH AE → cbÞ that is also compatible with the b → sγ constraint. However, this region is further constrained by Fig. 7 in which we plot contours of ImðXY Ã Þ in the plane ½δ; θ, and the allowed parameter space lies within the range jImðXY Ã Þj ≤ 0.1. There are three allowed strips (with one being around δ ¼ π) in the  region of large BRðH AE → cbÞ (i.e., 0 ≥ θ ≥ −0.5). Consequently, the democratic 3HDM is a candidate model for a possible signal in future 3b searches for H AE as carried out in [12], although the parameter space for a large BRðH AE → cbÞ is much smaller than that in the flipped 3HDM, and is likely to require δ ≠ 0. Figures 8 and 9 are with the same parameter choice of tan β ¼ 40, tan γ ¼ 10 in the democratic 3HDM, and are the plots that correspond to Figs. 4 and 5 in the flipped 3HDM. Contours of BRðt → H AE bÞ × BRðH AE → cbÞ and BRðt → H AE bÞ × ½BRðH AE → cbÞ þ BRðH AE → csÞ are plotted in the plane ½δ; θ. The (small) allowed region can be read off from Fig. 7, most of it being around δ ¼ π, but with −0.6 ≤ θ ≤ −1.2 being excluded from Fig. 6.

V. CONCLUSIONS
In summary, we have studied a 3HDM wherein two charged Higgs bosons states exist, one of which we have assumed to be lighter than the top quark and the other one heavier. Hence, the light state can be produced in (anti)top decays via t → bH AE , particularly at hadron colliders like the LHC, via the pp → tt process, which herein has a significant cross section (nearing the nb level) so that the main focus of our analysis has been on this H AE production channel. Amongst the possible H AE decay modes in a 3HDM we have selected here the fermionic ones, i.e., H AE → cs, cb, and τν, which are those exploited in collider searches, at both past (LEP and Tevatron) and present (LHC) machines. Amongst these three channels, we have concentrated on H AE → cb as it offers a twofold experimental advantage. On the one hand, the irreducible background from W AE → cb decays is suppressed by the Cabibbo-Kobayashi-Maskawa (CKM) matrix. On the other hand, it can be filtered out by requiring a b-tag of one the two jets that eventually emerges in the detector. Furthermore, from a theoretical point of view this decay  mode may be a privileged probe of the underlying 3HDM structure. This is because the BRðH AE → cbÞ can be large in the flipped and democratic versions of the 3HDM, but not in the type I, type II, and lepton-specific structures, while being compatible with experimental constraints, chiefly, those from b → sγ.
We have then performed the first comprehensive study of the decay mode H AE → cb in terms of the four fundamental parameters of the charged Higgs sector of the 3HDM (β, γ, θ, and δ) over the available m H AE range. We found that the parameter space for a large BRðH AE → cbÞ is much bigger in the flipped 3HDM than in the democratic 3HDM. Our emphasis has been on the interval 80 GeV < m H AE < 90 GeV, to which the LHC has no sensitivity at present, the reason being that no experimental searches have yet been attempted for the decays H AE → cb=cs at this collider. For the purpose of encouraging such searches, we have mapped out the 3HDM parameter spaces of the flipped and democratic types that can be accessible at the LHC as a function of its increased luminosity, concluding that they should be accessible in the near future by exploiting established experimental techniques. In fact, this can be achieved by resorting to both appearance and disappearance searches. The former would have direct sensitivity to the H AE → cb channel while the latter would have indirect access one to it, via the absence of the expected number of W AE → lν (l ¼ e, μ) and W → qq events originating from pp → tt with standard top decay for both t andt. Similarly positive prospects are expected for future e þ e − colliders, like FCC-ee, CEPC, and ILC, where the H AE state would be pair produced via e þ e − → H þ H − . At such colliders the QCD backgrounds are much reduced with respect to those at the LHC, which greatly facilitates the extraction of the H AE → cb mode.