Search for flavor-changing neutral currents in top quark decays t → Hc and t → Hu in multilepton final states in proton-proton collisions at 13 TeV with the ATLAS detector

Flavor-changing neutral currents are not present in the Standard Model at tree level and are suppressed in loop processes by the unitarity of the Cabibbo-Kobayashi-Maskawa matrix; the corresponding rates for top quark decay processes are experimentally unobservable. Extensions of the Standard Model can generate new flavor-changing neutral current processes, leading to signals which, if observed, would be unambiguous evidence of new interactions. A data set corresponding to an integrated luminosity of 36 . 1 fb − 1 of pp collisions at a center-of-mass energy of ﬃﬃﬃ s p ¼ 13 TeV recorded with the ATLAS detector at the Large Hadron Collider is used to search for top quarks decaying to up or charm quarks with the emission of a Higgs boson, with subsequent Higgs boson decay to final states with at least one electron or muon. No signal is observed and limits on the branching fractions B ð t → Hc Þ < 0 . 16% and B ð t → Hu Þ < 0 . 19% at 95% confidence level are obtained (with expected limits of 0.15% in both cases). determined assuming zero FCNC branching fraction; the postfit nonprompt lepton background yield includes the effect of nonzero FCNC branching fraction.


I. INTRODUCTION
In the Standard Model (SM), the mass eigenstates in the quark sector couple diagonally to the photon, Z boson, and Higgs boson, with the result that quark flavors can only change at tree level by emission of W AE bosons (charged currents). Although processes that change quark flavors without external emission of W AE bosons-i.e., flavorchanging neutral currents (FCNC)-occur via loops in the SM, they are suppressed by the Glashow-Iliopoulos-Maiani mechanism [1]. The decay of a top quark to a Higgs boson and a lighter up-type quark q (t → Hq) is estimated to have a branching fraction of about 3 × 10 −15 in the SM [2], which is unobservable with any current or foreseeable data set. An observation of this process with current sensitivity would be unambiguous evidence of physics beyond the Standard Model (BSM). Searches for t → Hq decays are part of a broader FCNC program that includes searches for tγq, tZq, and tgq interactions [3][4][5][6][7].
Models of BSM physics can feature nontrivial flavor structures that produce tree-level or large effective loopinduced tHq couplings. Tree-level couplings are generic in two-Higgs-doublet models unless discrete symmetries are introduced to forbid them [8], and can also be present in models with heavy vectorlike quarks [9]. The Cheng-Sher ansatz [10] of off-diagonal light Higgs boson interactions in models with multiple Higgs doublets gives couplings that scale as λ tHq ∼ ffiffiffiffiffiffiffiffiffiffiffiffiffi ffi 2m t m q p =v, where m t is the top quark mass, m q is the light up-type quark mass, and v is the SM Higgs field vacuum expectation value. This would lead to a branching fraction Bðt → HcÞ ≈ 0.15%, close to current experimental bounds. Smaller loop-induced enhancements can be present in two-Higgs-doublet models even without off-diagonal tree-level couplings [11], the minimal supersymmetric Standard Model [12], R-parity-violating supersymmetry [13], models with warped extra dimensions [14], and composite Higgs boson models [15]. A summary of expectations for t → Hq branching fractions in various BSM models can be found in Ref. [16].
The ATLAS and CMS collaborations have carried out searches [17][18][19][20][21] for tHq interactions with 7, 8, and 13 TeV pp collision data from the Large Hadron Collider (LHC). These primarily searched for tt production where one top quark decays via t → Wb and the other decays via t → Hq; the analyses are distinguished by the targeted Higgs boson decay. The results at 13 TeV benefit from the large integrated luminosity delivered by the LHC in 2015-2016 and from the tt cross section being larger at 13 than at 8 TeV. Using 13 TeV data, ATLAS obtained Bðt → HcÞ < 0.22% with H → γγ decays [20] and CMS obtained Bðt → HcÞ < 0.47% with H → bb decays [21], in both cases at 95% confidence level (C.L.) and assuming Bðt → HuÞ ¼ 0. Similar limits apply for Bðt → HuÞ assuming Bðt → HcÞ ¼ 0.
In this paper, a search for production of tt pairs in which one top quark decays via t → Hq is reported, targeting multilepton final states with either three light leptons (l ¼ e or μ) or two light leptons of the same electric charge. The multilepton final states can be produced via Higgs boson decays which involve light leptons in the final state through H → WW Ã , H → ττ in which both τ-leptons decay leptonically, or H → ZZ Ã . A tt pair with a t → Hq decay, followed by H → WW Ã , would feature the intermediate state ðWbÞðWW Ã qÞ, which can yield the final states llb3q2ν or lllbq3ν (q representing any quark lighter than the bquark). Events with hadronically decaying τ-lepton candidates are vetoed to avoid overlap with dedicated searches for those Higgs boson decay modes.
The sought-after tt, t → Hq signature with Higgs boson decays into final states with leptons is in many respects similar to the corresponding channel for ttH production, although t → Hq has one fewer b-jet and one fewer lightquark jet for the same lepton multiplicity in the absence of additional radiation. The sample of events and background simulations used for the 13 TeV ATLAS search for ttH production in the multilepton final state [22] is therefore leveraged in this search, similar to the strategy used in the 8 TeV multilepton t → Hq search [17]. Unlike the 8 TeV search, however, in this analysis multivariate discriminants (boosted decision trees, BDTs) are used, optimized to separate the FCNC signal from SM processes.
The analysis proceeds as follows. The same-charge dilepton and trilepton categories (with no hadronic τ-lepton candidates) from the 13 TeV ttH search [22] are used. The same event preselection, calibration, and SM simulation samples are used as in the ttH search, but the ttH process is now treated as a background and fixed to the expected SM rate. The FCNC processes t → Hu and t → Hc are simulated and BDTs are trained to separate these from SM processes in the two categories. Care is taken to account for FCNC signal contamination in control regions which are used to constrain backgrounds arising from lepton production in hadron decays and photon conversions. The results are obtained by fitting the data distributions of the BDT discriminants.

II. ATLAS DETECTOR AND OBJECT RECONSTRUCTION
The ATLAS experiment [23] at the LHC is a multipurpose particle detector with a forward-backward symmetric cylindrical geometry and near 4π coverage in solid angle. 1 It consists of an inner tracking detector surrounded by a thin superconducting solenoid providing a 2 T axial magnetic field, electromagnetic and hadron calorimeters, and a muon spectrometer. The inner tracking detector covers the pseudorapidity range jηj < 2.5. It consists of silicon pixel, silicon microstrip, and transition radiation tracking detectors. A new innermost silicon pixel layer was installed prior to data-taking at ffiffi ffi s p ¼ 13 TeV [24]. Lead/ liquid-argon (LAr) sampling calorimeters provide electromagnetic (EM) energy measurements with high granularity for jηj < 3.2. A hadron steel/scintillator-tile calorimeter covers the central pseudorapidity range (jηj < 1.7). The endcap and forward regions are instrumented with LAr calorimeters for both the EM and hadronic energy measurements up to jηj ¼ 4.9. The muon spectrometer surrounds the calorimeters and is based on three large air-core toroidal superconducting magnets with eight coils each. The muon spectrometer includes a system of precision tracking chambers and fast detectors for triggering. A combined hardware and software trigger system is used to select events. The first-level trigger is implemented in hardware and uses a subset of the detector information to reduce the accepted rate to a design maximum of 100 KHz. This is followed by a software-based trigger that further reduces the accepted event rate to a sustained rate of about 1 KHz. This analysis uses an integrated luminosity of 36.1 fb −1 of pp collision data with ffiffi ffi s p ¼ 13 TeV collected during 2015 and 2016, corresponding to approximately 30 × 10 6 tt pair production events [25]. The mean number of pp interactions per bunch crossing in the data set is 24. A full description of the reconstruction and selection of physics objects can be found in Ref. [22], and only a brief description follows. Events in this analysis are selected using triggers which require the presence of one or two light leptons [26]. The single-lepton triggers have transverse momentum (p T ) thresholds that vary from 20 to 26 GeV depending on lepton flavor and instantaneous luminosity. In the dilepton triggers the thresholds for the higher-p T lepton vary from 12 to 22 GeV depending on lepton flavor and instantaneous luminosity.
Muon candidates are formed using inner detector tracks and muon spectrometer tracks, track segments, or for jηj < 0.1, calorimeter signals consistent with the passage of a minimum-ionizing particle. They are required to satisfy p T > 10 GeV and jηj < 2.5 and to pass loose identification requirements [27], as well as transverse and longitudinal track impact parameter requirements with respect to the primary vertex. The primary vertex in an event is defined as the reconstructed pp collision vertex with the highest P p 2 T of associated tracks with p T > 400 MeV. Electron candidates are formed from energy clusters in the electromagnetic calorimeter associated with inner detector tracks. They are required to satisfy p T > 10 GeV and 1 ATLAS uses a right-handed coordinate system with its origin at the nominal interaction point (IP) in the center of the detector and the z axis along the beam pipe. The x axis points from the IP to the center of the LHC ring, and the y axis points upwards. Cylindrical coordinates ðr; ϕÞ are used in the transverse plane, ϕ being the azimuthal angle around the z axis. The pseudorapidity is defined in terms of the polar angle θ as η ¼ − ln tanðθ=2Þ. Angular distance is measured in units of ΔR ≡ ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ffi ðΔηÞ 2 þ ðΔϕÞ 2 p . jη cluster j < 2.47, excluding the transition region 1.37 < jη cluster j < 1.52 between the barrel and endcap electromagnetic calorimeters. A likelihood discriminant is used to separate isolated prompt electrons from hadronic jets, photon conversions, and nonisolated electrons from hadron decays; two working points of the discriminant (loose and tight [28]) are used in this analysis. Electron candidates must also pass transverse and longitudinal track impact parameter requirements.
Jets are reconstructed using clusters built from energy deposits in the calorimeters [29-31] using the anti-k t algorithm [32,33] with radius parameter R ¼ 0.4. Considered jets are required to satisfy p T > 25 GeV and jηj < 2.5. Jets with p T < 60 GeV and jηj < 2.4 are required to be matched to the primary vertex using their associated inner detector tracks [34]. Jets containing b-hadrons (b-jets) are tagged using a multivariate discriminant combining information about secondary vertices, reconstructed decay chains, and impact parameters of tracks associated with the jet [35,36]. The working point for b-jet identification used in this analysis corresponds to an average efficiency of 70% for jets containing b-hadrons with p T > 20 GeV and jηj < 2.5 in simulated tt events. Jets which contain c-hadrons but not b-hadrons have approximately a one in twelve probability of being misidentified as b-jets; the same probability for lightquark or gluon jets is one in 380, leading to different responses for the t → Hu and t → Hc processes.
Hadronically decaying τ-lepton (τ had ) candidates are reconstructed from hadronic jets associated with either one or three inner detector tracks with a total charge of AE1 [37,38]. A BDT discriminant is used to distinguish τ had candidates from quark-or gluon-initiated jets. A working point with 55% (40%) efficiency for one-prong (threeprong) τ had decays is used for the veto. Candidates with p T > 25 GeV and jηj < 2.5 are considered.
The missing transverse momentum, with magnitude E miss T , is defined as the negative vector sum of the transverse momenta of all calibrated and identified leptons and jets and the remaining unclustered energy of the event, which is estimated from low-p T tracks associated with the primary vertex but not with any considered lepton or jet [39].
To eliminate ambiguity between reconstructed objects and reduce the fraction of leptons arising from hadron decay, the following additional requirements are imposed: if two electrons are separated by ΔR < 0.1, only the higherp T electron is considered; any electron within ΔR ¼ 0.1 of a selected muon is rejected; jets within ΔR ¼ 0.3 of a selected electron are removed; any τ had candidate within ΔR ¼ 0.2 of a selected electron or muon is ignored; and muons must be separated by ΔR > minð0.4; 0.04 þ ð10 GeVÞ=p T;μ Þ from a jet surviving the above selection.
To further reject backgrounds, BDTs are trained to discriminate against electrons arising from asymmetric trident processes e AE → e AE e þ e − in detector material which may induce an apparent change of the electron charge, and against leptons which arise from decays of b-hadrons [22]. The former discriminant ("charge misassignment BDT") uses track and calorimeter cluster information; the latter ("nonprompt lepton BDT") uses information about additional particle activity and properties of low-p T jets formed from tracks near the lepton, including the output of b-tagging algorithms run on those jets. Working points with efficiencies of 95% (60%-98%) are defined for the charge misassignment (nonprompt lepton) discriminants.
During the initial loose (L) selection of leptons, no isolation, charge misassignment discriminant, or nonprompt lepton discriminant requirements are imposed. Electrons are required to pass the loose electron identification discriminant selection. Two other lepton selections are used in the analysis: (i) Modified loose (L*), in which the lepton must pass calorimeter-and track-based isolation criteria and the nonprompt lepton discriminant selection.

III. SIMULATION, EVENT SELECTION, AND ANALYSIS
The simulated pp → tt, t → Hq signals were generated with next-to-leading-order (NLO) QCD matrix elements computed by MADGRAPH5_AMC@NLO [40], with top quark decays performed by MadSpin [41]; PYTHIA 8 [42] was used for Higgs boson decay, parton showering, hadronization, and underlying-event generation. Either the top quark or antiquark may undergo the FCNC decay in this sample. The total top quark pair production cross section used to normalize the FCNC signal was set to 832 þ40 −46 pb, as calculated with the TOP++2.0 program at next-to-next-to-leading order in perturbative QCD, including soft-gluon resummation to next-tonext-to-leading-log order [25]. The systematic uncertainties include variation of factorization and renormalization scales as well as uncertainties in parton distribution functions (PDFs) and the QCD coupling α s [43][44][45][46].
The simulations of SM background processes are the same as those used in Ref. [22]. In particular, the major processes ttZ, ttW, and ttH were generated at NLO in QCD with MADGRAPH5_AMC@NLO interfaced to PYTHIA 8 for parton showering, hadronization, particle decay, and underlying-event generation. The top quark and Higgs boson masses were set to 172.5 and 125.0 GeV, respectively. Higgs boson decay branching fractions were taken from Ref. [47]. In all the preceding samples, the matrix element calculations used the NNPDF 3.0 NLO PDF set [48], while the parton shower calculations used the A14 tune of PYTHIA 8 parameters [49] and the NNPDF 2.3 LO PDF set [46]. Diboson production was generated at NLO in QCD with SHERPA [50,51] using the CT10 PDF set [52]. The response of the ATLAS detector to generated events was simulated with a GEANT4-based detector model [53,54] with parametrization of the calorimeter response for some minor backgrounds [55].
The two categories of events used in this analysis, samecharge dilepton (2lSS) and trilepton (3l) candidates, are selected by the same requirements as in Ref. [22]. In both cases the leptons identified by the trigger must correspond to leptons selected for this analysis, with sufficiently high p T for the trigger to be fully efficient.
(i) Events in the 2lSS category must have at least four reconstructed jets, of which one or two must be b-tagged jets. Exactly two light-lepton candidates meeting L criteria (as described in Sec. II) must be found, along with no τ had candidates. The leptons are then also required to pass the T selection. Both leptons must have p T > 20 GeV and have the same charge. (ii) Events in the 3l category must have at least two reconstructed jets, of which at least one must be a b-tagged jet. Exactly three light-lepton candidates meeting L criteria must be found, along with no τ had candidates. The total charge of the leptons must be AE1. Of the three leptons, one (designated l 0 ) is of opposite charge to the other two, and the other two leptons are designated l 1 and l 2 in order of increasing angular separation ΔR from l 0 . The two leptons l 1 and l 2 are required to meet the T criteria and have p T > 15 GeV. The remaining lepton, l 0 , is less likely to be nonprompt than either of the same-charge leptons l 1 or l 2 . It is correspondingly only required to meet the L* criteria and to have p T > 10 GeV. To remove contamination from hadron decay chains including l þ l 0− , both invariant masses mðl 0 l 1 Þ and mðl 0 l 2 Þ must exceed 12 GeV. To remove contamination from ttZ, a Z boson veto is imposed: jmðl þ l − Þ − 91.2 GeVj > 10 GeV for every opposite-charge lepton pair of the same flavor (e þ e − or μ þ μ − ). Finally, contamination from Z → llγ ðÃÞ → lll 0 ðl 0 Þ, where one lepton has low momentum and is not reconstructed, is removed by requiring jmð3lÞ − 91.2 GeVj > 10 GeV. After these selections, the t → Hq signal is dominated by H → WW Ã (85% of the 2lSS and 71% of the 3l category) with subleading contributions from H → ττ (12% and 16% respectively) and H → ZZ Ã (2% and 9% respectively). The fraction of tt events with a consequent t → Hq decay which are expected to be reconstructed and selected is 5.1 × 10 −4 (2.6 × 10 −4 ) for the 2lSS (3l) category. 2 Following the initial selections, the largest sources of background are those arising from nonprompt leptons (from hadron decays, photon conversions, and charge misassignment), mainly from tt decays, and prompt lepton backgrounds from ttV production (V ¼ W or Z) with leptonic decays of the vector boson. Further BDT discriminants are trained to separate the FCNC signal from these two background sources.
Inputs to the BDTs include lepton flavor and kinematic observables, jet properties including whether they are btagged, angular separations between objects, the E miss T , and the quantity m eff ≡ E miss T þ H T , where H T is the scalar sum of the p T of leptons and jets in the event. Signal events can be distinguished from ttV and nonprompt lepton background by having only one true b-jet and being relatively soft events with low m eff and H T . The spin correlation in the dominant H → WW Ã decay, and the presence of an off-mass-shell W Ã boson, also yields a distinct signature where, in both categories, one lepton often has low p T , and in the 3l category, both mðl 0 l 1 Þ and ΔRðl 0 ; l 1 Þ are small.
The variables used in the training of the BDTs in the two categories are shown in Table I, and example distributions are shown in Fig. 1. Good agreement is observed between the data and expected background distributions in each variable. In the 2lSS category, the variables that most TABLE I. Variables used to construct the BDT discriminants for the 2lSS and 3l categories. The symbol "×" indicates that this variable is used in the respective BDT. The "best Z candidate" is the opposite-charge lepton pair with same flavor with invariant mass closest to 91.2 GeV; if no such pair exists, zero is assigned for the invariant mass.

Variable
2lSS 3l These values include the effects of selection acceptance, detector efficiency, and decay branching fractions. strongly separate t → Hq from the nonprompt lepton background are the number of b-jets, the p T of the lower-p T lepton, and the angular separation ΔR of the lower-p T lepton from the closest jet. The separation from ttV processes comes mainly from the number of b-jets, m eff , E miss T , and the p T of the higher-p T lepton. In the 3l category the strongest separation from nonprompt lepton backgrounds comes from ΔRðl 0 ; l 1 Þ, the invariant masses mðl 0 ; l 1 Þ and mðl 0 ; l 2 Þ, the p T of the highest-p T lepton, and the invariant mass of the opposite-charge, same-flavor lepton pair with mass closest to that of the Z boson. The strongest separation from ttV comes from the invariant masses mðl 0 ; l 1 Þ and mðl 0 ; l 2 Þ, m eff , the number of b-jets, and the invariant mass of the three leptons. In both the 2lSS and 3l categories and against both backgrounds, better separation is achieved for t → Hu than t → Hc signals, as the latter is more likely to have a second b-tagged jet arising from the hadronization products of the charm quark. In the 2lSS category, t → Hu and t → Hc are sufficiently similar that only one discriminant is trained for the two decay modes. In the 3l category the two signals are treated separately.
In each case, two separate discriminants are trained: one to separate t → Hq from nonprompt leptons and one to separate t → Hq from ttV processes. The former is designated BDTðtt; XÞ and the latter BDTðttV; XÞ for a given category/signal choice X (i.e., 2lSS, 3l t → Hu, or 3l t → Hc). The expected distributions for backgrounds with nonprompt leptons are obtained using the procedures described in Sec. IV. The two discriminants are combined into a single discriminant, designated BDT(X), via a linear combination that yields the best expected limit on the FCNC branching fraction.
The BDT discriminant outputs in the 2lSS and 3l categories are binned into six or four bins, respectively, with bin widths optimized to provide the best expected limits. Every bin contains a roughly equal number of signal events. The bin boundaries used for the fits to t → Hu and t → Hc signals in the 2lSS category are optimized separately, although the discriminant used is the same for the two decays.

IV. BACKGROUND ESTIMATION
The estimation of rates and kinematic distributions of SM processes that form backgrounds with prompt leptons in the signal categories is performed using simulation. The processes considered include the following: (i) ttW, ttðZ=γ Ã → llÞ, ttH, and ttWW; (ii) ttt and tttt; (iii) single top quark production in the sand t-channels, tW, tZ, tWZ, tHb, and tHW; (iv) production of two or three W or Z=γ Ã bosons. Details of the simulations used are given in Ref. [22].
The estimation of the nonprompt lepton backgrounds, including the contribution from charge-misassigned electrons, uses data-driven methods following Ref. [22]. One major modification to the treatment of leptons from hadron decays and conversions is made, arising from the lower expected jet multiplicity in t → Hq events compared to ttH production. A summary of the procedure is given below. The self-consistency of the methods is checked by predicting the nonprompt lepton yields in simulated SM tt and ttγ events; good agreement is observed.
Rates for electron charge misassignment as functions of electron p T and jηj are determined from a sample of Z → ee events where both electrons are reconstructed with the same charge. The contribution of these events to the 2lSS signal region is determined by applying these misassignment rates to events selected with the 2lSS selection criteria except requiring opposite-charge leptons instead of same-charge leptons.
The estimation of nonprompt lepton background contributions other than electron charge misassignment is performed using the so-called matrix method [56,57]. This technique uses control regions with the same kinematic properties as the signal region, but with changed lepton identification criteria that enhance the nonprompt lepton contribution, to statistically estimate the fraction of signal region events that involve nonprompt leptons. Lepton candidates that meet L, but not T, identification criteria (T) are much more likely to be of nonprompt origin than those that meet T criteria; if the probabilities ("efficiencies") of prompt and nonprompt leptons to be identified asT or T are known, the fraction of tight-lepton events of nonprompt origin can be determined by solving a system of equations. The matrix method can be applied to estimate contributions arising from multiple nonprompt leptons in the same event. In this analysis, the two leptons in the 2lSS category events, and the two same-charge leptons in the 3l category events, are analyzed for a nonprompt contribution. The opposite-sign lepton l 0 in 3l events is found in simulation to be prompt 97% of the time, so the matrix method is not applied to that lepton.
The nonprompt efficiencies are measured in control regions with the same selection criteria as the 2lSS signal category except that the number of jets must be two or three, one lepton need only satisfy L criteria, and the lowerp T lepton is only required to have p T > 15 GeV. These are then separated into TT or TT events. The expected number of events from SM processes with only prompt leptons in these regions is determined from simulation and subtracted from the observed number of events, giving a yield of nonprompt lepton events which is then used to determine the nonprompt efficiencies. In the case of a nonzero t → Hq branching fraction, a significant fraction of the signal will be reconstructed in these control regions. This will act as an additional source of prompt leptons which is not accounted for in the SM prediction and will bias the nominal efficiencies which are determined assuming zero signal contribution. For Bðt → HqÞ ¼ 0.2%, the FCNC process would contribute approximately 30% of the prompt lepton contribution in the low-jet-multiplicity control regions with T leptons. This effect is accounted for in two ways. First, the nonprompt efficiencies are derived under the two hypotheses Bðt → HqÞ ¼ 0 and Bðt → HqÞ ¼ 0.2% and both values are used to predict the yield of nonprompt leptons in the signal categories; the two hypotheses result in nonprompt yields differing by ≈40% for the 2lSS and ≈30% for the 3l category. This overall normalization correction from possible signal contamination is then scaled proportionally to the FCNC branching fraction in the fit. Second, the change in the shape of the FCNC discriminant response for the nonprompt background in the signal regions under the two hypotheses is derived. The difference is assigned as a systematic uncertainty on the nonprompt background discriminant shape.
The separation of FCNC signal from nonprompt lepton background by the BDTðtt; XÞ discriminants is sufficiently strong that the impact of these systematic uncertainties in the nonprompt background estimate on the signal extraction is small. Tests with MC simulation indicate that the procedure correctly recovers the branching fraction of injected FCNC signals.

V. SYSTEMATIC UNCERTAINTIES
The same model of systematic uncertainties in background processes (including ttH) is used as in Ref. [22], with the additional normalization and BDT shape uncertainties in nonprompt lepton backgrounds described in Sec. IV. As the measured ttH cross section is compatible with the SM predictions, the SM rate is assumed with appropriate theoretical uncertainties. Acceptance uncertainties from the choice of parton distribution functions and QCD scale for the major backgrounds simulated with MADGRAPH5_AMC@NLO are calculated using SYSCALC [58].
The t → Hq signal processes are subject to their own theoretical uncertainties, primarily in the modeling of the parent tt system. Systematic uncertainties are assigned for the tt cross section, the variation of BDT response with the choice of renormalization and factorization scale, the modeling of parton showers, the event generator, and the amount of initial/final-state radiation. The systematic uncertainty model includes components from (i) light lepton, τ had , and jet selection and energy/ momentum scale and E miss T modeling; (ii) b-jet tagging efficiency and the probability for c-jets and light-quark or gluon jets to be misidentified as b-jets; (iii) the cross section and MC modeling of simulated backgrounds and signals; (iv) the statistical uncertainties in the control regions for nonprompt lepton backgrounds, the matrix method efficiencies, and the applicability to the 2lSS and 3l category events of the matrix method efficiencies derived at low jet multiplicity in same-charge dilepton events; (v) electron charge misassignment; (vi) pp integrated luminosity (determined using a methodology similar to that described in Ref. [59]); (vii) modeling of multiple pp interactions per bunch crossing. The background-related systematic uncertainties with the largest impact on the final result are found to be those associated with the statistical uncertainty in the nonprompt lepton background estimation, the nonprompt lepton efficiencies used in the matrix method, and the cross section for diboson production in association with b-quarks. Excluding the correction for signal contributions to nonprompt lepton control regions, systematic uncertainties on the background lead to an uncertainty in the determined signal decay branching fractions of 0.04%. Systematic uncertainties in the signal processes are primarily associated with the matching of matrix element calculations with parton shower algorithms and different choices of parton shower algorithms. The relative systematic uncertainty in the signal yield prediction for a given signal branching fraction is 8%. The uncertainty in the background estimate due to the signal contributions in nonprompt lepton control regions depends on the signal decay branching fraction; for a true branching fraction of 0.2%, the corresponding systematic uncertainty on the determined branching fraction is 0.02%.

VI. RESULTS
Binned maximum-likelihood fits to the distributions of the 2lSS and 3l FCNC discriminants are performed to extract the best-fit values of the t → Hq branching fractions. The profile likelihood technique is used, in which systematic uncertainties are modeled as nuisance parameters ⃗ θ which are allowed to vary in the fit, constrained by Gaussian or log-normal probability density penalty functions multiplying the likelihood function L. The test statistic q B is obtained from the profile log-likelihood ratio as q B ≡ −2 ln Λ B ¼ −2 ln ½LðB;⃗ θÞ=LðB;⃗ θÞ, whereB and ⃗ θ are the t → Hq branching fraction and nuisance parameter values that give the global maximum likelihood and⃗ θ are the nuisance parameter values which maximize L for a given branching fraction B. The uncertainties in the best-fit branching fraction valueB are determined by the variation of q B by one unit from its minimum, and the distribution of q B is used to set 95% C.L. upper limits on the branching fractions Bðt → HqÞ using the CL s method [60]. Due to the near-degeneracy of the BDT response to t → Hu and t → Hc signals, during the fits, one of these branching fractions is set to zero while the other is permitted to float. Expected and observed yields of events in the signal categories, before and after the nuisance parameters are adjusted in the fits, are shown in Table II. The distributions of the FCNC discriminant for the data and the best-fit signal-plus-background models are shown in Fig. 2.
The results of the fits are shown in Tables III and IV. The best-fit branching fractions are compatible with zero, and 95% C.L. upper limits are set, as shown in Fig. 3. Statistical uncertainties are dominant in the result. No variations of the nuisance parameters by more than 1σ of the prior systematic uncertainty are observed; the largest variations are observed in the nuisance parameters associated with statistical uncertainties in the nonprompt lepton background estimate and in one of the normalization systematic uncertainties in the 3l nonprompt lepton background.
To confirm the self-consistency of the nonprompt lepton background estimate, a number of checks were performed. There is no evidence of a BDT response shape distortion in the nonprompt lepton background estimate during the fit TABLE II. Expected SM background (including nonprompt leptons), FCNC contributions, and observed data yields in the signal categories. The FCNC contribution shown in the "prefit" rows is the expected yield for a 0.2% branching fraction, and the result in the "postfit" rows is the best-fit yield from the combined fit of the 2lSS and 3l categories. The uncertainties shown for the FCNC yields reflect systematic uncertainties given a specific branching fraction (prefit) or the uncertainties in the yield from the full fit (postfit). SM backgrounds are determined as described in Sec. IV. The nonprompt lepton background estimate includes electron charge misassignment. The nonprompt lepton component of the prefit SM background is determined assuming zero FCNC branching fraction; the postfit nonprompt lepton background yield includes the effect of nonzero FCNC branching fraction.

Category
Nonprompt In the t → Hu fit, Bðt → HcÞ is set to zero, and vice versa. The binning, which is that used in the fit, is chosen so as to have roughly equal signal yields in each bin. The FCNC signals, normalized to their best-fit branching fractions from the combined fit to the 2lSS and 3l categories, are shown as filled red histograms stacked above the background components; only t → Hu is large enough to be visible. The hashed band indicates the total uncertainty in the signal-plus-background prediction, including the statistical uncertainty in the bestfit FCNC signal. The dashed red lines show the expected contribution of the respective FCNC decay with the 95% C.L. upper limit branching fraction (0.19% for t → Hu, 0.16% for t → Hc). which would be consistent with a FCNC signal. In addition, a second fit that decorrelates FCNC contributions in the signal categories from their impact on the nonprompt lepton efficiency estimates finds best-fit branching fractions consistent with those of the nominal fit.

VII. CONCLUSION
Any observable branching fraction for the FCNC decays t → Hq would indicate new physics beyond the Standard Model. A search for tt production events in which one top quark or antiquark undergoes a t → Hq decay was carried out with an integrated luminosity of 36.1 fb −1 of pp collision data with ffiffi ffi s p ¼ 13 TeV collected in 2015 and 2016 using the ATLAS detector at the LHC. Two final states are targeted: two same-charge light leptons with four or more jets, and three light leptons with two or more jets. These are sensitive primarily to H → WW Ã decays, with subleading contributions from H → ττ and H → ZZ Ã . Specialized boosted decision trees using the kinematic properties of the final-state particles are used to distinguish FCNC signals from nonprompt lepton backgrounds and from ttW and ttZ production. Potential contamination from FCNC signal in the nonprompt lepton background control regions is treated in a self-consistent manner. No evidence of FCNC decays is found and the upper limits set on the branching fractions are Bðt → HcÞ < 0.16% and Bðt → HuÞ < 0.19% at 95% C.L.

ACKNOWLEDGMENTS
We thank CERN for the very successful operation of the LHC, as well as the support staff from our institutions without whom ATLAS could not be operated efficiently. We acknowledge the support of ANPCyT