Search for Higgs and Z boson decays to J/ ## and #(nS)# with the ATLAS detector

A search for the decays of the Higgs and Z bosons to J= ψγ and ϒ ð nS Þ γ ( n ¼ 1 ; 2 ; 3 ) is performed with pp collision data samples corresponding to integrated luminosities of up to 20 . 3 fb − 1 collected at ﬃﬃﬃ s p ¼ 8 TeV with the ATLAS detector at the CERN Large Hadron Collider. No significant excess of events is observed above expected backgrounds and 95% C.L. upper limits are placed on the branching fractions. In the J= ψγ final state the limits are 1 . 5 × 10 − 3 and 2 . 6 × 10 − 6 for the Higgs and Z boson decays, respectively, while in the ϒ ð 1 S; 2 S; 3 S Þ γ final states the limits are ð 1 . 3 ; 1 . 9 ; 1 . 3 Þ × 10 − 3 and ð 3 . 4 ; 6 . 5 ; 5 . 4 Þ × 10 − 6 , respectively.

Search for Higgs and Z Boson Decays to J=ψγ and ϒðnSÞγ with the ATLAS Detector G. Aad et al. * (ATLAS Collaboration) (Received 15 January 2015;published 26 March 2015) A search for the decays of the Higgs and Z bosons to J=ψγ and ϒðnSÞγ (n ¼ 1; 2; 3) is performed with pp collision data samples corresponding to integrated luminosities of up to 20.3 fb −1 collected at ffiffi ffi s p ¼ 8 TeV with the ATLAS detector at the CERN Large Hadron Collider. No significant excess of events is observed above expected backgrounds and 95% C.L. upper limits are placed on the branching fractions. In the J=ψγ final state the limits are 1.5 × 10 −3 and 2.6 × 10 −6 for the Higgs and Z boson decays, respectively, while in the ϒð1S; 2S; 3SÞγ final states the limits are ð1.3; 1.9; 1.3Þ × 10 −3 and ð3.4; 6.5; 5.4Þ × 10 −6 , respectively. DOI: 10.1103/PhysRevLett.114.121801 PACS numbers: 14.80.Bn, 13.38.Dg, 14.70.Hp, 14.80.Ec Rare decays of the recently discovered Higgs boson [1,2] to a quarkonium state and a photon may offer unique sensitivity to both the magnitude and sign of the Yukawa couplings of the Higgs boson to quarks [3][4][5][6]. These couplings are challenging to access in hadron colliders through the direct H → qq decays, owing to the overwhelming QCD background [7].
Among the channels proposed as probes of the light quark Yukawa couplings [4,6], those with the heavy quarkonia J=ψ or ϒðnSÞ (n ¼ 1; 2; 3), collectively denoted as Q, in the final state are the most readily accessible, without requirements for dedicated triggers and reconstruction methods beyond those used for identifying the J=ψ or ϒ. In particular, the decay H → J=ψγ may represent a viable probe of the Hcc coupling [4], which is sensitive to physics beyond the Standard Model (SM) [8,9], at the Large Hadron Collider (LHC). The expected SM branching fractions for these decays have been calculated to be BðH →J=ψγÞ¼ð2.8AE0.2Þ×10 −6 , B½H →ϒðnSÞγ¼ ð6.1 þ17.4 −6.1 ;2.0 þ1.9 −1.3 ;2.4 þ1.8 −1.3 Þ×10 −10 [5]. No experimental information on these branching fractions exists. These decays are a source of background and potential control sample for the nonresonant decays H → μ þ μ − γ. These nonresonant decays are sensitive to new physics [10].
Rare decay modes of the Z boson have attracted attention focused on establishing their sensitivity to new physics [11]. Several estimates of the SM branching fraction for the decay Z → J=ψγ are available [12][13][14] with the most recent being ð9.96 AE 1.86Þ × 10 −8 [14]. Measuring these Z → Qγ branching fractions, benefiting from the larger production cross section relative to the Higgs case, would provide an important benchmark for the search and eventual observation of H → Qγ decays. Additionally, experimental access to resonant Qγ decay modes would also provide an invaluable tool for the more challenging measurement of inclusive associated Qγ production, which has been suggested as a promising probe of the nature of quarkonium production in hadronic collisions [15,16].
This Letter presents a search for decays of the recently observed Higgs boson and the Z boson to J=ψγ and ϒðnSÞγ final states. The decays J=ψ → μ þ μ − and ϒðnSÞ → μ þ μ − are used to reconstruct the quarkonium states. The search is performed with a sample of pp collision data corresponding to an integrated luminosity of 19.2 fb −1 (20.3 fb −1 ) for the J=ψγ ½ϒðnSÞγ analysis, respectively, recorded at a center-of-mass energy ffiffi ffi s p ¼ 8 TeV with the ATLAS detector [22], described in detail in Ref. [23]. Higgs boson production is modeled using the POWHEG-BOX Monte Carlo (MC) event generator [24][25][26][27][28], separately for the gluon fusion (ggF) and vector-boson fusion (VBF) processes calculated in quantum chromodynamics (QCD) up to next-to-leading order in α S . The Higgs boson transverse momentum (p T ) distribution predicted for the ggF process is reweighted to match the calculations of Refs. [29,30], which include QCD corrections up to next-to-next-to-leading order and QCD soft-gluon resummations up to next-to-next-to-leading logarithms. Quark mass effects in ggF production [31] are also accounted for.
Physics beyond the SM that modifies the charm coupling can also change production dynamics and branching fractions. In this analysis we assume the production rates and dynamics for a SM Higgs boson with m H ¼ 125 GeV, obtained from Ref. [32], with an uncertainty on the b-hadron decays, the measured transverse decay length L xy between the dimuon vertex and the primary pp vertex is required to be less than three times its uncertainty σ L xy . In this case, the primary pp vertex is defined as the reconstructed vertex with the highest P i p 2 Ti of all associated tracks used to form the vertex.
Photon reconstruction is seeded by clusters of energy in the electromagnetic calorimeter. Clusters without matching tracks are classified as unconverted photon candidates. Clusters matched to tracks consistent with the hypothesis of a photon conversion into an e þ e − pair are classified as converted photon candidates [44]. Reconstructed photon candidates are required to have transverse momentum p γ T > 36 GeV, pseudorapidity jη γ j < 2.37, excluding the barrel/endcap calorimeter transition region 1.37 < jη γ j < 1.52, and to satisfy the "tight" photon identification criteria [45]. To further suppress the contamination from jets, an isolation requirement is imposed. The sum of the transverse momentum of all tracks and calorimeter energy deposits within ΔR ¼ 0.2 of the photon direction, excluding those associated with the reconstructed photon, is required to be less than 8% of the photon's transverse momentum.
Combinations of a Q → μ þ μ − candidate and a photon, satisfying Δϕðμ þ μ − ; γÞ > 0.5, are retained for further analysis. To improve the sensitivity of the search, the events are classified into four exclusive categories, based upon the pseudorapidity of the muons and the photon reconstruction classification. Events where both muons are within the region jη μ j < 1.05 and the photon is (is not) classified as a conversion constitute the "barrel converted" (BC) ["barrel unconverted" (BU)] category. Events where at least one of the muons is outside the region jη μ j < 1.05 and the photon is (is not) classified as a conversion constitute the "endcap converted" (EC) ["endcap unconverted" (EU)] category. The number of candidates observed in each category following the complete event selection is shown in Table I.
The total signal efficiency (kinematic acceptance, trigger, and reconstruction efficiencies) in the J=ψγ final state is 22% and 12% for the Higgs and Z boson decays, respectively. The corresponding efficiencies for the ϒðnSÞγ final state are 28% and 15%. The m μμγ resolution is similar for both the Higgs and Z boson decays and varies between 1.2% and 1.8%. The m μμ resolution is 1.4% and 2.4% for the barrel and endcap categories, respectively.
The main source of background, referred to as inclusive QCD background, is dominated by inclusive quarkonium production where a jet in the event is reconstructed as a photon. For the ϒðnSÞγ final state, events containing Z → μ þ μ − decays with final-state photon radiation (FSR) constitute a second source of background, a contribution which is found to be negligible in the J=ψγ final state. The normalization of both of these background sources is extracted directly from a fit to data. The modeling of the inclusive QCD background shape, obtained with a datadriven approach, and of the Z → μ þ μ − background shape, obtained from simulation, is described in the following two paragraphs.
The background from inclusive QCD processes is modeled with a nonparametric data-driven approach using templates to describe the kinematic distributions. The approach exploits a sample of loosely selected μ þ μ − γ events, around 2400 in the J=ψγ channel and around 3200 in the ϒðnSÞγ channel. These control samples are formed from events satisfying the nominal Qγ selection, but with relaxed dimuon and photon transverse momenta (p γ T > 25 GeV and p μμ T > 25 GeV) and isolation requirements (separate fractional calorimeter energy and track momentum isolation for the photon and dimuon system of less than 60%). Contamination of this sample from signal events is expected to be negligible. Probability density functions (pdfs) used to model the p μμ T , p γ T , Δηðμ þ μ − ; γÞ and Δϕðμ þ μ − ; γÞ distributions of this control sample, independently for each category, are constructed using Gaussian kernel density estimation [46]. To account for kinematic correlations, the distributions of p γ T , Δηðμ þ μ − ; γÞ and Δϕðμ þ μ − ; γÞ are estimated in eight exclusive regions of p μμ T . In the case of the dimuon and photon isolation variables, correlations are accounted for by using twodimensional histograms derived in five exclusive regions of p μμ T . The m μμ distributions are modeled using Gaussian pdfs, with parameters derived from a fit to the control sample. In the ϒðnSÞγ channel, the data control sample is corrected for contamination from Z → μ þ μ − γ decays. The pdfs of these kinematic and isolation variables are sampled to generate an ensemble of pseudocandidates, each with a complete Qγ four-vector and an associated pair of correlated dimuon and photon isolation values. The nominal selection requirements are imposed on the ensemble and the surviving pseudocandidates are used to construct templates for the kinematic distributions, notably the inclusive QCD background m μμγ and p μμ T distributions. The background from Z → μ þ μ − γ decays is modeled with templates derived from a sample of simulated Z boson events with m μμ in the ϒðnSÞ mass region. To validate this background model with data, the sidebands of the m μμγ distribution in several validation regions, defined by relaxed kinematic or isolation requirements, are used to compare the prediction of the background model with the data. Good agreement within the statistical uncertainties is observed.
The composition of the inclusive QCD background and the Z → μ þ μ − γ decay contribution is investigated with data. The details of the composition do not enter directly the background estimation for this search, but the composition itself is a crucial input in feasibility studies for future searches or measurements, where projections of these backgrounds to different center-of-mass energies or luminosity conditions are needed. To facilitate this study, the selection requirements on m μμ and jL xy =σ L xy j are relaxed to include the sideband regions. In the J=ψγ final state, a simultaneous unbinned maximum likelihood fit to the m μμ and jL xy =σ L xy j distributions is performed. Once the simultaneous fit is performed, the composition of the subset of events satisfying the nominal m μμ and jL xy =σ L xy j requirements is estimated. After the complete event selection, around 56% of the events originate from prompt J=ψ production, 3% from nonprompt J=ψ production (from b-hadron decays) and 41% are combinatoric backgrounds from nonresonant dimuon events.
A separate simultaneous fit to the m μμγ and m μμ distributions of the same sample of candidate J=ψ events finds no significant contribution from Z → μ þ μ − γ decays, a conclusion that is also supported by a study based on simulated Z → μ þ μ − events. For the ϒðnSÞγ final state a simultaneous fit is performed to the m μμγ and m μμ distributions. After the full event selection, inclusive ϒðnSÞ production accounts for 7% of events, 27% of the events are produced in Z → μ þ μ − γ decays, and 66% of the events are associated with combinatoric backgrounds from nonresonant dimuon events. The contribution from Z → μ þ μ − γ decays is in agreement with the MC expectation.
Trigger efficiencies and efficiencies for muon and photon identification are determined from samples of Z → ll, Z → llγ (l ¼ e; μ), and J=ψ → μ þ μ − decays in data [43,47]. The systematic uncertainty on the expected signal yield associated with the trigger efficiency is estimated to be 1.7%. The photon (both converted and unconverted) and muon reconstruction and identification efficiency uncertainties are estimated to be 0.5% (0.7%) and 0.4% (0.4%) for the Higgs boson (Z boson) signal, respectively. An uncertainty on the integrated luminosity of 2.8% is derived using the method described in Ref. [48]. The photon energy scale uncertainty, determined from Z → e þ e − and validated using Z → llγ decays [49], is propagated through the simulated signal samples as a function of η γ and p γ T . The uncertainty associated with the description of the photon energy scale in the simulation is found to be less than 0.2% of the three-body invariant mass while the uncertainty associated with the photon energy resolution is found to be negligible relative to the overall three-body invariant mass resolution. Similarly, the systematic uncertainty associated with the muon momentum measurement is determined using data samples of J=ψ → μ þ μ − and Z → μ þ μ − decays and validated using ϒðnSÞ → μ þ μ − decays [43]. For the p T range relevant to this analysis, the systematic uncertainties associated with the muon momentum scale are negligible.
The uncertainty in the shape of the inclusive QCD background is estimated through the study of variations in the background modeling procedure. The shape of the pdf is allowed to vary around the nominal shape within an envelope associated with shifts in the p μμ T and p γ T distributions. Furthermore, a separate background model, generated without removing the contamination from Z → μ þ μ − γ decays, provides an upper bound on potential mismodeling associated with this process.
Results are extracted by means of a simultaneous unbinned maximum likelihood fit, performed to the selected events with 30 GeV < m μμγ < 230 GeV separately in each of the analysis categories. In the J=ψγ final state, the fit is performed on the m μμγ and p μμγ T distributions, while for the ϒðnSÞγ candidates a similar fit is performed using the m μμγ , p μμγ T , and m μμ distributions. The latter distribution provides discrimination between the three ϒðnSÞ states and constrains the Z → μ þ μ − γ background normalization. No significant Z → Qγ or H → Qγ signals are observed, as shown in Figs. 1 and 2.
Upper limits on the branching fractions for the Higgs and Z boson decays to J=ψγ and ϒðnSÞγ are set using the CL s modified frequentist formalism [50] with the profile likelihood ratio test statistic [51]. The expected SM production cross sections are assumed for the Higgs and Z bosons. The results are summarized in Table II.
The 95% C.L. upper limit on the branching fraction for H → J=ψγ decays corresponds to about 540 times the expected SM branching fraction. The upper limits on the Z → J=ψγ and Z → ϒðnSÞγ branching fractions significantly constrain the allowed range of values obtained from theoretical calculations [12][13][14]. Upper limits are also set on the combined branching fractions B½H → ϒðnSÞγ < 2.0 × 10 −3 and B½Z → ϒðnSÞγ < 7.9 × 10 −6 , where the relative contribution of each final state to the potential