Search for the decay $\eta'\to\gamma\gamma\eta$

Using a data sample of $1.31\times10^{9} ~J/\psi$ events collected with the BESIII detector, a search for $\eta'\to\gamma\gamma\eta$ via $J/\psi\to\gamma\eta'$ is performed for the first time. No significant $\eta'$ signal is observed in the $\gamma\gamma\eta$ invariant mass spectrum, and the %upper limit of the branching fraction of $\eta'\to\gamma\gamma\eta$ is determined to be less than $1.33 \times 10^{-4}$ at the 90$\%$ confidence level.


I. INTRODUCTION
The η ′ meson provides a unique opportunity for understanding the distinct symmetry-breaking mechanisms present in low-energy Quantum Chromodynamics (QCD) [1]- [5], and its decays play an important role in exploring the effective theory of QCD at low energy [6].
Unlike η ′ → γγπ 0 decay, the η ′ → γγη decay has not been observed to date.The most stringent upper limit, reported by GAMS-4π setup, on the branching fraction of this decay is 8 × 10 −4 at the 90% confidence level (CL) [10].The BESIII experiment using J/ψ radiative decays has observed a series of η ′ new decay modes [11]- [17], and in this paper we present a search for η ′ → γγη in the J/ψ radiative decay.

II. DETECTOR AND MONTE CARLO SIMULATION
The BESIII detector is a magnetic spectrometer [18] located at the Beijing Electron Position Collider (BEPCII) [19].The cylindrical core of the BESIII detector consists of a helium-based multilayer drift chamber (MDC), a plastic scintillator time-of-flight system (TOF), and a CsI(Tl) electromagnetic calorimeter (EMC), which are all enclosed in a superconducting solenoidal magnet providing a 1.0 T (0.9 T in 2012) magnetic field.The solenoid is supported by an octagonal flux-return yoke with resistive plate counter muon identifier modules interleaved with steel.The acceptance of charged particles and photons is 93% over 4π solid angle.The charged-particle momentum resolution at 1 GeV/c is 0.5%, and the dE/dx resolution is 6% for the electrons from Bhabha scattering.The EMC measures photon energies with a resolution of 2.5% (5%) at 1 GeV in the barrel (end cap) region.The time resolution of the TOF barrel part is 68 ps, while that of the end cap part is 110 ps.
Monte Carlo (MC) simulations are used to estimate backgrounds and determine the detection efficiencies.The geant4-based [20] simulation software boost [21] includes the geometric and material description of the BESIII detector, detector response, and digitization models, as well as the tracking of the detector running conditions and performance.Production of the charmonium state J/ψ is simulated with kkmc [22], while the decays are generated with evtgen [23] for known decay modes with branching fractions taken from the Particle Data Group (PDG) [24] and by lundcharm [25] for the remaining unknown decays.
In this analysis, the program evtgen is used to generate a J/ψ → γη ′ MC sample with an angular distribution of 1 + cos 2 θ γ , where θ γ is the polar angle of the radiative photon in the J/ψ rest frame.The decays η ′ → γω (ρ), ω (ρ) → γη are generated using the VMD model [7,8] with ω (ρ) exchange.For the non-resonant η ′ → γγη decay, the VMD model is also used to generate the MC sample with ω(1420) or ρ(1450) exchange.We use a sample of 1.225 × 10 9 simulated J/ψ events to study the backgrounds in which the J/ψ decays generically (inclusive MC sample).The analysis is performed in the framework of the BESIII offline software system [26] which incorporates the detector calibration, event reconstruction, and data storage.

III. EVENT SELECTION AND BACKGROUND ESTIMATION
In the reconstruction of J/ψ → γη ′ with η ′ → γγη and η → γγ, candidate events must have no charged particle and at least five photons.Charged particles are identified by tracks in the active region of the MDC, corresponding to | cos θ| < 0.93, where θ is the polar angle of the charged track with respect to the beam direction.They are also required to pass within ±10 cm of the interaction point in the beam direction and 1 cm of the beam line in the plane perpendicular to the beam.The photon candidate showers must have minimum energy of 25 MeV in the barrel region (| cos θ| < 0.8) or 50 MeV in the end cap region (0.86 < | cos θ| < 0.92).Showers in the region between the barrel and the end caps are poorly measured and excluded.A requirement of EMC cluster timing with respect to the most energetic photon (−500 ns < T < 500 ns) is used to suppress electronic noise and energy deposits unrelated to the event.To select J/ψ → γη ′ , η ′ → γγη (η → γγ) signal events, only the events with exactly five photon candidates are selected, and the most energetic photon is taken as the radiative photon from the J/ψ decay.
A four-constraint (4C) kinematic fit imposing energymomentum conservation is performed to the γγγγγ hypothesis and the χ 2 is required to be less than 200.To distinguish the photons from η ′ and η decays, a variable ) 2 , is introduced, where M (γγη) is the invariant mass of four of five selected photons (expect for the radiative photon) for reconstructing the η ′ meson, M (γγ) is the invariant mass of photon pairs for reconstructing the η meson, while σ 1 = 11.7 MeV/c 2 and σ 2 = 9.7 MeV/c 2 are the mass resolutions of η ′ and η, respectively, obtained from the MC simulations, m(η ′ ) and m(η) are the η ′ and η nominal masses, respectively.We then require |M (γγ) − m(η)| < 50 MeV/c 2 and the combination with the minimum value of δ 2 η ′ η is chosen.Next the δ 2 η ′ η is required to be less than δ 2 ηη , which is defined as ) 2 , where M 1 (γγ) and M 2 (γγ) are the invariant masses of arbitrary two of five selected photons, to suppress the background events from J/ψ → γηη.
To improve the mass resolution and further suppress background events, a five-constraint (5C) kinematic fit imposing energy-momentum conservation with a η mass constraint is performed under the γγγη hypothesis, where the η candidate is reconstructed with the pair of photons described above, and the χ 2 5C is required to be less than 30.The χ 2 5C distribution is shown in Fig. III.In addition, the invariant masses of all the two photon pairs are required not to be in the π 0 mass region, |M (γγ) − m(π 0 )| > 18 MeV/c 2 , to suppress the background events with π 0 in the final state.
To remove the mis-combinationed photon pairs in η candidates, the η decay angle θ decay , defined as the polar angle of each photon in the corresponding γγ rest frame with respect to the η direction in the J/ψ rest frame, is required to satisfy | cos θ decay | < 0.95.An event is vetoed if any two of five selected photons (except for the combination for the η candidate) satisfy The resulting γγη invariant mass distribution, after these requirements, is shown in Fig. IV(a), where no significant η ′ peak is observed.Detailed MC studies indicate that the background events accumulating near the lower side of the η ′ signal region are mainly from J/ψ → γη ′ , η ′ → π 0 π 0 η (Class I), which is shown as the dotted (green) curve in Fig. IV(a).The peaking background is from J/ψ → γη ′ , η ′ → γω, ω → γπ 0 , and is shown as the solid area in Fig. IV(a).The remaining background events are dominated by those J/ψ decays without η ′ in the states (Class II), e.g., J/ψ → γηπ 0 and J/ψ → ωη (ω → γπ 0 , η → γγ) decays.They constitute a smooth distribution in the η ′ signal region as illustrated by the dashed (pink) curve in Fig. IV(a).5C distributions in MC simulations and data.Dots with error bars are data, the wide (blue) solid-curve is the sum of expected backgrounds from MC simulations, the grid area is from signal MC with arbitrary normalization, the (green) dotted-curve is the Class I (J/ψ → γη ′ , η ′ → π 0 π 0 η) background, the (pink) dashed-curve is the Class II (J/ψ → γηπ 0 and J/ψ → ωη (ω → γπ 0 , η → γγ)) background, and solid area is the peaking background.

IV. SIGNAL YIELD AND BRANCHING FRACTION
An unbinned maximum likelihood fit to the M (γγη) distribution is performed to determine the η ′ → γγη signal yield.In the fit, the probability density function (PDF) for the signal component is represented by the signal MC shape, which is obtained from the signal MC sample generated with an incoherent mixture of ρ, ω and the non-resonant components according to the fractions from the theoretical prediction [7,8].The Class I and Class II background shapes are obtained from MC simulations and fixed, but the numbers are free parameters.Both the shape and the yield for the peaking background are fixed to the MC simulation and their expected intensities.The fit shown in Fig. IV(a) yields 24.9 ± 10.3 η ′ → γγη events with a statistical significance of 2.6σ, and the branching fraction is calculated from where N obs is the number of observed events determined from the fit to the γγη mass spectrum, ε is the MCdetermined detection efficiency, which is obtained from the signal MC sample described above; B(η → γγ) and B(J/ψ → γη ′ ) are the branching fractions of η → γγ and J/ψ → γη ′ quoted from the PDG [24], respectively.
With the number of signal events and a detection efficiency of 11.4% the branching fraction is measured to be where the first uncertainty is statistical and the second systematic.

V. SYSTEMATIC UNCERTAINTIES
The systematic uncertainties on the upper limit measurement are summarized in Table I.The uncertainty due to the photon reconstruction is determined to be 0.5% per photon in the EMC barrel and 1.5% per photon in the EMC endcap [27].Thus the uncertainty associated with the five reconstructed photons is 3% (0.6% per photon) by weighting the uncertainties according to the polar angle distribution of the five photons from data.The uncertainties associated with the other selection criteria, e.g., kinematic fit with χ 2 5C < 30, the number of photons equal to 5, π 0 veto (|M (γγ) − m(π 0 )| > 18MeV/c 2 ) and cos θ decay , are studied using the J/ψ → γη ′ → γγω, ω → γπ 0 decay control sample [9].The systematic uncertainty for each of the applied selection criteria is numerically estimated from the difference of the number of events with and without the corresponding requirement.The resultant efficiency differences between data and MC simulations (2.7%, 0.5%, 1.9%, and 0.3%, respectively) are taken as the corresponding systematic uncertainties.2)).The arrow is the position of the upper limit on the signal yields at 90% CL.
To suppress the multi-η backgrounds and remove the mis-combinations of photons, an event is vetoed if any two of five selected photons (except for the combination for the η candidate) satisfy |M (γγ)−m(η)| < 35 MeV/c 2 .To estimate the systematic uncertainty, this requirement varied by ± 10 MeV/c 2 , and the maximum change to the nominal result is taken as the systematic uncertainty.
The signal shape is obtained from the MC simulation in the nominal fit for the η ′ decay.The uncertainty due to the signal shape is considered by convolving a Gaussian function to account for the difference in the mass resolution between data and MC simulation.In the fit to the γγη distribution, the signal PDF is the signal MC shape convolving a Gaussian function with a fixed width of 1.5 MeV [9], and the changes of the signal yields is taken as the uncertainty due to the signal shape.
The uncertainty due to the Class I background and the peaking background are estimated by varying the numbers of expected background events by one standard deviation according to the uncertainties on the branching fractions values in PDG [24].
To take into account the systematic uncertainty due to signal model (VMD model), fits with alternative signal models for the different components, for example, a coherent sum for the ρ, ω−components and a uniform angular distribution in phase space for the nonresonant process is performed.The resultant changes in the branching fractions (involving efficiency changes) are taken as the uncertainty related to the signal model.
To take into account the systematic uncertainties associated with the fit of the mass spectrum coming from the background events and the fit range, alternative fits with different fit ranges, background shapes and the number of background events are performed.The largest number of the signal yield among these cases is chosen to calculate the upper limit of the branching fraction at the 90% CL.
Assuming all systematic uncertainties in Table I are independent, the total systematic uncertainty, obtained from their quadratic sum, is 8.7%.Since no significant η ′ peak is seen, we use the Bayesian method to obtain the signal upper limit.Unbinned maximum likelihood fits are performed on the γγη mass spectrum with a series of input signal events, and the distribution of normalized likelihood values is taken as the PDF for the expected number of events.
The final upper limit on the branching fraction is determined by convolving the normalized likelihood curve L(N ) with the systematic uncertainties as a Gaussian function (G(µ, σ) = G(0, σ sys )) to obtain the smeared likelihood L ′ (N ′ ), which is written as where σ sys = N • σ rel , N and σ rel are the input signal yield and the corresponding uncertainty, respectively.Figure IV(b) shows the likelihood distribution before and after convolving the Gaussian function.The upper limit on the number of η ′ → γγη events, N ′ UL , is determined to be 40 at the 90% CL .The corresponding upper limit on the branching fraction of η ′ → γγη is determined to be B(η ′ → γγη) < 1.33 × 10 −4 at the 90% CL.

VII. SUMMARY
With a data sample of 1.31 ×10 9 J/ψ events collected with the BESIII detector, we report on a search for the doubly radiative decay η ′ → γγη for the first time, where the η ′ meson is produced via the J/ψ → γη ′ process.The observed signal yields in the γγη invariant mass spectrum corresponds to 2.6σ, this signal corresponds to a branching fraction of (8.25±3.41±0.72)×10−5 .We also present an upper limit of the branching fraction of 1.33 × 10 −4 at the 90% CL.The obtained result is in tension with a recent theoretical prediction of 2.0 × 10 −4 [8] within the frame work of the linear σ model and the VMD model.

FIG. 2 .
FIG. 2. (a) Results of the fit to M (γγη).The black dots with error bars are data, and the others are the results of the fit described in the text.(b) Likelihood distribution before (black dots) and after (blue squares) taking into account systematic uncertainties (see Eq. (2)).The arrow is the position of the upper limit on the signal yields at 90% CL.

TABLE I .
Summary of relative systematic uncertainties for the upper limit on the branching fraction measurement (in %).