Study of $b\to c$ interference in the decay $B^{-}\to [K^{+}{\pi }^{-}{]}_D{K}^{*-}$

Using a sample of $232\times {10}^6{\rm Y}(4S)\to B\overline{B}$ events collected with the BABAR detector at the PEP-II $B$ factory we study the decay $B^{-}\to [K^{+}{\pi }^{-}{]}_D{K}^{*-}$ where the $K^{+}{\pi }^{-}$ is either from a Cabibbo-favored ${\overline{D}}^0$ decay or doubly-suppressed $D^0$ decay. We measure two observables that are sensitive to the Cabibbo-Kobayashi-Maskawa angle $\gamma$; the ratio $R$ of the charge-averaged branching fractions for the suppressed and favored decays; and the charge asymmetry $A$ of the suppressed decays: $R=0.046\pm 0.031(\mathrm{stat})\pm 0.008(\mathrm{syst})$, $A=-0.22\pm 0.61(\mathrm{stat})\pm 0.17(\mathrm{syst})$.

Figure 1

The result of the neural network training and verification (see text). The training samples are shown as histograms. The signal (Monte Carlo simulation) is the shaded histogram peaking to the right; the background (off-peak data recorded 40 MeV below the resonance) is the histogram with a peak near 0. The data samples used to check the NN are overlaid as data points. The vertical bar and the arrow indicate the requirement used to select signal candidates.

Figure 2

Distributions of $m_{\mathrm{ES}}$ for the wrong-sign (top) and right-sign (bottom) decays. These decay categories are defined in the text. The curves result from a simultaneous fit to these distributions with identical PDFs for both samples.

Figure 3

Result of a simultaneous fit to the wrong-sign and right-sign $m_{\mathrm{ES}}$ distributions with identical PDFs for both samples. Here the wrong-sign sample shown in the top plot of Fig. 2 is split by charge to measure $A$. The upper plot shows the $m_{\mathrm{ES}}$ distribution of the $B^{+}\to [K^{-}{\pi }^{+}{]}_D{K}^{*+}$ decays while the lower plot presents the same for the $B^{-}\to [K^{+}{\pi }^{-}{]}_D{K}^{*-}$ decays. The curves are the results of the fit.

Figure 4

Constraints on $r_B$. The BABAR $B^{-}\to D_{CP}{K}^{*-}$ (GLW) [13] result is combined with this analysis. The dashed (dotted) curve shows 1 minus the confidence level to exclude the abscissa value as a function of $r_B$ derived from the GLW (ADS) only measurements. When both the GLW and ADS results are combined the curve above the shaded area is obtained. Horizontal lines show the exclusion limits at the 1, 2, and 3 standard deviation levels.

Figure 5

1 minus exclusion confidence level curve for $\gamma$ obtained from the BABAR $B^{-}\to D_{CP}{K}^{*-}$ (GLW) result combined with this analysis. The graphical conventions are described in the caption of Fig. 4.