Measurements of Branching Fractions, Polarizations, and Direct $CP$-Violation Asymmetries in $B\to \rho K^{*}$ and $B\to f_0(980)K^{*}$ Decays

We report searches for $B$-meson decays to the charmless final states $\rho K^{*}$ and $f_0(980)K^{*}$ with a sample of $232\times {10}^6$ $B\overline{B}$ pairs collected with the BABAR detector at the PEP-II $e^{+}{e}^{-}$ collider. We measure in units of ${10}^{-6}$ the following branching fractions, where the first error quoted is statistical and the second systematic, or upper limits are given at the 90% confidence level : $\mathcal{B}(B^{+}\to {\rho }^0{K}^{*+})<6.1$, $\mathcal{B}(B^{+}\to {\rho }^{+}{K}^{*0})=9.6\pm 1.7\pm 1.5$, $\mathcal{B}(B^0\to {\rho }^{-}{K}^{*+})<12.0$, $\mathcal{B}(B^0\to {\rho }^0{K}^{*0})=5.6\pm 0.9\pm 1.3$, $\mathcal{B}\mathbf{(}{B}^{+}\to f_0(980)K^{*+}\mathbf{)}=5.2\pm 1.2\pm 0.5$, and $\mathcal{B}\mathbf{(}{B}^0\to f_0(980)K^{*0}\mathbf{)}<4.3$. For the significant modes, we also measure the fraction of longitudinal polarization and the charge asymmetry: $f_L(B^{+}\to {\rho }^{+}{K}^{*0})=0.52\pm 0.10\pm 0.04$, $f_L(B^0\to {\rho }^0{K}^{*0})=0.57\pm 0.09\pm 0.08$, ${\mathcal{A}}_{CP}(B^{+}\to {\rho }^{+}{K}^{*0})=-0.01\pm 0.16\pm 0.02$, ${\mathcal{A}}_{CP}(B^0\to {\rho }^0{K}^{*0})=0.09\pm 0.19\pm 0.02$, and ${\mathcal{A}}_{CP}\mathbf{(}{B}^{+}\to f_0(980)K^{*+}\mathbf{)}=-0.34\pm 0.21\pm 0.03$.

Figure 1

sPlots [13] for the invariant mass of $K\pi$ in ${\rho }^{+}{K}^{*0}$ (left) and $\pi \pi$ in ${\rho }^0/f_0(980)K^{*0}$ (right) up to the higher-mass regions. The points with error bars show the data, and the solid (dashed) lines show the projected PDFs of the signal and nonresonant background [nonresonant background only: $\rho K\pi$ in ${\rho }^{+}{K}^{*0}$; the sum of $f_0(1370)K^{*}$, $\pi \pi K^{*}$, and $\pi \pi K\pi$ in ${\rho }^0{K}^{*0}$]. The arrows show the nominal fit regions.

Figure 2

Projections of the multidimensional fit onto $m_{\mathrm{ES}}$ for events passing a signal-to-total likelihood probability ratio cut with the plotted variable excluded for (a) ${\rho }^0{K}^{*+}$, (b) ${\rho }^{+}{K}^{*0}$, (c) ${\rho }^{-}{K}^{*+}$, (d) ${\rho }^0{K}^{*0}$, (e) $f_0(980)K^{*+}$, and (f) $f_0(980)K^{*0}$. The points with error bars show the data; the solid, dashed, and dotted lines show the total, background, and continuum PDF projections, respectively.

Figure 3

Distribution of $-2\ln \mathcal{L}(\mathcal{B},f_L)$ for $B^{+}\to {\rho }^{+}{K}^{*0}$ (left) and $B^0\to {\rho }^0{K}^{*0}$ (right) decays. The solid dots correspond to the central values and the curves give contours in $\Delta \sqrt{-2\ln \mathcal{L}(\mathcal{B},f_L)}=1$ steps.