SCS classes of diagrams contributing to the decays $ D ^0 \!\rightarrow K ^0_{\rm\scriptscriptstyle S} K ^\pm \pi ^\mp $ . The color-favored (tree) diagrams (a) contribute to the $ K ^{*{\pm}}_{0,1,2} \!\rightarrow K ^0_{\rm\scriptscriptstyle S} \pi ^\pm $ and $ ( a ^{{}}_{0,2} ,\rho )^{\pm}\!\rightarrow K ^0_{\rm\scriptscriptstyle S} K ^\pm $ channels, while the color-suppressed exchange diagrams (b) contribute to the $ ( a ^{{}}_{0,2} ,\rho )^{\pm}\!\rightarrow K ^0_{\rm\scriptscriptstyle S} K ^\pm $ , $ K ^{*{0}}_{0,1,2} \!\rightarrow K ^+ \pi ^- $ and $\overline{ K }{} ^{*{0}}_{0,1,2} \!\rightarrow K ^- \pi ^+ $ channels. Second-order loop (penguin) diagrams (c) contribute to the $ ( a ^{{}}_{0,2} ,\rho )^{\pm}\!\rightarrow K ^0_{\rm\scriptscriptstyle S} K ^\pm $ and $ K ^{*{\pm}}_{0,1,2} \!\rightarrow K ^0_{\rm\scriptscriptstyle S} \pi ^\pm $ channels, and, finally, OZI-suppressed penguin annihilation diagrams (d) contribute to all decay channels.

Mass (left) and $\Delta m$ (right) distributions for the $ D ^0 \!\rightarrow K ^0_{\rm\scriptscriptstyle S} K ^- \pi ^+ $ (top) and $ D ^0 \!\rightarrow K ^0_{\rm\scriptscriptstyle S} K ^+ \pi ^- $ (bottom) samples with fit results superimposed. The long-dashed (blue) curve represents the \Dsttt signal, the dash-dotted (green) curve represents the contribution of real $ D ^0$ mesons combined with incorrect \pislow and the dotted (red) curve represents the combined combinatorial and $ D ^0 \!\rightarrow K ^0_{\rm\scriptscriptstyle S} \pi ^+ \pi ^- \pi ^0 $ background contribution. The vertical solid lines show the signal region boundaries, and the vertical dotted lines show the sideband region boundaries.

Dalitz plots of the $ D ^0 \!\rightarrow K ^0_{\rm\scriptscriptstyle S} K ^- \pi ^+ $ (left) and $ D ^0 \!\rightarrow K ^0_{\rm\scriptscriptstyle S} K ^+ \pi ^- $ (right) candidates in the two-dimensional signal region.

Efficiency function used in the isobar model fits, corresponding to the average efficiency over the full dataset. The coordinates \sKSpi and $ m^{2}_{ K ^0_{\rm\scriptscriptstyle S} K }$ are used to highlight the approximate symmetry of the efficiency function. The $z$ units are arbitrary.

Distributions of \sKpi (upper left), \sKSpi (upper right) and $ m^{2}_{ K ^0_{\rm\scriptscriptstyle S} K }$ (lower left) in the $ D ^0 \!\rightarrow K ^0_{\rm\scriptscriptstyle S} K ^- \pi ^+ $ mode with fit curves from the best \glass model. The solid (blue) curve shows the full PDF $P_{ K ^0_{\rm\scriptscriptstyle S} K ^- \pi ^+ }(\sKSpi,\sKpi)$, while the other curves show the components with the largest integrated fractions.

Distributions of \sKpi (upper left), \sKSpi (upper right) and $ m^{2}_{ K ^0_{\rm\scriptscriptstyle S} K }$ (lower left) in the $ D ^0 \!\rightarrow K ^0_{\rm\scriptscriptstyle S} K ^- \pi ^+ $ mode with fit curves from the best \lassc model. The solid (blue) curve shows the full PDF $P_{ K ^0_{\rm\scriptscriptstyle S} K ^- \pi ^+ }(\sKSpi,\sKpi)$, while the other curves show the components with the largest integrated fractions.

Distributions of \sKpi (upper left), \sKSpi (upper right) and $ m^{2}_{ K ^0_{\rm\scriptscriptstyle S} K }$ (lower left) in the $ D ^0 \!\rightarrow K ^0_{\rm\scriptscriptstyle S} K ^+ \pi ^- $ mode with fit curves from the best \glass model. The solid (blue) curve shows the full PDF $P_{ K ^0_{\rm\scriptscriptstyle S} K ^+ \pi ^- }(\sKSpi,\sKpi)$, while the other curves show the components with the largest integrated fractions.

Distributions of \sKpi (upper left), \sKSpi (upper right) and $ m^{2}_{ K ^0_{\rm\scriptscriptstyle S} K }$ (lower left) in the $ D ^0 \!\rightarrow K ^0_{\rm\scriptscriptstyle S} K ^+ \pi ^- $ mode with fit curves from the best \lassc model. The solid (blue) curve shows the full PDF $P_{ K ^0_{\rm\scriptscriptstyle S} K ^+ \pi ^- }(\sKSpi,\sKpi)$, while the other curves show the components with the largest integrated fractions.

Decay rate and phase variation across the Dalitz plot. The top row shows $|\mcM_{ K ^0_{\rm\scriptscriptstyle S} K ^\pm \pi ^\mp }(\sKSpi, \sKpi)|^2$ in the best \glass isobar models, the center row shows the phase behavior of the same models and the bottom row shows the same function subtracted from the phase behavior in the best \lassc isobar models. The left column shows the $ D ^0 \!\rightarrow K ^0_{\rm\scriptscriptstyle S} K ^- \pi ^+ $ mode with $ D ^0 \!\rightarrow K ^0_{\rm\scriptscriptstyle S} K ^+ \pi ^- $ on the right. The small inhomogeneities that are visible in the bottom row relate to the \glass and \lassc models preferring slightly different values of the $ K ^{*}_{{}}({892})^{{\pm}}$ mass and width.

Comparison of the phase behavior of the various \kpswave parameterizations used. The solid (red) curve shows the \lassc parameterization, while the dashed (blue) and dash-dotted (green) curves show, respectively, the \glass functional form fitted to the charged and neutral $S$-wave channels. The final two curves show the \glass forms fitted to the charged \kpswave in $ D ^0 \!\rightarrow K ^0_{\rm\scriptscriptstyle S} \pi ^+ \pi ^- $ decays in Ref. \cite{Aubert:2008bd} (triangular markers, purple) and Ref. \cite{delAmoSanchez:2010xz} (dotted curve, black). The latter of these was used in the analysis of $ D ^0 \!\rightarrow K ^0_{\rm\scriptscriptstyle S} K ^\pm \pi ^\mp $ decays by the CLEO collaboration \cite{Insler:2012pm}.

Smooth functions, $c_{ K ^0_{\rm\scriptscriptstyle S} K ^\pm \pi ^\mp }( m^{2}_{ K ^0_{\rm\scriptscriptstyle S} K } , \sKSpi)$, used to describe the combinatorial background component in the $ D ^0 \!\rightarrow K ^0_{\rm\scriptscriptstyle S} K ^- \pi ^+ $ (left) and $ D ^0 \!\rightarrow K ^0_{\rm\scriptscriptstyle S} K ^+ \pi ^- $ (right) amplitude model fits.

Two-dimensional quality-of-fit distributions illustrating the dynamic binning scheme used to evaluate $\chi^2$ . The variable shown is $\frac{d_i - p_i}{\sqrt{p_i}}$ where $d_i$ and $p_i$ are the number of events and the fitted value, respectively, in bin $i$. The $ D ^0 \!\rightarrow K ^0_{\rm\scriptscriptstyle S} K ^- \pi ^+ $ ( $ D ^0 \!\rightarrow K ^0_{\rm\scriptscriptstyle S} K ^+ \pi ^- $ ) mode is shown in the left (right) column, and the \glass (\lassc) isobar models are shown in the top (bottom) row.

Animated gif made out of all figures.

Signal yields and estimated background rates in the two-dimensional signal region. The larger mistag rate in the $ D ^0 \!\rightarrow K ^0_{\rm\scriptscriptstyle S} K ^+ \pi ^- $ mode is due to the different branching fractions for the two modes. Only statistical uncertainties are quoted.

Blatt-Weisskopf centrifugal barrier penetration factors, $B_J(q, q_0, d)$ \cite{BlattWeisskopf}.

Angular distribution factors, $\Omega_J(p_{ D ^0 } + p_{ C }, p_{ B } - p_{ A })$. These are expressed in terms of the tensors $T^{\mu\nu} = -g^{\mu\nu} + \frac{p^\mu_{ A B }p^\nu_{ A B }}{\mRsq}$ and $T^{\mu\nu\alpha\beta} = \frac{1}{2}(T^{\mu\alpha}T^{\nu\beta} + T^{\mu\beta}T^{\nu\alpha}) - \frac{1}{3}T^{\mu\nu}T^{\alpha\beta}$.

Particle ordering conventions used in this analysis.

Modulus and phase of the relative amplitudes between resonances that appear in both the $ D ^0 \!\rightarrow K ^0_{\rm\scriptscriptstyle S} K ^- \pi ^+ $ and $ D ^0 \!\rightarrow K ^0_{\rm\scriptscriptstyle S} K ^+ \pi ^- $ modes. Relative phases are calculated using the value of $\delta_{ K ^0_{\rm\scriptscriptstyle S} K \pi }$ measured in $\psi (3770)$ decays \cite{Insler:2012pm}, and the uncertainty on this value is included in the statistical uncertainty. The first uncertainties are statistical and the second systematic.

Values of $\chi^2/\mathrm{bin}$ indicating the fit quality obtained using both \kpswave parameterizations in the two decay modes. The binning scheme for the $ D ^0 \!\rightarrow K ^0_{\rm\scriptscriptstyle S} K ^- \pi ^+ $ ( $ D ^0 \!\rightarrow K ^0_{\rm\scriptscriptstyle S} K ^+ \pi ^- $ ) mode contains 2191 (2573) bins.

Coherence factor observables to which the isobar models are sensitive. The third column summarizes the CLEO results measured in quantum-correlated decays \cite{Insler:2012pm}, where the uncertainty on $\delta_{ K ^0_{\rm\scriptscriptstyle S} K \pi } - \delta_{ K ^* K }$ is calculated assuming maximal correlation between $\delta_{ K ^0_{\rm\scriptscriptstyle S} K \pi }$ and $\delta_{ K ^* K }$.

$\mathrm{SU}(3)$ flavor symmetry predictions \cite{Bhattacharya:2012fz} and results. The uncertainties on phase difference predictions are calculated from the quoted magnitude and phase uncertainties. Note that some theoretical predictions depend on the $\eta $ -- $\eta ^{\prime}$ mixing angle $\theta_{\eta -\eta ^{\prime} }$ and are quoted for two different values. The bottom entry in the table relies on the CLEO measurement \cite{Insler:2012pm} of the coherence factor phase $\delta_{ K ^0_{\rm\scriptscriptstyle S} K \pi }$, and the uncertainty on this phase is included in the statistical uncertainty, while the other entries are calculated directly from the isobar models and relative branching ratio. Where two uncertainties are quoted the first is statistical and the second systematic.

Nominal values for isobar model parameters that are fixed in the model fits, or used in constraint terms. These values are taken from Refs. \cite{PDG2014,Abele:1998qd,Bargiotti:2003ev,Dunwoodie} as described in Sect. ???.

Matrices $\mathbf{U}$ relating the fit coordinates $\mathbf{b'}$ to the \lassc form factor coordinates $\mathbf{b}=\mathbf{Ub'}$ defined in Sect. ???.

Additional fit parameters for \glass models. This table does not include parameters that are fixed to their nominal values. The first uncertainties are statistical and the second systematic.

Additional fit parameters for \lassc models. This table does not include parameters that are fixed to their nominal values. The first uncertainties are statistical and the second systematic.

Change in \ntll value when removing a $\rho $ resonance from one of the models.

Listing of abbreviations required to typeset the systematic uncertainty tables.

Lookup table filenames.

Supplementary material full pdf

This .zip archive contains supplementary material for the publication LHCb-PAPER-2015-026. The files contained are: supplementary.pdf : An overview of the extra figures *.pdf, *.png, *.eps : The figures in various formats