Learning to classify from impure samples with high-dimensional data

A persistent challenge in practical classification tasks is that labeled training sets are not always available. In particle physics, this challenge is surmounted by the use of simulations. These simulations accurately reproduce most features of data, but cannot be trusted to capture all of the complex correlations exploitable by modern machine learning methods. Recent work in weakly supervised learning has shown that simple, low-dimensional classifiers can be trained using only the impure mixtures present in data. Here, we demonstrate that complex, high-dimensional classifiers can also be trained on impure mixtures using weak supervision techniques, with performance comparable to what could be achieved with pure samples. Using weak supervision will therefore allow us to avoid relying exclusively on simulations for high-dimensional classification. This work opens the door to a new regime whereby complex models are trained directly on data, providing direct access to probe the underlying physics.

Figure 1

The AUC and training time of CWoLa (solid) and LLP (dashed) as the batch size is varied. Training times are measured on an NVIDIA Tesla K80 GPU using CUDA 8.0, TensorFlow 1.4.1, and Keras 2.1.2. AUC is a measure of classifier performance and is 1 for a perfect classifier and 0.5 for a completely random one.

Figure 2

Classifier performance (AUC) shown for both CWoLa (solid) and LLP (dashed) trained on two mixed samples with various signal fractions $f_1$, $1-f_1$ as the number of training data is varied between 100k and 1M. Each training is repeated 10 times and the 25th, 50th, and 75th percentiles are shown. The $f_1=0.0$ CWoLa curve corresponds to full supervision. CWoLa outperforms LLP by this metric, though both methods work quite well.