Generalization in Quantum Machine Learning: A Quantum Information Standpoint

Quantum classification and hypothesis testing (state and channel discrimination) are two tightly related subjects, the main difference being that the former is data driven: how to assign to quantum states $\rho (x)$ the corresponding class $c$ (or hypothesis) is learnt from examples during training, where $x$ can be either tunable experimental parameters or classical data “embedded” into quantum states. Does the model generalize? This is the main question in any data-driven strategy, namely the ability to predict the correct class even of previously unseen states. Here we establish a link between quantum classification and quantum information theory, by showing that the accuracy and generalization capability of quantum classifiers depend on the (Rényi) mutual information $I(C:Q)$ and $I_2(X:Q)$ between the quantum state space $Q$ and the classical parameter space $X$ or class space $C$. Based on the above characterization, we then show how different properties of $Q$ affect classification accuracy and generalization, such as the dimension of the Hilbert space, the amount of noise, and the amount of neglected information from $X$ via, e.g., pooling layers. Moreover, we introduce a quantum version of the information bottleneck principle that allows us to explore the various trade-offs between accuracy and generalization. Finally, in order to check our theoretical predictions, we study the classification of the quantum phases of an Ising spin chain, and we propose the variational quantum information bottleneck method to optimize quantum embeddings of classical data to favor generalization.

Popular Summary

Data-driven strategies are responsible for the tremendous advances in many fields such as pattern recognition, where the classification algorithm is directly ”learnt” from data rather than being built via human-designed mathematical models. Quantum machine learning is a rapidly evolving new discipline that applies data-driven strategies to the quantum realm, where faster algorithms and more accurate estimation methods are theoretically available. Nonetheless, much theoretical work is still needed to understand how to fully exploit genuinely quantum features, such as entanglement, the no-cloning theorem, and the destructive role of quantum measurements, for machine learning applications.

Here we use tools from quantum information theory to study the problem of generalization in quantum data classification, where the data are either already given in a quantum form or first embedded onto quantum states. We show that after ”training” a quantum classifier using a few examples, the classification error that we get with new, previously unseen data can be bounded by entropic quantities, more specifically by the mutual information between the quantum state space and other classical spaces. Loosely speaking, our theorem shows that a quantum classifier generalizes well when it is capable of discarding as much as possible of the information that is unnecessary to predict the class.

We apply our general results to study fundamental properties, e.g., introducing a quantum version of the bias-variance trade-off, and to derive new algorithms, such as what we call the variational quantum information bottleneck principle. Our predictions are then tested in numerical experiments.

Figure 1

(a) Example binary classification of quantum states that depend on some external parameters $x$. (b) Classification of classical data using a quantum embedding circuit with $L$ layers. The classical data are sampled from an unknown distribution $P(c,x)$, where $x$ describes, e.g., images of animals and $c$ specifies the kind of animal, e.g., a cat. The classical input $x$ is embedded into a quantum state $\rho (x)$ via layers of $x$-dependent and $x$-independent gates. A POVM $\{{\mathrm{\Pi }}_c\}$ is performed at the end of the circuit. The predicted class of $x$ corresponds to the measurement outcome $c$. (c) Any POVM can be expressed as a unitary circuit followed by a projective measurement $|c⟩⟨c|$ on a suitably large ancillary system. (d) In quantum channel discrimination, the images $x$ live in the physical world; a quantum probe senses the outside world and $\rho (x)$ is the scattered state of light collected by the detector, which depends on the outside objects. The detector then classifies the image with a POVM, as in (c).

Figure 2

Summary of the error sources for given parametric quantum states $\rho (x)$ and a finite number of training samples. The classification error $R({\mathrm{\Pi }}^{\ast })$ is the average loss with the unknown optimal measurement ${\mathrm{\Pi }}^{\ast }$. The average testing error $R({\mathrm{\Pi }}^{\mathcal{T}})$ replaces ${\mathrm{\Pi }}^{\ast }$ with the POVM ${\mathrm{\Pi }}^{\mathcal{T}}$ estimated from the training set $\mathcal{T}$ via empirical risk minimization. The testing error $R^{{\mathcal{T}}^{\prime }}({\mathrm{\Pi }}^{\mathcal{T}})$ is a finite sample approximation of $R({\mathrm{\Pi }}^{\mathcal{T}})$ over the testing set ${\mathcal{T}}^{\prime }$. The difference between the average testing error and the Bayes risk $R^{\mathrm{Bayes}}$ is split into the approximation error $\mathcal{A}$ and the generalization error ${\mathcal{G}}^{\mathcal{T}}$. The training error typically behaves as $R({\mathrm{\Pi }}^{\ast })$.

Figure 3

Summary of some of the main conclusions of this paper. The approximation and generalization errors are respectively mathematically related to the training and testing errors over some datasets, and they cannot be simultaneously minimized (bias-variance trade-off). We use quantum information quantities to bound these errors and show how they are affected by the dimensionality of the quantum Hilbert space, noise, and “information pooling.”

Figure 4

(a) Example data distributions for two different classes $c=\{0,1\}$: Gaussian distributions with different means $\pm 10$ and the same standard deviation $7$. The corresponding Bayes risk is $R^{\mathrm{Bayes}}\simeq 7.6\mathrm{\%}$. (b) Average classification error $R$ (5) versus generalization bound $\mathcal{B}$ (10) for the angle encoding $\rho (x)=|\psi (x)⟩⟨\psi (x){|}^{\otimes N_Q}$, where $|\psi (x)⟩=\cos (x/2)|0⟩+\sin (x/2)|1⟩$ and the $x$ values have been normalized such that $x\in [0,2\pi ]$. The number inside each circle represents the value of $N_Q=1,\dots ,10$.

Figure 5

Geometric visualization of a good embedding, where each point in the sphere represents a state. Points belonging to different classes are plotted with different colors. A good embedding should cluster points with the same class and separate points belonging to different classes.

Figure 6

(a) Average risk (approximation error) $R$ (5) versus generalization error $\mathcal{B}$ (10) for the optimal embeddings obtained by solving the IB equations using either pure (26) or mixed (25) single-qubit states for different values of $\beta \in [1,3]$. Some values of $R$ and $\mathcal{B}$ for particular $\beta$ are also shown with markers. (b),(c) Bloch sphere visualizations of the quantum embeddings $\rho (x)$ for different values of $x$ sampled from either $P(x|0)$ (red) or $P(x|1)$ (yellow), obtained by solving the IB Eq. (26) for single-qubit encodings, using the two shown values of $\beta$. In all three subfigures, the data distributions are those of Fig. 4.

Figure 7

(a) Fidelity between two ground states of the quantum Ising model with different values of the magnetic field $h$ for $L=100$. The model displays a quantum phase transition at the critical value $h=1$, separating ordered ($|h|<1$) and disordered ($|h|>1$) phases. (b) Testing error in quantum phase recognition as a function of the magnetic field $h$. We use the fidelity classifier with a training set of $T$ random elements per phase. Each fidelity is estimated via a swap test with $S$ shots. For each $h$, the fidelity is calculated $1000$ times. Solid lines represent the mean fidelity, while shaded areas are the confidence intervals within a standard deviation.

Figure 8

(a) Union of the training and testing sets from a generated 2-moon dataset. Data (filled circles) with different colors belong to two different classes. Wrongly classified data in the testing set after training are marked with a red diagonal cross ($\beta =30$) or with a blue cross ($\beta =1.5$). (b) Fidelity $F$ between two embeddings for $\beta =30$, using the data from (a). Data points are ordered to first have all points from the first class and then all points from the second class. Dark black points represent $F\simeq 0$, while light yellow points represent $F\simeq 1$. (c) Fidelity $F$ between two embeddings, as in (b) but for $\beta =1.5$. White points represent $F=1$ while dark red points have infidelity $1-F\simeq {10}^{-7}$.