Reinforcement Learning with Neural Networks for Quantum Feedback

Machine learning with artificial neural networks is revolutionizing science. The most advanced challenges require discovering answers autonomously. In the domain of reinforcement learning, control strategies are improved according to a reward function. The power of neural-network-based reinforcement learning has been highlighted by spectacular recent successes such as playing Go, but its benefits for physics are yet to be demonstrated. Here, we show how a network-based “agent” can discover complete quantum-error-correction strategies, protecting a collection of qubits against noise. These strategies require feedback adapted to measurement outcomes. Finding them from scratch without human guidance and tailored to different hardware resources is a formidable challenge due to the combinatorially large search space. To solve this challenge, we develop two ideas: two-stage learning with teacher and student networks and a reward quantifying the capability to recover the quantum information stored in a multiqubit system. Beyond its immediate impact on quantum computation, our work more generally demonstrates the promise of neural-network-based reinforcement learning in physics.

Popular Summary

Artificial neural networks are revolutionizing science and technology, from image recognition and language processing to drug discovery and playing complex games. Here, we apply advanced techniques from the field of deep-reinforcement learning to solve formidable challenges in quantum physics. In particular, we demonstrate that artificial neural networks can discover novel strategies to improve the performance of quantum computing devices.

Building large-scale quantum computers is an outstanding challenge because of the fragility of qubits, the fundamental units of a quantum computer. While generic quantum error correction schemes have been invented, it is a critical open problem to find efficient strategies to protect qubits against noise in the specific hardware setting of an arbitrary real-world device. Discovering hardware-adapted strategies automatically using artificial neural networks may provide a solution to this problem, but it is a formidable task because of the huge search space of strategies.

Here, we employ reinforcement learning, a state-of-the-art approach widely seen as a promising building block for artificial intelligence, to make a neural network discover algorithms for quantum error correction for arbitrary hardware in a fully general way. Importantly, the neural network finds the strategies on its own, without human guidance. A key novelty of our work is the development of a measure that describes, at any time, how well the information stored in the quantum device is preserved, which elegantly helps to solve the large-search-space problem.

Given the flexibility of our approach in terms of hardware constraints, we expect our results to have an immediate impact across all fields that are striving to build quantum computers and simulators.

Figure 1

(a) The general setting of this work: A few-qubit quantum device with a neural-network-based controller whose task is to protect the quantum memory residing in this device against noise. Reinforcement learning lets the controller (“RL agent”) discover on its own how to best choose gate sequences, perform measurements, and react to measurement results by interacting with the quantum device (“RL environment”). (b) visualizes the flexibility of our approach (schematic). Depending on the type of noise and hardware setting, different approaches are optimal (DD, dynamical decoupling; DFS, decoherence-free subspace). By contrast, the RL approach is designed to automatically discover the best strategy adapted to the situation. In (c), we show the conventional procedure to select some QEC algorithm and then produce hardware-adapted device instructions (possibly reiterating until an optimal choice is found). We compare this to our approach (d) that takes care of all these steps at once and provides QEC strategies fully adapted to the concrete specifications of the quantum device.

Figure 2

The neural networks. (a) At each time $t$, the “state-aware” network receives a representation of the map $\mathrm{\Phi }$ describing the quantum evolution of arbitrary initial logical qubit states up to that time (represented by four evolved states $\widehat{\rho }$; see main text). It outputs the probabilities for the different actions (gates) defining the agent’s policy. Like any neural network, it is a nonlinear function that can be decomposed into layerwise linear superposition of neuron values using trainable weights (visualized by connections) and the application of nonlinear activation functions. Examples for actions are shown here (bit flip, cnot, measurement). Each of those can be applied to various qubits (or qubit pairs), resulting in approximately 10–20 actions. (b) The recurrent network receives the most recent measurement result (if any) and also outputs the action probabilities. Its long short-term memory (LSTM) neurons learn to keep track of information accumulated in previous time steps (schematically indicated by the recurrent connections here).

Figure 3

Reinforcement learning for the state-aware network with a four-qubit setup (all qubits can be measured, as indicated by the detectors and the red color; all qubit pairs can be subject to cnot gates, as indicated by the links). (a) Training progress in terms of the average recoverable quantum information ${\mathcal{R}}_Q$ evaluated at the final step of a 200-step gate sequence. One “epoch” involves training on a batch of 64 trajectories (gate sequences). Eventually, the network performs even better than a combination of encoding and periodic parity checks due to the adaptive recovery sequences that are triggered by unexpected measurements (i.e., upon error detection). In this example, that leads to an approximate 15% increase of the decoherence time over a nonadaptive scheme. (b) Typical gate sequences at different training stages, mostly converged strategy at bottom. Qubits participating in encoding (holding information about the logical qubit state) are indicated with light blue background. (c) Time evolution of ${\mathcal{R}}_Q$ at the same three training stages (red, blue, green) for individual trajectories (dashed: averaged over many trajectories). Jumps are due to measurements. (d) Evolution depending on stochastic measurement results indicated as “0” or “1” and also via the color (red or orange) of the measurement gate symbol. The policy strongly deviates between the different branches, demonstrating that RL finds adaptive quantum-feedback strategies, where the behavior depends on the measurement outcomes. Note that even the mostly converged strategy is still probabilistic to some degree.

Figure 4

Visualizing the operation of the state-aware network. (Same scenario as in Fig. 3) (a) Sequence of quantum states visited during a standard repetitive detection cycle (after encoding) displayed for an initial logical qubit state $|+x⟩=(|0⟩+|1⟩)/\sqrt{2}$ in the absence of unexpected measurement outcomes (no errors). Each state $\widehat{\rho }$ is represented by its decomposition into eigenstates, where, e.g., $\circ •••+•\circ •\circ \equiv |0111⟩+|1010⟩$. The eigenstates are sorted according to decreasing eigenvalues (probabilities). Eigenstates of less than 5% weight are displayed semitransparently. In this particular example, each eigenstate is a superposition of two basis states in the $z$ basis. (b) Gate sequence triggered upon encountering an unexpected measurement. Again, we indicate the states $\widehat{\rho }$ with bars now showing the probabilities. By further measurements, this sequence tries to disambiguate the error. (c) Visualization of neuron activations (300 neurons in the last hidden layer), in response to the quantum states (more precisely, maps $\mathrm{\Phi }$) encountered in many runs. These activations are projected down to 2D using the t-SNE technique [50], which is a nonlinear mapping that tries to preserve neighborhood relations while reducing the dimensionality. Each of the $2\times {10}^5$ points corresponds to one activation pattern and is colored according to the action taken. The sequences of (a) and (b) are indicated by arrows, with the sequence (b) clearly proceeding outside the dominant clusters (which belong to the more typically encountered states). Qubits are numbered 1,2,3,4, and gate cnot13 has control qubit 1 and target 3. (d) The enlargement shows a set of states in a single cluster which is revisited periodically during the standard detection cycle; this means we stroboscopically observe the time evolution. The shading indicates the time progressing during the gate sequence with a slow drift of the state due to decoherence. (See the Supplemental Material [51] for an extended discussion.)

Figure 5

(a) Scenarios with different qubit connectivity. cnot gates are allowed only between qubits connected by a line. In each case, we display the “standard” gate sequence discovered by the net, as well as a sequence involving corrective actions (triggered by an unexpected measurement). From top to bottom: chain with fixed measurement location, chain with arbitrary measurements, ring connected to an ancilla. The red rectangle highlights an interval during which the quantum state is not dominated by a single component, indicating that the precise location of the error still must be pinpointed. These nontrivial detection and recovery sequences are considerably more complex than the periodic detection cycle. (b) The effective enhancement of the decoherence time via error correction for the different scenarios. Here, $T_{\mathrm{dec}}=1200$ is the single-qubit decoherence time (in units of the gate time that defines the time step), and $T_{\mathrm{eff}}$ is extracted from the decay of ${\mathcal{R}}_Q$ after 200 time steps. The differences can be traced back to the lengths of the detection cycles. (c) Behavior as a function of the measurement error. The network discovers that redundancy is needed; i.e., the number of measurements in the gate sequences increases (in this plot, from one per cycle to about six).

Figure 6

Countering dephasing by measurements (adaptive phase estimation). (a) The setting: A data qubit whose phase undergoes a random walk due to a fluctuating field. Since the field is spatially correlated, its fluctuations can be detected by measuring the evolution of nearby ancilla qubits, which can then be exploited for correction. In this example, the allowed actions are measurements along $x$, $y$ ($MX$, $MY$), and the idle operation. (b) For one ancilla, the network performs measurements periodically, finding the optimal measurement interval. This can be seen by comparing the coherence time enhancement as a function of the interval (in units of the single-qubit decoherence time $T_{\text{single}}$) for the network (circles) with the analytical predictions (curves; see the Supplemental Material [51]). The remaining differences are due to the discretization of the interval (not considered in the analytics). The coupling between noise and ancilla (${\mu }_2$) is different from that between noise and data qubit (${\mu }_1$), and the strategy depends on the ratio indicated here. (c) Coherence time enhancement for different numbers of ancillas (here, ${\mu }_2={\mu }_3={\mu }_4=4{\mu }_1$). (d) Gate sequences. For two ancillas, the network discovers an adaptive strategy, where measurements on qubit 2 are rare, and the measurement basis is decided based on previous measurements of qubit 3. The arrows show the measurement result for qubit 3 in the equator plane of the Bloch sphere. (e) The two-ancilla adaptive strategy (overview; see, also, the Supplemental Material [51]). Here, a brute-force search for strategies is still (barely) possible, becoming infeasible for higher ancilla numbers due to the exponentially large search space of adaptive strategies [${\mu }_2=3.8{\mu }_1$, ${\mu }_3=4.1{\mu }_1$ in (d),(e)].

Figure 7

(a) Training the state-aware network to implement decoding back into the original data qubit near the end of a 200-step gate sequence. Blue: recoverable quantum information ${\mathcal{R}}_Q$. Orange: (rescaled, shifted) overlap ${\mathcal{O}}_Q$ between final and initial states of the data qubit. Inset displays the decoding gate sequence. (b) Training the recurrent network in a supervised way to imitate the state-aware network. Again, we show ${\mathcal{R}}_Q$ and ${\mathcal{O}}_Q$ evolving during training. Dashed blue or orange lines depict the performance of the state-aware network. (c) To investigate the workings of the recurrent network, we display some of the LSTM neuron activations in the second-to-last layer. Quantum states are illustrated even though the network is unaware of them. The network’s input consists of the observed measurement result (if any) and of the actual gate choice (made in the previous time step, here aligned on top of the current neuron activations). The example gate sequence displayed here first shows the repetitive standard error-detection pattern of repeated parity measurements that is interrupted once an unexpected measurement is encountered, indicating an error. During error recovery (boxed region), the network first tries to pinpoint the error and then applies corrections. The neurons whose behavior changes markedly during recovery are depicted in gray. Neuron $1r$ obviously keeps track of whether recovery is ongoing, while $3r$ acts like a counter keeping time during the standard periodic detection pattern. (See the Supplemental Material [51] for more information.)