Robust optimization for quantum reinforcement learning control using partial observations

The current quantum reinforcement learning control models often assume that the quantum states are known a priori for control optimization. However, full observation of the quantum state is experimentally infeasible due to the exponential scaling of the number of required quantum measurements on the number of qubits. In this paper, we investigate a robust reinforcement learning method using partial observations to overcome this difficulty. This control scheme is compatible with near-term quantum devices, where the noise is prevalent and predetermining the dynamics of the quantum state is practically impossible. We show that this simplified control scheme can achieve similar or even better performance when compared to the conventional methods relying on full observation. We demonstrate the effectiveness of this scheme on examples of quantum state control and the quantum approximate optimization algorithm (QAOA). It has been shown that high-fidelity state control can be achieved even if the noise amplitude is at the same level as the control amplitude. Besides, an acceptable level of optimization accuracy can be achieved for a QAOA with a noisy control Hamiltonian. This robust control optimization model can be trained to compensate for the uncertainties in practical quantum computing.

Figure 1

The idea of the QAOA. The parametrized unitary controls are implemented on near-term quantum devices to transfer between the states. The fidelity $|\langle {\psi }_f|U(\vec{\gamma },\vec{\beta }){|{\psi }_i\rangle |}^2$ between the initial and target states is obtained by repeating the experiments and measuring the final states. The parameters $(\vec{\gamma },\vec{\beta })$ are optimized using classical algorithms.

Figure 2

Representation of the quantum state on the Bloch sphere. The quantum state moves to the next position under the action of unitary control. Part of the coordinates of the state are observed to calculate the reduced distance between the current state and the target state.

Figure 3

The diagrammatic representation of the quantum RL control algorithm with replay buffer. The agent generates an action which is either ${\gamma }_j$ or ${\beta }_j$, depending on the alternating control Hamiltonian chosen for the current step. This process is repeated until $t=2p$. Experience tuples $({\vec{o}}_t,a_t,r_t,{\vec{o}}_{t+1},f_t)$ are stored into the replay buffer. A subset of tuples is sampled from the buffer for updating the agent.

Figure 4

The framework of TD3. At the start of the training, the actor and two critics of the predict network are randomly initialized. Then their parameters are copied to the target network. In the target network, the target $\mathrm{Q}$ value $\widehat{Q}$ is chosen as the smaller one of the estimated optimal $\mathrm{Q}$ values of two target critics to avoid overestimation. The history $h_t^l$ contains $l$ pairs of past observations and actions as $\{{\vec{o}}_i,a_i\}, i=t-l,...,t-1$. $\mu ({\vec{o}}_t,h_t^l)$ is an actor which generates the action $a_t$ based on ${\vec{o}}_t$ and the history $h_t^l$. $Q({\vec{o}}_t,a_t,h_t^l)$ denotes the estimated $\mathrm{Q}$ value.

Figure 5

The evolution of average fidelity during the training. Curves with different colors correspond to different levels of perturbation. Panel (a) shows the average fidelities achieved by the current $n$ trajectories. Panel (b) shows the average fidelities within each epoch, where each epoch contains 200 trajectories of training. As a result, 20 000 trajectories of training in panel (a) can be divided into a total of 100 epochs in panel (b).

Figure 6

Numerical results on the generalizability of our method. (a) The trained models are tested with different noise levels. ${\delta }_{\mathrm{train}}$ is the noise level for training the model, and ${\delta }_{\mathrm{test}}$ is used for testing the model with the same initial and target states. For each ${\delta }_{\mathrm{test}}$, 10 000 trajectories are generated with random and independent perturbations. Then the average fidelity achieved by these trajectories is calculated. (b) The model trained with $\delta =0.1$ is tested on random initial and target states. One hundred thousand pairs of randomly generated initial and target states are tested. The frequency distribution of the achieved fidelities is depicted in this figure.

Figure 7

The average fidelities of the two-qubit state control during the training. Panel (a) shows the training curves of the average fidelities for different levels of perturbations. Panel (b) shows the average fidelity of every 200 trajectories at different noise levels. The total number of epoch is $M=100$.

Figure 8

The test results of the trained models with different noise levels.

Figure 9

The averaged fidelities during training for the two-qubit case with different numbers of observations.

Figure 10

The evolution of average approximation ratio $\overline{R_p}$ during training. (a) The approximation ratio averaged over the current $n$ trajectories. (b) The average approximation ratio of each epoch. Each epoch contains 400 trajectories.