Reinforcement-learning-assisted quantum optimization

We propose a reinforcement learning (RL) scheme for feedback quantum control within the quantum approximate optimization algorithm (QAOA). We reformulate the QAOA variational minimization as a learning task, where an RL agent chooses the control parameters for the unitaries, given partial information on the system. Such an RL scheme finds a policy converging to the optimal adiabatic solution of the quantum Ising chain that can also be successfully transferred between systems with different sizes, even in the presence of disorder. This allows for immediate experimental verification of our proposal on more complicated models: the RL agent is trained on a small control system, simulated on classical hardware, and then tested on a larger physical sample.

Figure 1

Scheme of (a) a single step of Reinforcement Learning for QAOA; (b) the “episodes” loop in each $k\mathrm{th}$ training “epoch,” with the “policy” and “state-value” neural networks ${\mathrm{\Pi }}_{{\theta }_k}$ and $V_{{\theta }_k^{\prime }}^{\mathrm{\Pi }}$.

Figure 2

(a) Residual energy density ${\epsilon }_P^{\mathrm{res}}$, Eq. (3), vs $P$. The target state is ferromagnetic with $h=0$. Full symbols: results from RL only; empty symbols: a local optimization (LO) supplements the RL actions ($\mathrm{RL}+\mathrm{LO}$); data are averaged over 50 test runs. The black dashed line is the lower bound of Eq. (4). (b) The schedule $s_t={\gamma }_t/({\gamma }_t+{\beta }_t)$. Full blue lines denote $s_t$ learned after $N_{\mathrm{epo}}=1024$ epochs on a chain of $N=128$ sites. After LO (dashed red lines), they all collapse on the minimum corresponding to the iterative LO solution [14] (black empty squares). (Inset) Same data for $N_{\mathrm{epo}}=128$ training epochs, where not all the LO optimized actions sets fall onto the iterative LO solution.

Figure 3

(a) Residual energy, Eq. (3), vs $P$ for a single instance of the random TFIM: comparison between bare RL and RL followed by local optimization ($\mathrm{RL}+\mathrm{LO}$) results. (b) The optimized $s_t$ obtained with different procedures. Empty squares: the iterative LO process of Ref. [14]; Blue circles: $\mathrm{RL}+\mathrm{LO}$ performed directly on a $N=128$ chain; gray lines: ${\mathrm{RL}}_{N=8}+{\mathrm{LO}}_{N=128}$, i.e., training of a $N=8$ chain used as Ansatz for LO of the $N=128$ chain. (c) The residual energy obtained by training the policy on a single disordered instance with $N_{\mathrm{train}}$ sites and tested on 10 instances of length $N_{\mathrm{test}}=128$, before (red squares) and after (blue circles) a local optimization. Grey lines connect the same disorder instance.

Figure 4

(a) Comparison between the residual energy curves obtained with different protocols: QAOA with random initialization (blue circles), QAOA with linear initialization (green squares), RL (red triangles), and $\mathrm{RL}+\mathrm{LO}$ (black triangles). The data refer to a chain of $N=64$ spins. (b) data collapse of the residual energy curves after rescaling $P\to P/{log}_{2}N$.

Figure 5

(a) Learned actions after 1024 training epochs on a chain of $N=64$ (blue lines), and their local optimization (red dashed lines). (b) Optimal parameter sets from QAOA with local optimization (blue lines), their average (thick green line) and the linear Ansatz (black line). In both panels, $P=10$ and the target transverse field is $h=\frac{\sqrt{5}-1}{2}$. In both panels, we indicate the typical residual energy found for those parameters sets.

Figure 6

Residual energy obtained by training the policy on a system of $N_{\mathrm{train}}$ spins and then applying the actions on a test system of $N_{\mathrm{test}}=128$ spins. Red squares refer to the direct transfer of the RL actions, while blue circles have a local optimization on top. The inset shows the same data in log scale, where it is possible to appreciate the difference in the average performance depending on the value of $N_{\mathrm{train}}$.

Figure 7

Difference between the residual energy during the training and QAOA optimal value $\frac{1}{2(P+2)}$, for a uniform Ising chain of $N=32$ spins and episode length $P=10$. We compare two choices of the reward function given by Eqs. (A1) and (A2), blue and red lines, respectively. The black solid line is an heuristic upper bound for the convergence speed of the residual energy obtained from RL protocol to the optimal value.

Figure 8

Average residual energy at the end of the training versus the number of episodes per training epoch. The data are obtained for a uniform Ising chain of $N=32$ spins and episode length $P=10$.