Reinforcement learning for autonomous preparation of Floquet-engineered states: Inverting the quantum Kapitza oscillator

I demonstrate the potential of reinforcement learning (RL) to prepare quantum states of strongly periodically driven nonlinear single-particle models. The ability of Q-learning to control systems far away from equilibrium is exhibited by steering the quantum Kapitza oscillator to the Floquet-engineered stable inverted position in the presence of a strong periodic drive within several shaking cycles. The study reveals the potential of the intraperiod (micromotion) dynamics, often neglected in Floquet engineering, to take advantage over pure stroboscopic control at moderate drive frequencies. Without any knowledge about the underlying physical system, the algorithm is capable of learning solely from tried protocols and directly from simulated noisy quantum measurement data, and is stable to noise in the initial state and sources of random failure events in the control sequence. Model-free RL can provide new insights into automating experimental setups for out-of-equilibrium systems undergoing complex dynamics, with potential applications in quantum information, quantum optics, ultracold atoms, trapped ions, and condensed matter.

Figure 1

Prototypical example of Floquet engineering: high-frequency periodic modulation stabilizes the metastable equilibrium at the inverted position of a classical pendulum (left) and a quantum oscillator (right). (a) Schematic Floquet control setup displaying the step-periodic drive (green), and the control protocol (cyan). The purpose of this work is to prepare states localized at the inverted position (b) in the presence of strong periodic modulation without knowledge about the physical system, using reinforcement learning (see Supplemental Material, video 1 [115]).

Figure 2

Distributions of the nearest (N) [i.e., $\{-4,0\}$ and $\{0,+4\}]$ and next-nearest (NN) $\left[\right\{-4,+4\left\}\right]$ one-flip excitations of the SD protocols (see text). The dashed vertical line marks the fidelity of the absolute maximum. The data refer to the sample of all local protocol minima which satisfy $F_h\left(t_f\right)>98\%$. The model parameters are the same as in Fig. 3.

Figure 3

Running fidelity estimate during train (left) and test (right) stages of Q-learning for the quantum Kapitza oscillator. The three rows show data for stochastic rewards only (top), stochastic rewards and noisy initial condition (middle), and stochastic rewards, noisy initial condition and stochastic environment (bottom). The horizontal black lines show the average fidelity of local infidelity minima computed using one-flip SD (dashed dotted line), and the best SD realization (dashed line) out of ${10}^6$ local minima samples obtained with deterministic cost function. The horizontal red dashed line shows the best-encountered protocol using RL during the train stage. The RL data points (red) are averaged over 100 seed realizations of the pseudorandom number generator, with the uncertainty window (shaded area) computed using a bootstrapping approach. The green curve shows the exponentially attenuated exploration schedule $\varepsilon \left(n_{\mathrm{ep}}\right)$, normalized within [0,1] for display purposes: unity corresponds to no exploration. The oscillator parameters are $N_T=15$ periods with eight steps each, $\mathrm{\Omega }/{\omega }_0=10, A=2$, and $m{\omega }_0=1$.

Figure 4

RL agent performs better in nonstroboscopic (8 bangs per cycle) than stroboscopic (1 bang per cycle) control at moderate drive frequencies. See Fig. 3 for the parameters.

Figure 5

Protocols (upper row) and the corresponding fidelities (lower row) against time. The oscillator parameters are $\mathrm{\Omega }/{\omega }_0=10, A=2$, and $m{\omega }_0=1$. The protocols contain 8 steps per period for a total of $N_T=15$ periods.

Figure 6

Fidelity distributions for random protocols on linear (left), and a semilog (middle) scale, and the sample of local SD minima (right). The oscillator parameters are $\mathrm{\Omega }/{\omega }_0=10, A=2$, and $m{\omega }_0=1$. The protocols contain 8 steps per period for a total of $N_T=15$ periods.

Figure 7

Initial state and target states at the inverted position for $\mathrm{\Omega }/{\omega }_0=10, A=2$, and $m{\omega }_0=1$.

Figure 8

Train and test stages for the quasi-Gaussian target state $\langle \theta |{\psi }_{*}\rangle \propto exp(-a^{-1/4}{cos}^2\theta )$, localized at the inverted position. The details are the same as in Fig. 3, main text.

Figure 9

Numerical experiment training the agent using binary quantum data in a deterministic environment starting from a fixed initial state, while testing on noisy initial states and/or stochastic environment. The red data show the agent's estimated fidelity, while the true fidelity is shown in blue. The mismatch between the agent's estimate (red) and the true fidelity (blue) arises due to the absence of noise and stochasticity during the train stage. The left side (train curves) shows the same data for better comparison. The parameters are the same as in Fig. 3 of the main text.

Figure 10

Training behavior as a function of the number of train episodes: worst run (let), best run (middle), and average performance (right) over 100 seed realizations of the pseudorandom number generator. The oscillator parameters are $\mathrm{\Omega }/{\omega }_0=10, A=2$, and $m{\omega }_0=1$. The protocols contain 8 steps per period for a total of $N_T=15$ periods. The target is the Floquet eigenstate. The initial state is noise free and the environment is deterministic.

Figure 11

Training behavior as a function of the number of train episodes: ${10}^4$(left), ${10}^5$ (middle), and ${10}^6$(right). The oscillator parameters are $\mathrm{\Omega }/{\omega }_0=10, A=2$, and $m{\omega }_0=1$. The protocols contain 8 steps per period for a total of $N_T=15$ periods. The target is the Floquet eigenstate.

Figure 12

Training behavior as a function of the number of train episodes for the classical Kapitza pendulum. The oscillator parameters are $\mathrm{\Omega }/{\omega }_0=10, A=2$, and $m{\omega }_0=1$. The protocols contain 8 steps per period for a total of $N_T=4$ periods. The target is ${\theta }_{*}=\pi$ and $p_{*}=0$. See Supplemental Material, video 3 [115] for a visualization of the best-encountered RL protocol.