Neural-Network Heuristics for Adaptive Bayesian Quantum Estimation

Quantum metrology promises unprecedented measurement precision but suffers in practice from the limited availability of resources such as the number of probes, their coherence time, or nonclassical quantum states. The adaptive Bayesian approach to parameter estimation allows an efficient use of resources because of adaptive experiment design. For its practical success, fast numerical solutions for the Bayesian update and the adaptive experiment design are crucial. Here we show that neural networks can be trained to become fast and strong experiment-design heuristics using a combination of an evolutionary strategy and reinforcement learning. Neural-network heuristics are shown to outperform established heuristics for the technologically important example of frequency estimation of a qubit that suffers from dephasing. Our method of creating neural-network heuristics is very general and complements the well-studied sequential Monte Carlo method for Bayesian updates to form a complete framework for adaptive Bayesian quantum estimation.

Popular Summary

Imagine we want to measure a physical quantity, for example a magnetic field. In order to measure as precisely as possible, we don't detect the magnetic field “directly” but we use a quantum sensor as a “middleman”, consisting for example of an atom, which is sensitive to the magnetic field. Most information about the magnetic field can be gained from a series of measurements if we use adaptive measurements which depend on previous measurement results. But we have to keep in mind that some measurements are easier or faster to realize than others which has to be factored in. This brings us to the question considered in our work: What is the best strategy for choosing adaptive measurements? “A trained neural network” is our answer. Using reinforcement learning we are able to factor in all details of our quantum sensor such that the neural network outperforms previously known strategies.

The role of the neural network is that of a “brain in the sensor” which takes fast decisions on how to adjust the measurements. Using computer simulations of the quantum sensor, we create a virtual training environment in which the neural network gains experience and improves its behavior. Reinforcement learning rewards good decisions of the neural network and punishes bad ones. Finally, combining a trained neural network with a real quantum sensor gives us a “smart” quantum sensor.

Our method is very general and can be used to make all kinds of adaptive quantum sensors smarter. We envisage that the next generation of quantum sensors will be adaptive and smart.

Figure 1

(a) The setup for adaptive Bayesian quantum estimation. First, prior information [e.g., about the prior $p(\mathit{\theta })$ and the available resources] is passed to the heuristic, which returns an experiment design $e_1$. From the quantum experiment, we obtain a measurement outcome $d_1$, which is used to numerically compute the Bayesian update of $p(\mathit{\theta })$. Then, (updated) information is again passed to the heuristic and this cycle is repeated in the same manner until some exit condition is fulfilled. (b) The training procedure for the NNs. First, the NN learns to imitate the behavior of a known heuristic (which may have been found by the cross-entropy method). Then we use this NN as a starting point for RL. (c) The RL setup for generating training data. An agent (depicted as a NN) interacts with a RL environment. The RL environment defines the estimation problem. It consists of a simulated quantum experiment [depicted with initial state $\rho$ and parameter-dependent quantum channel $\mathrm{\Phi }(\mathit{\theta })$] and a numerical Bayesian update. It also stores information about available resources for experiment design. (d) The evolutionary principle of a simple cross-entropy method. NNs are treated as vectors of parameters sampled from a Gaussian distribution $\mathcal{N}(\mathit{\mu },\mathrm{\Sigma })$ with constant covariance. At the beginning, the NNs are initialized randomly.

Figure 2

Comparison of the Bayes risk for different experiment-design heuristics. The right column shows Bayesian estimation with a limited number of experiments and the left column shows Bayesian estimation with a limit on the available time (limited to the maximal value plotted on the $x$ axis). We study frequency estimation without $T_2$ relaxation (top row) and with $T_2$ relaxation (second and third rows, with different values of $T_2$ as stated in the plot titles) as well as the simultaneous estimation of the frequency $\omega$ and relaxation rate $T_2^{-1}$ (bottom row). The Bayes risk is calculated numerically from ${10}^4$ episodes; see Appendix pp2 for details. TRPO is pretrained with the heuristic specified in parentheses. The lines are linear interpolants to guide the eye.

Figure 3

Ninety-five percent credible regions for multiparameter estimation of $(\omega ,T_2^{-1})$. The credible regions correspond to the uncertainty in the posterior distributions obtained by running one Bayesian estimation (episode) for each heuristic while keeping the true parameter fixed. All heuristics use a uniform prior for $\omega \in (0,1)$, $T_2^{-1}\in (0.09,0.11)$.

Figure 4

Illustration of Bayes risk calculation for time-limited environments. (a) Three time series, each of which corresponds to one episode (i.e., one Bayesian estimation): the “zeroth” data point at time 0 corresponds to the uncertainty $c$ in the prior [see Eq. (B9)], and the $k\mathrm{th}$ data point in each time series corresponds to the $k\mathrm{th}$ experiment of that episode; its $x$ coordinate is the accumulated time used for the first $k$ experiments, and its $y$ coordinate is the traced covariance $\mathrm{tr}\left[{\mathrm{cov}}_{\mathit{\theta }|D_k}\left(\mathit{\theta }\right)\right]$. The time limit is $T$, which is overstepped only with the last experiment of each episode. Linear interpolation of each time series (dashed lines) and evaluation of the linear interpolants at equally spaced times spanning $[0,T]$ yields (b), depicted with 20 equally spaced times. Then, we can take a timewise mean of the data (i.e., for each of the 20 equally spaces times, we calculate the mean over all time series). This gives us the mean of the interpolated traced covariance [depicted as red stars in (c)], which serves as an approximate time-resolved Bayes risk.

Figure 5

Analytic approximations of a flat prior that vanishes at the endpoints of its domain. The plotted functions correspond to Eq. (C5) with $a=0$ and $b=1$ for different values of $k$.

Figure 6

Distribution of experiment designs ($y$ axis). On the $x$ axis we show the experiment number (e.g., $x=10$ corresponds to the tenth experiment in each episode). For readability, we cut off $y$ data at $t=500$. In the time-limited cases, CEM sometimes reaches the numerical threshold for the number of experiments by choosing many experiments with very small $t$. This manifests itself in the corresponding plots as a bright line close to $t=0$ (see, e.g., the case of $\omega$ estimation, $T_2=100$, time limited). If the gray background of the plots is visible, there are no experiment designs for the corresponding values.

Figure 7

As for Fig. 6, the only difference being that this figure shows data for different estimation problems as specified in the titles of the subfigures.