Demonstrating Scalable Randomized Benchmarking of Universal Gate Sets

Randomized benchmarking (RB) protocols are the most widely used methods for assessing the performance of quantum gates. However, the existing RB methods either do not scale to many qubits or cannot benchmark a universal gate set. Here, we introduce and demonstrate a technique for scalable RB of many universal and continuously parametrized gate sets, using a class of circuits called randomized mirror circuits. Our technique can be applied to a gate set containing an entangling Clifford gate and the set of arbitrary single-qubit gates, as well as gate sets containing controlled rotations about the Pauli axes. We use our technique to benchmark universal gate sets on four qubits of the Advanced Quantum Testbed, including a gate set containing a controlled-S gate and its inverse, and we investigate how the observed error rate is impacted by the inclusion of non-Clifford gates. Finally, we demonstrate that our technique scales to many qubits with experiments on a 27-qubit IBM Q processor. We use our technique to quantify the impact of crosstalk on this 27-qubit device, and we find that it contributes approximately $2/3$ of the total error per gate in random many-qubit circuit layers.

Popular Summary

Quantum computers are scaling up rapidly, and with that comes a need to measure the performance of large-scale quantum devices. Randomized benchmarking protocols, the most widely used methods today, are limited in scalability, as they require hard classical computations unless limited to quantum gates that are easy to simulate classically. To fully assess a quantum computer, researchers need to study universal gate sets, which can perform general quantum computations. Our benchmarking technique, mirror randomized benchmarking (MRB) of universal gate sets, overcomes this classical computation roadblock and can scale to thousands of qubits on a variety of universal gate sets.

We use MRB of universal gate sets to study multiple gate sets on two processors, providing experimental evidence that it is a reliable method for measuring average error. We perform our experiments on up to 27 qubits and use our results to understand the magnitude of various types of errors. Our results show that MRB on many qubits reveals and quantifies crosstalk, a major error source in current processors that is not fully captured by running one- and two-qubit circuits. We find that such errors dominate in the many-qubit regime on a 27-qubit IBM Q processor, highlighting the importance of scalable benchmarks.

MRB with universal gate sets will allow experimentalists to assess the performance of more qubits and a wider variety of gates on their quantum processors than existing techniques. Additionally, our method can be adapted to create other benchmarks, providing further interesting information about quantum processors.

Figure 1

Scalable randomized benchmarking of universal gate sets. (a) Randomized mirror circuits combine a simple reflection structure with randomized compiling to enable scalable and robust RB of universal gate sets. (b) Data and fits to an exponential obtained by using our method—MRB of universal gate sets—to benchmark a universal gate set on $n=1$, 2, 3, 4 qubits of the Advanced Quantum Testbed and the average error rates of $n$-qubit layers ($r_{\mathrm{\Omega }}$, where $\mathrm{\Omega }$ is the layer sampling distribution) extracted from these decays. (c) We benchmark each connected set of $n$ qubits for $n=1$, 2, 3, 4, enabling us to map out the average layer error rate ($r_{\mathrm{\Omega }}$) for each subset of qubits.

Figure 2

Randomized mirror circuits over universal gate sets. To construct a randomized mirror circuit of benchmark depth $d$ (and total depth $2d+2$), we first sample a random depth $d+1$ circuit. This circuit alternates between layers of randomly sampled one-qubit gates and layers of randomly sampled two-qubit gates. It can be thought of as consisting of a single initial layer of random one-qubit gates followed by $d/2$ composite layers (see the inset). We then append to this circuit its inverse, i.e., the circuit in reverse with each layer replaced with its inverse. This creates a depth $2d+2$ circuit that, if run perfectly, always returns the all-zeros bit string. This circuit is susceptible to systematic addition or cancellation of errors between the two halves of the circuit. To prevent this unwanted effect, we then apply randomized compiling to the circuit. We insert a layer of random single-qubit Pauli gates (cyan) after each one-qubit gate layer. In order to guarantee that this randomly compiled circuit still always, if run perfectly, returns a single bit string $s$, our procedure (i) changes the rotation angles in the two-qubit gates (orange) if these gates are not Clifford gates, (ii) adds in single-qubit Pauli axis rotations following the two-qubit gates (red), and (iii) adds in correction Pauli gates (purple) prior to each single-qubit gate layer. The yellow boxes show gates that are compiled together to create the final circuit of depth $2d+2$. This circuit contains $d$ composite layers, which we call its benchmark depth.

Figure 3

Investigating the reliability of MRB using simulations. We simulate MRB on $n$ all-to-all-connected qubits for $n=1$, 2, 4 with the gate set $({\mathbb{G}}_1,{\mathbb{G}}_2)=(\mathbb{S}\mathbb{U}(2),\{\mathsf{c}\mathsf{s},{\mathsf{c}\mathsf{s}}^{†}\})$ with randomly sampled gate-dependent errors. From left to right, the columns show results from simulations with crosstalk-free error models consisting of (a),(d) only stochastic errors, (b),(e) a combination of stochastic and Hamiltonian errors, and (c),(f) only Hamiltonian errors (see Sec. 4d for details). (a)–(c) The MRB error rate per qubit [$r_{\mathrm{\Omega },\text{perQ}}=1-(1-r_{\mathrm{\Omega }}{)}^{1/n}$] versus the average composite layer error rate per qubit [${\epsilon }_{\mathrm{\Omega },\text{perQ}}=1-(1-{\epsilon }_{\mathrm{\Omega }}{)}^{1/n}$] for each randomly sampled error model. The MRB error rate $r_{\mathrm{\Omega }}$ closely approximates ${\epsilon }_{\mathrm{\Omega }}$, and the agreement is closest under purely stochastic errors. (d)–(f) The relative error ${\delta }_{\mathrm{rel}}=(r_{\mathrm{\Omega },\text{perQ}}-{\epsilon }_{\mathrm{\Omega },\text{perQ}})/{\epsilon }_{\mathrm{\Omega },\text{perQ}}$ divided by its uncertainty ${\sigma }_{{\delta }_{\mathrm{rel}}}$ for each randomly sampled error model (${\sigma }_{{\delta }_{\mathrm{rel}}}$ is calculated via a standard nonparametric bootstrap). The MRB error rate $r_{\mathrm{\Omega }}$ is biased toward very slightly underestimating ${\epsilon }_{\mathrm{\Omega }}$ for $n>2$ qubits, which is expected from our theory (see the main text).

Figure 4

The Advanced Quantum Testbed. We perform MRB experiments on four qubits (Q4–Q7) of AQT’s eight-qubit superconducting transmon processor (AQT@LBNL Trailblazer8-v5.c2). The processor includes eight fixed frequency transmons coupled in a ring geometry. Each qubit (purple) has its own control line (orange) and readout resonator (cyan) coupled to a shared readout bus (red) for multiplexed readout.

Figure 5

Randomized benchmarking of universal gate sets on four qubits of the Advanced Quantum Testbed. We use MRB to benchmark $n$-qubit layers constructed from three different gate sets, on each connected $n$-qubit subset of a linearly connected set of four qubits $\{Q4,Q5,Q6,Q7\}$ in an eight-qubit superconducting transmon processor (AQT@LBNL Trailblazer8-v5.c2). The rows correspond to results from three different choices of gate set, each consisting of a two-qubit gate set ${\mathbb{G}}_2$ and a single-qubit gate set ${\mathbb{G}}_1$. From top to bottom, the rows correspond to a universal gate set containing two non-Clifford entangling gates and the set of all single-qubit gates [${\mathbb{G}}_2=\{\mathsf{c}\mathsf{s},{\mathsf{c}\mathsf{s}}^{†}\}$, ${\mathbb{G}}_1=\mathbb{S}\mathbb{U}(2)$]; a universal gate set containing a Clifford entangling gate and the set of all single-qubit gates [${\mathbb{G}}_2=\{\mathsf{c}\mathsf{z}\}$, ${\mathbb{G}}_1=\mathbb{S}\mathbb{U}(2)$]; and a nonuniversal, Clifford gate set [${\mathbb{G}}_2=\{\mathsf{c}\mathsf{z}\}$, ${\mathbb{G}}_1={\mathbb{C}}_1$, where ${\mathbb{C}}_1$ is the one-qubit Clifford group]. (a)–(c) MRB decays for the qubit subsets $\{Q4\}$, $\{Q4,Q5\}$, $\{Q4,Q5,Q6\}$, and $\{Q4,Q5,Q6,Q7\}$. Violin plots and points show the distribution and mean, respectively, of the MRB circuit’s observed polarization ($S_d$) versus benchmark depth ($d$). The curve is a fit of the mean of $S_d$ (${\overline{S}}_d$) to ${\overline{S}}_d=A{p}^d$. The average error rate of an $n$-qubit layer ($r_{\mathrm{\Omega }}$) is given by $r_{\mathrm{\Omega }}=(4^n-1)(1-p)/4^n$. The observed ${\overline{S}}_d$ decays exponentially, as predicted by our theory for MRB. (d)–(f) The estimated error rate $r_{\mathrm{\Omega }}$ for each qubit subset that we benchmark. (g)–(i) Predictions for the average layer error rate of three- and four-qubit subsets (hatched) based on the experimental one- and two-qubit error rates (unhatched) and the assumption of no crosstalk errors. The difference between (d)–(f) and (g)–(i) quantifies the contribution of crosstalk errors to the average error rate of an $n$-qubit layer, for $n=3$, 4. For all three gate sets and $n=4$, we see that crosstalk errors are contributing approximately 0.7% error to $r_{\mathrm{\Omega }}$, which is approximately $1/3$ of $r_{\mathrm{\Omega }}$.

Figure 6

Estimating crosstalk errors on AQT. We estimate the contribution of crosstalk errors to the layer error rate $r_{\mathrm{\Omega }}$ for $n=3$, 4 qubits by taking the difference between each experimental error rate ($r_{\mathrm{\Omega }}$) and a corresponding prediction ($r_{\mathrm{\Omega },\mathrm{pred}}$) obtained from the experimental one- and two-qubit error rates and the assumption of no crosstalk. We find that crosstalk contributes approximately 0.2%–0.4% to $r_{\mathrm{\Omega }}$ for $n=3$ (which is $1/8–1/4$ of $r_{\mathrm{\Omega }}$) and approximately 0.7% to $r_{\mathrm{\Omega }}$ for $n=4$ (which is $1/3$ of $r_{\mathrm{\Omega }}$).

Figure 7

Estimating the infidelity of dressed four-qubit layers. Estimates of the error rates of individual ${\mathbb{G}}_1$-dressed layers containing a single two-qubit gate ($\mathsf{c}\mathsf{s}$, ${\mathsf{c}\mathsf{s}}^{†}$, or $\mathsf{c}\mathsf{z}$), obtained by fitting an $n$-qubit depolarizing model to the four-qubit MRB data. This scalable analysis technique enables extraction of additional information about each layer’s error from MRB data. To validate our results against an established technique, we compare to infidelities independently estimated using cycle benchmarking [9]. We observe qualitative agreement. The cycle benchmarking experiments measure the infidelities of layers dressed with one-qubit gates sampled from a different gate set (the Pauli group) to those used in our MRB experiments, so exact agreement is not expected.

Figure 8

Validating MRB by comparison to DRB. (a) The structure of DRB circuits, which is a method for benchmarking an $n$-qubit layer set, when applied to a universal gate set. DRB is known to be reliable, but it is exponentially expensive in $n$ for universal gate sets—because, for a universal gate set, its circuits start by implementing a Haar random unitary from $\mathbb{S}\mathbb{U}(2^n)$. (b) The error rates obtained when running equivalent DRB and MRB experiments, on every one- and two-qubit subset of the four qubits we benchmark on AQT. The close agreement between the DRB and MRB error rates is experimental evidence that MRB is reliable. The inset shows the polarization at benchmark depth $d=0$ ($S_0$) for $n$-qubit DRB with $n=1$, 2, 3. The rapid decay in $S_0$ is due to the overhead in implementing a Haar-random unitary, and it makes DRB of universal gate sets infeasible on more than around 2–3 qubits.

Figure 9

Randomized benchmarking of a universal gate set on a 27-qubit IBM Q processor. We run MRB on $n$-qubit subsets of the ibmq_montreal processor, for 15 exponentially spaced $n$ from $n=1$ to $n=27$. (a) The MRB decays and fits to an exponential for the six subsets of qubits illustrated in (b). The observed polarization decays exponentially in all cases. Because of the minimal overhead in MRB circuits, we obtain an exponential decay even for 27 qubits and can extract a low-uncertainty estimate of the average error rate of 27-qubit layers [$r_{\mathrm{\Omega }}=28(1)\%$]. (c) The observed error rate per qubit $r_{\mathrm{\Omega },\text{perQ}}=1-(1-r_{\mathrm{\Omega }}{)}^{1/n}\approx r_{\mathrm{\Omega }}/n$ (red circles) versus $n$ increases rapidly with $n$, even though the circuits have a constant expected two-qubit gate density $\xi =1/4$. This increase in $r_{\mathrm{\Omega },\text{perQ}}$ is due to two-qubit gate crosstalk, not spatial variations in gate error rates. This is confirmed by comparison to predictions for $r_{\mathrm{\Omega },\text{perQ}}$ (blue diamonds) obtained from one- and two-qubit error rates, for each one-qubit and connected two-qubit subset, and the assumption of no crosstalk. (d) The ratio of the observed ($r_{\mathrm{\Omega },\text{perQ}}$) to predicted ($r_{\mathrm{\Omega },\text{perQ},\mathrm{pred}}$) per-qubit error rate shows that crosstalk errors cause the per-qubit error rate $r_{\mathrm{\Omega },\text{perQ}}$ to increase by approximately 250%–300% when $n\ge 15$. (e) The one- and two-qubit error rates are obtained using simultaneous one-qubit MRB on all 27 qubits (blue boxes) and two-qubit MRB, on all pairs of connected qubits, run simultaneously on the qubit pairs in eight distinct groupings (the purple and green boxes show two such groups).

Figure 10

Investigating the reliability of MRB in the presence of measurement error. We simulate MRB with error models with no measurement error (a),(f) and with four types of measurement error: (e),(j) amplitude damping error on one qubit, (d),(i) amplitude damping error distributed across all qubits, (c),(h) bit flip error on one qubit, and (b),(g) bit flip error distributed across all qubits. For all simulations, we use the gate set $\mathbb{G}=(\{\mathsf{c}\mathsf{s},{\mathsf{c}\mathsf{s}}^{†}\},\mathbb{S}\mathbb{U}(2))$ and randomly generated Hamiltonian and stochastic gate errors. (a)–(e) We compare the MRB error rate per qubit $r_{\mathrm{\Omega },\text{perQ}}$ to the actual per-qubit error rate ${\epsilon }_{\mathrm{\Omega },\text{perQ}}$, estimated via sampling. We observe close agreement between the MRB error rate and ${\epsilon }_{\mathrm{\Omega }}$ with each type of measurement error. (f)–(j) The distribution of the relative error ${\delta }_{\mathrm{rel}}=(r_{\mathrm{\Omega },\text{perQ}}-{\epsilon }_{\mathrm{\Omega },\text{perQ}})/{\epsilon }_{\mathrm{\Omega },\text{perQ}}$ divided by its uncertainty ${\sigma }_{{\delta }_{\mathrm{rel}}}$, for models with each type of measurement error. The mean ${\delta }_{\mathrm{rel}}$ (red dashed line) is shown for each distribution.

Figure 11

MRB with varied-strength measurement error. We simulate MRB with the gate set $\mathbb{G}=(\{\mathsf{c}\mathsf{s},{\mathsf{c}\mathsf{s}}^{†}\},\mathbb{S}\mathbb{U}(2))$ with randomly generated gate errors and four types of measurement error: (a) bit flip error distributed across all qubits, (b) bit flip error on one qubit, (c) amplitude damping error distributed across all qubits, and (d) amplitude damping error on one qubit. We plot the relative error in the MRB error rate ${\delta }_{\mathrm{rel}}=(r_{\mathrm{\Omega },\text{perQ}}-{\epsilon }_{\mathrm{\Omega },\text{perQ}}){/\epsilon }_{\mathrm{\Omega },\text{perQ}}$ divided by its uncertainty ${\sigma }_{{\delta }_{\mathrm{rel}}}$, versus the strength of the measurement error. We observe no systematic change in the error in the MRB error rate with as the strength of measurement error changes.

Figure 12

Comparing the success rates of circuits that differ only by virtual phase gates. Our experiments are designed so that each randomly sampled circuit containing $\mathsf{c}\mathsf{z}$ and single-qubit Clifford gates [the $(\{\mathsf{c}\mathsf{z}\},{\mathbb{C}}_1)$ gate set] differs from a randomly sampled circuit containing $\mathsf{c}\mathsf{z}$ and Haar-random single-qubit unitaries [the $(\{\mathsf{c}\mathsf{z}\},\mathbb{S}\mathbb{U}(2))$ gate set] only by the angles in its ${\mathsf{z}}_{\theta }$ gates. Here, we compare the observed polarization of each pair of circuits that differ only by the angles in these virtual ${\mathsf{z}}_{\theta }$ gates. For many of these circuit pairs $(C_1,C_2)$, $C_1$ and $C_2$ have very different observed polarizations, meaning that they have very different success rates. (a) The observed polarization for the $(\{\mathsf{c}\mathsf{z}\},\mathbb{S}\mathbb{U}(2))$ circuits and their corresponding Clifford circuits. (b) The difference in observed polarization between $(\{\mathsf{c}\mathsf{z}\},\mathbb{S}\mathbb{U}(2))$ circuits and their corresponding Clifford circuits.

Figure 13

Fitting error models to MRB data and estimating gate error rates. We fit two types of error models to MRB data to estimate the infidelity of individual circuit layers. (a) By running two MRB experiments with two different two-qubit gate densities $\xi$, we can estimate the mean infidelity of a set of one or more two-qubit gates—here, $\mathsf{c}\mathsf{s}$ and ${\mathsf{c}\mathsf{s}}^{†}$—using basic linear algebra (see Appendix pp5-s3a). We call this procedure the two-densities heuristic. The estimates of the average gate error obtained from the two-densities heuristic (orange) are compared to independent estimates obtained from two more rigorous but more complex and computationally intensive procedures: fitting each set of two-qubit MRB data to (i) a depolarizing model (light blue) and (ii) a stochastic Pauli errors model (dark blue). (b) To fit a depolarizing model, we assign an error rate to each dressed layer and an error rate to each qubit’s readout. (c) To fit a Pauli stochastic model, we assign a Pauli stochastic channel to each possible gate except the virtual ${\mathsf{z}}_{\theta }$ gates and each qubit’s readout.

Figure 14

Estimating the infidelity of dressed four-qubit layers. By fitting error models to MRB data, we can estimate the infidelity of each ${\mathbb{G}}_1$-dressed layer used in the MRB circuits. Here, we show four different estimates of the infidelities of four-qubit layers containing a single $\mathsf{c}\mathsf{s}$, ${\mathsf{c}\mathsf{s}}^{†}$, or $\mathsf{c}\mathsf{z}$ gate on one of the three connected pairs of qubits. We fit a simple $n$-qubit depolarizing model to (i) the four-qubit data and (ii) the one- and two-qubit data and use both models to estimate the infidelity of four-qubit ${\mathbb{G}}_1$-dressed layers. The estimates from fitting to the one- and two-qubit data do not account for any additional crosstalk errors that occur in four-qubit layers, so the additional error estimated when fitting to the four-qubit data is a quantification of crosstalk. We also fit a more sophisticated stochastic Pauli error model to the four-qubit circuit data, resulting in comparable estimates to those obtained from the simple depolarizing model (which uses a scalable, less computationally intensive analysis). To validate our results against an established technique, we compare to infidelities independently estimated using cycle benchmarking [9]. We observe qualitative agreement. The cycle benchmarking experiments measure the infidelities of layers dressed with one-qubit gates sampled from a different gate set (the Pauli group) to that used in our MRB experiments [$\mathbb{S}\mathbb{U}(2)$ or ${\mathbb{C}}_1$, the single-qubit Clifford group], and these experiments are implemented on a different day than the MRB circuits, so exact agreement is not expected.

Figure 15

Comparing error models for two-qubit MRB data. For each pair of qubits we benchmark, we fit two error models, a depolarizing error model and a Pauli stochastic error model, to the data from the two-qubit RB experiments. Here, we compare the simulated observed polarization based on the fit models to the observed observed polarization for each circuit.

Figure 16

Comparing error models for four-qubit MRB data. We fit two error models, a depolarizing error model and a Pauli stochastic error model, to the data from our four-qubit MRB experiments. Here, we compare the simulated observed polarization based on the fit models to the observed polarization for each circuit. The mean-squared error in the observed polarization is approximately 40% smaller for the Pauli stochastic model than the depolarizing model.