Erratum : Glassy Chimeras could be blind to quantum speedup : Designing better benchmarks for quantum annealing machines

Copyright © and Moral Rights are retained by the author(s) and/ or other copyright owners. A copy can be downloaded for personal non-commercial research or study, without prior permission or charge. This item cannot be reproduced or quoted extensively from without first obtaining permission in writing from the copyright holder(s). The content must not be changed in any way or sold commercially in any format or medium without the formal permission of the copyright holders.

Recently, a programmable quantum annealing machine has been built that minimizes the cost function of hard optimization problems by adiabatically quenching quantum fluctuations. Tests performed by different research teams have shown that, indeed, the machine seems to exploit quantum effects. However experiments on a class of random-bond instances have not yet demonstrated an advantage over classical optimization algorithms on traditional computer hardware. Here we present evidence as to why this might be the case. These engineered quantum annealing machines effectively operate coupled to a decohering thermal bath. Therefore, we study the finite-temperature critical behavior of the standard benchmark problem used to assess the computational capabilities of these complex machines. We simulate both random-bond Ising models and spin glasses with bimodal and Gaussian disorder on the D-Wave Chimera topology. Our results show that while the worstcase complexity of finding a ground state of an Ising spin glass on the Chimera graph is not polynomial, the finite-temperature phase space is likely rather simple: Spin glasses on Chimera have only a zero-temperature transition. This means that benchmarking classical and quantum optimization methods using spin glasses on the Chimera graph might not be the best benchmark problems to test quantum speedup. We propose alternative benchmarks by embedding potentially harder problems on the Chimera topology. Finally, we also study the (reentrant) disorder-temperature phase diagram of the random-bond Ising model on the Chimera graph and show that a finite-temperature ferromagnetic phase is stable up to 19.85(15)% antiferromagnetic bonds. Beyond this threshold the system only displays a zero-temperature spin-glass phase. Our results therefore show that a careful design of the hardware architecture and benchmark problems is key when building quantum annealing machines. Quantum devices are gaining an increasing importance in everyday technology: They find applications in different technological areas such as (true) quantum random number generators, as well as quantum encryption systems for data transmission. The holy grail is to build a programmable quantum simulator with capabilities exceeding "traditional" computer hardware based on classical bits. The first programmable commercial device to exploit the unique power of quantum mechanics to perform computations is the D-Wave One quantum annealer [1]. In analogy to simulated annealing [2] where thermal fluctuations are adiabatically quenched to minimize a cost function, this machine is based on the quantum annealing optimization method [3][4][5][6][7][8][9][10][11] where quantum fluctuations replace thermal ones.
Tests by different research teams have shown that, indeed, the D-Wave quantum annealer optimizes using quantum effects [12][13][14][15][16]. Although it has been shown theoretically [17], as well as with numerical experiments [8] that quantum annealing should, in principle, outperform classical (thermal) optimization algorithms (such as simulated annealing [2]) on an algorithmic level, when applied to a class of random edgeweight instances, the quantum annealing machine has not yet shown a speedup over classical optimization methods [13]. In this work we present evidence why this might be the case: The D-Wave One and Two quantum annealing machines use a restrictive "Chimera" topology (see Fig. 1 for an example with 128 quantum bits) imposed due to fabrication constraints of the solid-state quantum bits.
The high connectivity between the spins within each block effectively renders the model quasi-two-dimensional. Note that the graph is not planar.
Probably the best benchmark problem to test the efficiency of optimization algorithms is a spin glass [18]. Both the disorder and frustration present produce a complex energy landscape that challenges optimization algorithms. As such, all current benchmarks of the quantum annealing machine attempt to find the ground state of a certain class of Ising spin glass on the Chimera topology. However, as shown in this work, instances belonging to this class of Ising spin glasses on the Chimera topology only have a spin-glass phase at zero temperature. Furthermore, the energy landscape of such problem seems to be simpler down to low temperatures than for a system with a finite-temperature transition because correlations only build up very close to absolute zero. Because quantum annealing excels in tunneling through barriers-barriers which do not seem to be present or very pronounced at finite, but low temperatures in this case-classical annealing schedules typically have an advantage for this particular class of system.
Although the worst-case complexity of finding a ground state of an Ising spin glass on the Chimera topology is worse than polynomial [19], it seems that the fact that the system only orders at zero temperature allows for an easy determination of typical ground-state instances using heuristic classical approaches [13,20]. As such, Ising spin glasses on the Chimera topology live up to their name: an amalgamation of both ordinary and complex behavior.
We reach these conclusion by studying the critical behavior of Ising spin glasses with both Gaussian and bimodal random bonds on the Chimera topology, as well as the randombond Ising model. Based on our findings, we propose stronger benchmark problems by embedding on the Chimera topology problems that should have a finite-temperature transition and so might be harder to optimize. Furthermore, our results show that a careful design of the hardware architecture and benchmark problems is key when building quantum annealing machines.
We should also mention that while the results of this paper provide a plausible explanation for the scaling behavior so far observed on random-bond Ising problems, it is also known that on quantum annealer implementations, the couplers and biases are influenced by various sources of noise and error, as demonstrated by the fact that gauge-transformed specifications of the same problem can give substantially different performance [13]. Classical simulated annealing does not, of course, have this issue, and it is currently unclear how much loss of efficiency these errors cause for the hardware.
The paper is structured as follows. In Sec. I we introduce the standard benchmark model. Results within the spin-glass sector are presented in Sec. II, followed by results within the ferromagnetic sector in Sec. III. In Sec. IV we discuss better benchmarks, followed by concluding remarks.

I. MODEL, OBSERVABLES AND ALGORITHM
We study the spin-glass Hamiltonian with S i ∈ {±1} Ising spins on the nodes of the Chimera graph. An example of the topology with 4×4 blocks of 8 spins is shown in Fig. 1. A chimera graph with k×k blocks has N = 8k 2 spins and a characteristic linear length scale of L = √ N .
The interactions J ij are either chosen from a Gaussian disorder distribution with zero mean and unit variance, or from a bimodal distribution P( where with a probability p a bond is ferromagnetic. Ordering in spin glasses is detected from the spin over- where "α" and "β" are two independent spin replicas with the same disorder. In the ferromagnetic case order is measured via the magnetization, i.e., m = (1/N ) i S α i . To detect the existence of a phase transition to high precision, we measure the Binder ratio [21] where O represents either the magnetization m for the ferromagnetic sector, or the spin overlap q in the spin-glass sector. The Binder ratio is a dimensionless function, which means that at a putative transition data for different characteristic system sizes L will cross when T = T c , T c the critical temperature (up to corrections to scaling). This means Using a finite-size scaling analysis, the critical temperature T O c and the critical exponent ν O can be determined. To uniquely determine the universality class of a system, two critical exponents are needed [22]. To this end, we also measure the susceptibility χ O = N O 2 , where O again represents either the magnetization m for the ferromagnetic sector, or the spinglass order parameter q for the spin-glass sector. The susceptibility scales as Simulations are done using the replica exchange Monte Carlo [23] method and simulation parameters are listed in Table I. Note that for each disorder instance we simulate two independent replicas to compute the spin-glass overlap. In the Gaussian case we test equilibration using the method developed in Ref. [24] adapted to the Chimera topology. For bimodal disorder we perform a logarithmic binning of the data. Once the last four bins agree within error bars we deem the system to be in thermal equilibrium. To obtain optimal values for the critical parameters, we determine these via the analysis method pioneered in Ref. [25] where the critical parameters are optimized using a Levenberg-Marquard minimization until the chi-square of a fit to a third-order polynomial is minimal. This approach is then bootstrapped to obtain statisticallysound error bars. Figure 2 shows a finite-size scaling analysis of the Binder ratio (top panel) for both Gaussian disorder (full symbols) and bimodal disorder with p = 0.50 (open symbols) in a semilogarithmic scale. The data scale extremely well, even far from the spin-glass transition temperature. We find that for both cases the same critical parameters, namely

II. RESULTS WITHIN THE SPIN-GLASS SECTOR
Note that spin glasses on two-dimensional square lattices have ν ≈ 3.45 [26], i.e., spin glasses on the Chimera topology are close to two space dimensions. Interestingly, the phase transition to a spin-glass phase only occurs at zero temperature, despite the Chimera graph be- ing nonplanar. Furthermore, the divergence of the correlation length is rather violent with ξ ∼ T −4 for T → 0. This suggests that the phase space and correlations are trivial for a spin glass defined on the Chimera topology down to very low temperatures. However, only when the system is very close to T = 0, strong correlations build up. One of the potential advantages of a quantum algorithm over a classical one lies in its ability to tunnel through barriers. Classical algorithms must "climb" over these barriers [27]. The aforementioned results imply that for arbitrarily large systems, the barriers of a spin glass defined on a Chimera graph at nonzero temper-  ature seem to be of "finite" height, while for any Ising spin glass with a finite transition temperature the barriers diverge below T q c for decreasing temperature T and increasing system size N . This could offer one explanation for why quantum annealing machines, such as D-Wave One and Two, cannot find a noticeable speedup over classical algorithms such as vanilla simulated annealing [2] on this class of problems. Furthermore, a spin glass on a Chimera graph seems to order in an almost "discontinuous" fashion at zero temperature with the highly-connected blocks of 8 spins behaving like a "super spin" on a two-dimensional-like planar lattice. Once the individual blocks order, the whole system suddenly orders. It is well known that quantum annealing has problems when first-order transitions are present [28][29][30][31], i.e., this could be a second reason why quantum annealing machines do not seem to outperform simple classical optimization methods on these problems.

III. RESULTS WITHIN THE FERROMAGNETIC SECTOR
For completeness, we also study the Ising ferromagnet on the Chimera graph and compute the disorder p (fraction of ferromagnetic bonds) vs temperature T phase diagram of the model. For no disorder, i.e., the pure ferromagnet where J ij = 1 ∀i, j in Eq. (1), an Ising model on the Chimera graph displays a two-dimensional-Ising-model-like behavior, likely due to the super-spin structure mentioned earlier. Figure 3 top panel, shows a finite-size scaling analysis of the ferromagnetic Binder ratio g m as a function of the scaling variable . The data scale extremely well for T m c = 4.1618(3) and ν m = 1. Note that the obtained value for the critical exponent ν m agrees with the value for the twodimensional Ising ferromagnet [22], therefore corroborating our assumption that the system might behave similar to a twodimensional super-spin Ising model. Figure 3, bottom panel, shows a finite-size scaling analysis of the ferromagnetic susceptibility χ m /( √ N ) 2−ηm as a function of ( √ N ) 1/νm (T − T m c ) using the estimate of the critical temperature determined from the finite-size scaling of the Binder ratio. The data scale very well with very small corrections to scaling using ν m = 1 and η m = 2/5. Note that the value of the critical exponent η m is slightly larger, yet close to the exact value of the two-dimensional Ising model (η = 1/4). Therefore, an Ising ferromagnet on the Chimera graph and the two-dimensional Ising model do not share the same universality class [32]. Summarizing: Finally, we study the random-bond version of the Ising model on the Chimera graph where a fraction p of ferromagnetic bonds is antiferromagnetic. We vary p and compute the critical temperature of the ferromagnetic phase. Figure  4 shows the (critical) temperature T vs disorder p phase diagram. The dotted (blue) line represents the Nishimori condition [33] exp(−2β) = p/(1 − p). The point where the phase boundary (solid line) crosses the Nishimori line, i.e., for p > p c = 0.1985 (15), ferromagnetic order is lost. This means that for any finite temperature and p ≤ p c a randombond Ising model on the Chimera graph is easily solved using any type of optimization algorithm. Therefore, to compare a quantum adiabatic optimizer to any classical optimization method strong enough disorder is needed. Figure 5 shows cartoons of the energy landscape for a system with a zero-temperature transition [panel (a), top] and a finite-temperature transition [panel (b), bottom] to a spinglass state. When the temperature is above the putative critical temperature T c , the energy landscape is typically simple with one dominant minimum. For a system that has a finitetemperature transition and for temperatures T < T c , the energy landscape becomes rough with barriers that grow with decreasing temperature T and increasing system size N until the ground state of the system is reached. However, for a  system where T c = 0-such as spin glasses on the Chimera topology-the energy landscape is likely much simpler with one dominant minimum until the ground state is reached. This means that for such a system a classical annealing schedule should perform well in comparison to quantum annealing which excels when the energy landscape has barriers. As such, it is no surprise that quantum annealing machines like D-Wave One or Two show no better scaling than vanilla classical simulated annealing. Note also that when the system with bimodal disorder is in the ferromagnetic phase (p < p c ), the energy landscape is likely also rather simple and reminiscent of the cartoon shown in the left-most panels of Fig. 5. This means that benchmarking quantum annealing machines that operate at low, but finite temperatures using spin glasses on a Chimera topology is likely not the best approach. Indeed, recent studies in a field [35,36] have shown that spin-glass instances can be efficiently computed classically. However, because there is likely no spin-glass state in a field [37][38][39], this is no surprise. To truly discern if quantum annealing machines (defined on the Chimera topology) display an advantage over  (15) ferromagnetic order is lost and the system is paramagnetic (PM). For p pc and T = 0 there is a zero-temperature spin-glass state [34]. , bottom] to a spin-glass state. For high enough temperatures, i.e., above the critical temperature Tc, the energy landscape is simple with one clear minimum that dominates and some "bumps along the way." For a system that has a finitetemperature transition and for temperatures T < Tc, the energy landscape becomes rough with clear barriers that render any classical annealing schedule inefficient, because the system can easily be trapped in a metastable state for decreasing temperature. These barriers grow with increasing system size N and decreasing temperature T until they form a rough energy landscape for T = 0 (ground state of the system). For a system where Tc = 0, such as spin glasses on the Chimera topology, the energy landscape is typically simple up until the ground state is reached. This means that for such a system a classical annealing schedule like simulated annealing should perform well in comparison to quantum annealing which excels when the energy landscape exhibits barriers.

IV. DISCUSSION
classical annealing algorithms, problems that displays a finitetemperature transition and therefore have a rough energy landscape for a range of finite temperatures need to be embedded in the Chimera topology. Given the current hardware constraints, we propose the two following benchmarks: Three-dimensional cubic lattices: The system has a finitetemperature transition to a spin-glass state at T c ≈ 0.96J for Gaussian disorder [T c ≈ 1.1J for bimodal disorder and p = 0.5] [25]. We estimate that a Chimera graph of 2048 qubits could be used to embed a relatively modest 3D system of 5 3 = 125 spins with periodic boundary conditions and one of size 8 3 = 512 with free boundary conditions. Note that current state-of-the art classical optimization algorithms [40,41] can estimate ground states to high accuracy of up to approximately 14 3 = 2744 spins.
Viana-Bray model: The Ising spin glass is defined on a random graph with average connectivity k [24,42]. For any k > 2, the system has a finite-temperature phase transition into a spin-glass state. To simplify the embedding, a random graph with Gaussian disorder and k = 3 could be studied where T c ≈ 0.748J. However, to be able to embed the long-range connections between the spins, we estimate that N 2 qubits might be needed to embed a system with N spins like in the mean-field Sherrington-Kirkpatrick model [43][44][45].
Rescaled Chimera systems: It is plausible that if the ratio of interactions within the cells and between the cells changes proportional to the system size, a mean-field-like finite-temperature spin-glass transition might emerge. For example, the random inter-cell interactions could be rescaled with is a nondecreasing function of the system size] while leaving the random intra-cell interactions untouched. We attempted to weaken the effects of the tightly-bound inter-cell spin clusters in the Chimera graph by setting the spin-spin interactions to 1/4 of all intra-cell interactions (on average). Although universality considerations would suggest that T c should still be zero when f (N ) = 1/4 ∀N , our data for systems up to approximately 3200 spins suggest T c ≈ 0.6(2). We do emphasize, however, that corrections to scaling are huge and a study with far larger systems might be needed.
One could, in principle, also embed a two-dimensional Ising spin glass on a square lattice, where it was first shown that quantum annealing displays an advantage over classical annealing by Santoro et al. via simulations at very low temperatures [8]. However, ground states of two-dimensional Ising spin glasses can be computed in polynomial time and the low-temperature behavior of this system is known to be unusual and still controversial [46]. As such, this might not be a robust and well-controlled benchmark, especially because the D-Wave machines operate at temperatures considerably higher than in the aforementioned study by Santoro et al. [8].
The aforementioned examples also illustrate a limitation of the Chimera topology: To embed many systems, a large overhead of quantum bits in the machine to simulated physical bits is needed. This is particularly the case because no longrange connections between the spins are present. Finally, at this point it is unclear if the critical behavior of an embedded system is the same as the critical behavior of the actual classical system. This is of utmost importance if one wants to use programmable quantum annealing machines as quantum simulators.

V. CONCLUSIONS
Although Barahona [19] has shown rigorously that spin glasses defined on graphs like the Chimera topology are worst-case NP-hard, the Chimera spin glass of the type so far used to compare quantum to classical annealers represents a hard, but typically easy optimization problem. Because such a spin glass on the Chimera topology only orders at zero temperature, classical thermal annealers will typically be able to efficiently estimate ground states for the system. The performance of these classical algorithms would considerably deteriorate if the problem to be optimized would exhibit a finitetemperature transition below which energy barriers diverge with decreasing temperature and increasing system size. To be able to show that quantum annealing machines based on the Chimera topology outperform classical annealing schedules, nontrivial embeddings in higher space dimensions or with long-range interactions, as outlined above, would be needed. Furthermore, at this point it is unclear how the overhead of the embedding scales with the size of the system and if the embedded system via edge contraction shares the same universality class as the true problem to be emulated-especially when simulated on the actual D-Wave hardware [47]. The latter open questions are subject of current research and we conclude by emphasizing that the design of the hardware topology in quantum annealing machines is of crucial importance.