Two-Tone Optomechanical Instability and Its Fundamental Implications for Backaction-Evading Measurements

While quantum mechanics imposes a fundamental limit on the precision of interferometric measurements of mechanical motion due to measurement backaction, the nonlinear nature of the coupling also leads to parametric instabilities that place practical limits on the sensitivity by limiting the power in the interferometer. Such instabilities have been extensively studied in the context of gravitational wave detectors, and their presence has recently been reported in Advanced LIGO. Here, we observe experimentally and describe theoretically a new type of optomechanical instability that arises in two-tone backaction-evading (BAE) measurements, designed to overcome the standard quantum limit, and demonstrate the effect in the optical domain with a photonic crystal nanobeam, and in the microwave domain with a micromechanical oscillator coupled to a microwave resonator. In contrast to the well-known oscillatory parametric instability that occurs in single-tone, blue-detuned pumping, which is characterized by a vanishing effective mechanical damping, the parametric instability in balanced two-tone optomechanics is exponential, and is a result of small detuning errors in the two pump frequencies. Its origin can be understood in a rotating frame as the vanishing of the effective mechanical frequency due to an optical spring effect. Counterintuitively, the instability occurs even in the presence of perfectly balanced intracavity fields, and can occur for both signs of detuning. We find excellent quantitative agreement with our theoretical predictions. Since the constraints on tuning accuracy become stricter with increasing probe power, it imposes a fundamental limitation on BAE measurements, as well as other two-tone schemes. In addition to introducing a new limitation in two-tone BAE measurements, the results also introduce a new type of nonlinear dynamics in cavity optomechanics.

While quantum mechanics imposes a fundamental limit on the precision of interferometric measurements of mechanical motion due to measurement backaction, the nonlinear nature of the coupling also leads to parametric instabilities that place practical limits on the sensitivity by limiting the power in the interferometer. Such instabilities have been extensively studied in the context of gravitational wave detectors, and their presence has recently been reported in Advanced LIGO. Here, we observe experimentally and describe theoretically a new type of optomechanical instability that arises in two-tone backaction-evading (BAE) measurements, a protocol designed to overcome the standard quantum limit. We demonstrate the effect in the optical domain with a photonic crystal nanobeam cavity and in the microwave domain with a micromechanical oscillator coupled to a microwave resonator. In contrast to the well-known parametric oscillatory instability that occurs in single-tone, blue-detuned pumping, and results from a two-mode squeezing interaction between the optical and mechanical modes,the parametric instability in balanced two-tone optomechanics results from single-mode squeezing of the mechanical mode in the presence of small detuning errors in the two pump frequencies. Counterintuitively, the instability occurs even in the presence of perfectly balanced intracavity fields and can occur for both signs of detuning errors. We find excellent quantitative agreement with our theoretical predictions. Since the constraints on tuning accuracy become stricter with increasing probe power, the instability imposes a fundamental limitation on BAE measurements as well as other two-tone schemes, such as dissipative squeezing of optical and microwave fields or of mechanical motion. In addition to identifying a new limitation in two-tone BAE measurements, the results also introduce a new type of nonlinear dynamics in cavity optomechanics.

I. INTRODUCTION
Interferometric position measurement of mechanical oscillators is the underlying principle of the Laser Interferometer Gravitational Observatory (LIGO) [1] and constitutes one of the most sensitive techniques for determining absolute distance available to date. In a similar vein, cavity optomechanical systems [2], which exploit radiation-pressure coupling of light and mechanical motion in micromechanical and nanomechanical systems, have achieved some of the most sensitive measurements of mechanical motion relative to the zero-point motion [3,4]. They can operate in a regime where measurement quantum backaction is relevant [5,6] and where cooling [7][8][9] and amplification [10][11][12] via radiationpressure backaction is accessible. In both settings, the quantum fluctuations of radiation pressure place a fundamental limitation on the displacement sensitivity [13,14]. Still, there can be other constraints. Radiation-pressure nonlinearities can equally well pose a limit to sensitivity. Indeed, the parametric oscillatory instability [10][11][12][15][16][17][18][19] is one of the most fundamental optomechanical ef-of measurement of mechanical motion [36]. These BAE measurements proceed by pumping an optomechanical system (that resides in the resolved sideband limit [7]) simultaneously on the upper and lower motional sidebands of the cavity, and allow in principle arbitrary measurement sensitivity; by increasing the probing power, measurement imprecision is decreased, without incurring additional measurement noise due to quantum backaction. Although theoretically proposed several decades ago, only recently have advances in cavity optomechanics made it possible to operate under conditions dominated by quantum backaction and carry out BAE measurements with both microwave [37][38][39][40] and optical [41] systems. The two-tone instability reported here arises from deviations from the ideal BAE configuration where there is a finite-frequency detuning error with respect to both optical and mechanical resonance frequencies. In contrast to the well-known parametric oscillatory instability that occurs in single-tone pumping on the upper motional sideband (or blue detuned in the bad-cavity limit), which results from a two-mode squeezing interaction (nondegenerate parametric down-conversion) between the optical and mechanical modes, and is associated with antidamping, the instability in balanced twotone optomechanics results from single-mode squeezing (degenerate parametric down-conversion) of the mechanical mode, and is associated with the optical spring effect. While the parametric oscillatory instability is dynamically classified as an unstable spiral, the two-tone instability is classified as a saddle point [42]. We gain further insight by showing that the balanced two-tone scheme in the good-cavity (i.e. resolved-sideband) limit can be mapped to single-tone optomechanics in the bad-cavity limit.
The threshold for the onset of the instability depends on the magnitude of the tuning errors, and is also inversely proportional to the optical pump power. For any given experimental inaccuracy in the pump frequency, a finite instability threshold exists in two-tone experiments, which ultimately limits the maximum probe power and thus the achievable sensitivity. As we show, these limitations can be prohibitive for strong pumping powers aiming to surpass the SQL. We emphasize that the twotone instability is intrinsic to the optomechanical interaction and does not arise from extraneous effects. It depends exclusively on tuning errors and power. Thus it stands in contrast to previously reported instabilities in BAE measurements associated with parametric driving [43,44], where the underlying cause has been attributed to the dependence of the mechanical frequency on temperature [45] or the presence of two-level systems [46]. In this sense, the two-tone instability poses fundamental constraints, and one may need to resort to active feedback techniques, as in the case of the parametric oscillatory instability [31,47].
While our focus is on BAE measurements, it is important to note that the phenomenon reported here can affect other two-tone optomechanical protocols, such as dissipative optomechanical squeezing of optical and microwave fields [48,49] or of mechanical motion [38,39,[50][51][52]. For example, in recent work on noiseless singlequadrature amplification of mechanical motion [53], the squeezing effect we report here produces significant deviations from the expected system behavior.

II. OBSERVATION OF INSTABILITY IN TWO-TONE PUMPING
In a BAE measurement the cavity, with resonance frequency ω c , is probed with two pump tones of equal power, each tuned to the upper and lower motional sideband of the cavity, i.e., at ω c ± Ω m . It can also be understood as a single pump tuned at ω c with full amplitude modulation at the mechanical frequency Ω m . In Fig. 1(a) we illustrate the scheme, and introduce a small detuning error ∆ c κ of the symmetrically spaced tones with respect to the cavity resonance, as well as an error ∆ m ∼ Γ m in modulation frequency. As a result, the two tones are detuned by ∆ c ± (Ω m + ∆ m ) from the cavity resonance. The motion of the oscillator, due to thermal noise from the environment and quantum backaction by the pump fields, is imprinted as fluctuations on the fields reflected from the optomechanical system. The corresponding output noise spectrum of the two probes exhibits two Lorentzians separated by 2∆ m , also shown in Fig. 1(a). When ∆ c = ∆ m = 0, an ideal BAE measurement is performed. The mechanical sidebands are superimposed on each other, but while thermal motion adds in quadrature, the quantum backaction noise is canceled from the measurement record [37,41,54]. A hallmark of BAE measurements is witnessing, as ∆ m is varied from a finite value to zero, a total mechanical noise that is lower than the sum total of the two individual mechanical noise spectra. The total evaded backaction, expressed in units of mechanical quanta, is equal to the optomechanical cooperativity C, proportional to the probing power.
Such BAE measurements have been carried out in the microwave domain in several experiments using mechanically compliant capacitors [37][38][39][40], and have recently been extended to the optical domain [41]. We carried out BAE experiments using an optomechanical photonic crystal nanobeam cavity operating at GHz frequencies mounted in a 3 He buffer-gas cryostat, as detailed in prior work [41,55]. The parameters of the devices are shown in Appendix C, and details of the measurement setup, which uses a quantum-limited heterodyne detection method, are contained in Ref. 41. Figure 2(a) shows our BAE measurement in the optical domain. In this experiment the cooperativity was set to C 2.6 and ∆ m scanned from a positive value toward zero. However, instead of the expected decrease in total noise, indicating BAE, an exponential increase in the total noise is observed near ∆ m /2π = 40 kHz. Upon a further decrease in ∆ m , the system leaves the linear regime as the optical cavity undergoes self-oscillations, i.e., we observe an instability. Pumping scheme leading to two-tone instability. (a) Frequency-space representation of optical backactionevading (BAE) measurement using two-tone pumping. An optomechanical system is pumped with two pumps that are placed on the lower and upper motional sideband of the cavity (shown in gray). Two detuning errors are introduced, due to imperfect knowledge of the mechanical oscillator frequency (∆m), and due to imperfect symmetric spacing around the cavities' resonance frequency ωc, as expressed by ∆c. Also shown are the mechanical resonance at frequency Ωm and the scattered mechanical sidebands. The inset shows an optomechanical system: a mechanical oscillator (position coordinatex) that is the moving mirror of a Fabry-Perot cavity and that is coupled to the cavity mode by radiation pressure. (b) An equivalent system, in which the two-tone pumping is mapped to a Hamiltonian that exhibits the same dynamics as (a), consisting of a single continuous pump field applied at ωc + ∆c and with the mechanical oscillator frequency obeying the substitution Ωm → −∆m. Note that ∆c and ∆m have been exaggerated for clarity.
To independently confirm the existence and universality of this phenomenon, we performed the measurement in an entirely different optomechanical system: an electromechanical system based on a mechanicallycompliant vacuum-gap capacitor coupled to a superconducting microwave resonator placed in a dilution refrigerator [56,57]. Figure 2(b) shows this experiment, with a cooperativity of C = 7 (here, measurement backaction includes classical noise, which should also be canceled). In this second measurement as well, we observe an exponential increase in the total noise as the detuning ∆ m is decreased. As ∆ m is decreased further, the noise saturates the HEMT amplifier in the detection chain, leading to an increased noise floor [ Fig. 2(b), yellow curve]. In both experiments, BAE is not observed under the given experimental conditions.
The origin of this instability is not a spurious effect in the experiments, as is evident from the observation that the same behavior occurs in two very different optomechanical systems measured with very different equipment. Instead, as shown below, the instability is a direct consequence of the optomechanical interaction in the presence of the small tuning errors ∆ m and ∆ c and depends only on these two parameters and the cooperativity. We next develop the theory behind this new instability in Sec. III, and perform a systematic experimental study in Sec. IV that fully confirms the theoretical predictions.

III. THEORY
It is well known that the anti-damping induced by pumping the cavity on the upper motional sideband (or blue detuning in the bad-cavity limit) can induce a parametric oscillatory instability. In principle, there exists another type of dynamical instability in this system, one associated with the optical spring effect, i.e., a change in the restoring force induced by light. This cannot occur in the resolved-sideband regime, in the relevant case of weak coupling between the mechanical mode and the cavity field. Indeed, optomechanical systems typically employ high-quality-factor oscillators, where the shift in mechanical frequency due to dynamical backaction can be neglected. However, as we show below, this instability may arise when pumping with two tones close to the upper and lower mechanical sidebands [ Fig. 1(a)], as in BAE measurements, for example. In fact, as we show, the situation when pumping with two tones and that for single-tone driving on the upper motional sideband are described by the same linearized equations. We model the system by the standard optomechanical Hamiltonian comprising one cavity mode with frequency ω c , one mechanical mode with frequency Ω m , and a nonlinear interactionĤ int = − g 0â †â (b † +b), whereâ (b) denote the optical (mechanical) annihilation operator. The cavity is driven by one or two coherent tones that produce a coherent intracavity field with amplitudeā(t). We move to the interaction picture with respect to the HamiltonianĤ 0 = ω lâ †â and linearize the operators, a →ā(t) + δâ andb →b + δb, whereā(t) is the coherent oscillation of the cavity field andb the static displacement of the mechanical oscillator. This yields the linearized Hamiltonian (1) where g(t) = g 0ā (t) denotes the field-enhanced coupling. We consider two distinct situations: single tone driving on the upper motional sideband and balanced two-tone driving on the upper and lower motional sidebands. In single-tone driving, we have ∆ c = Ω m and g(t) = g = const, whereas the two-tone driving is described through ∆ c ≈ 0 and g(t) = g cos[(Ω m + ∆ m )t]. Applying the rotating-wave approximation (RWA) in two-tone driving but not in single-tone driving, both situations can be described by the same Hamiltonian Here, ωLO is the optical frequency of the heterodyne local oscillator laser. In the sequence, the mechanical "tuning error" ∆m is varied from a positive value toward zero, where ∆m = 0 ideally corresponds to a BAE measurement. Because of the cavity tuning error ∆c, at ∆m/2π = 0.04 MHz a strong increase in the total mechanical noise is observed, as well as narrowing of the sideband, instead of the expected decrease due to backaction cancellation. This is the onset of the two-tone instability. (b) BAE measurement in the microwave domain with a mechanically compliant capacitor coupled to a superconducting microwave resonator (inset). The procedure is the same as in (a). Measurements with both positive and negative values of ∆m are shown. The measurement at ∆m/2π = 0.02 kHz occurs within the domain of instability (yellow curve), in which case the observed spectrum is distorted by saturation of the HEMT amplifier used in the detection chain. Note the logarithmic y axis.
We describe the optomechanical system in terms of quantum Langevin equations [2,58], which take into account decay into the cavity and mechanical bath. Eliminating the optical modes in frequency space, we arrive at an effective description for the mechanical mode only where the self-energy (effective coupling) is given through In Eq. (3) we have subsumed all noise contribution into a single generic noise input operatorξ in (ω), which does not play a role in the instability mechanism. So far we have not made any approximations (beyond the RWA in two-tone driving), and indeed Eq. (3) contains all the effects we wish to consider here. The self-energy Σ plays two roles. First, it couples δb † to δb, thus acting like the coupling in a degenerate parametric oscillator. Second, as a self-energy, its real part renormalizes the frequency of the mechanical resonator and its imaginary part modifies the effective damping.
Parametric oscillatory instability.-In single-tone driving on the upper motional sideband, a parametric oscillatory instability occurs if the optical antidamping overcomes the intrinsic damping. In this regime, the mechanical frequency is very large, so the right-hand side of Eq. (3) has hardly any effect and can be neglected in a RWA. The instability occurs because the mechanical damping is modified by the imaginary part of the susceptibility, Eq. (4), −2 Im[Σ(Ω m )] −Γ m C, where the cooperativity is C = 4g 2 /κΓ m . This yields the effective mechanical damping Γ eff Γ m (1 − C), thus recovering the standard instability threshold C = 1.
Two-tone instability.-In backaction-evading measurements, however, ∆ m is small, as it represents a tuning error. This makes the right-hand side of Eq. (3) near resonant, such that it cannot be neglected. The instability arises in a way similar to a degenerate parametric oscillator [59]. Since κ is now a large parameter, we can neglect the frequency dependence of Σ(ω) ≈ Σ(0) ≡ Σ ∈ R. Note that now the self-energy Σ coincides with the optical spring effect due to a single drive detuned by ∆ c from cavity resonance. This allows us to recast Eq. (3) again as an equation of motion This equation is the same quantum Langevin equation as one would write down for a damped degenerate parametric oscillator. It can intuitively be viewed as arising from an optical spring effect modulated at 2(Ω m + ∆ m ). The dynamical matrix corresponding to Eq. (5) has eigenvalues − Γm 2 ± i ∆ m (∆ m − 2Σ). For 2Σ < ∆ m , the eigenvalues have a negative real part (damping Γ m /2) and a finite imaginary part (effective frequency ∆ eff = Re ∆ m (∆ m − 2Σ)). As Σ increases, first the effective frequency ∆ eff vanishes, at which point the damping of the modes starts to be modified. The vanishing of the effective mechanical frequency corresponds to the mechanical oscillation phase locking to the modulated optical field. The threshold for instability occurs when The instability threshold can also be written in terms of normalized detunings∆ c ≡ ∆ c /(κ/2) and∆ m ≡ ∆ m /(Γ m /2), yielding 4C = (1 +∆ 2 c )(1 +∆ 2 m )/(∆ c∆m ). This equation can only be fulfilled for C ≥ 1, and we plot its contours in Fig. 3. Note that Eq. (6) also predicts that the instability can occur for both negative and positive values of the two detuning errors (provided they have the same sign), in contrast to the parametric oscillatory instability. In Appendix A we derive instability regimes of both the parametric oscillatory and two-tone instabilities using the full Hamiltonian, Eq. (2).
Apart from using locking techniques to reduce ∆ c , it may be possible to hold off the onset of the twotone instability by using active feedback. Feedback techniques to counter the parametric oscillatory instability have been considered [31,47]. In the case of the twotone instability, the feedback force would naturally be applied on the measured quadrature (and as such is not of the viscous damping type).

IV. EXPERIMENT
To validate Eq. (6), the threshold for the two-tone instability in terms of ∆ m and ∆ c , we perform a twodimensional scan in the clean and well-controlled setting of circuit electromechanics using the system comprising a mechanically compliant vacuum-gap capacitor coupled to a superconducting resonant circuit placed in a dilution refrigerator [56,57]. Here, pump frequency fluctuations and cavity frequency fluctuations are significantly smaller than in the optical domain, and the detuning can be accurately controlled. Full details on the system and experiment are given in Appendix B. We present measurements in the optical domain in Appendix C.  Fig. 4(b). The point ∆ m =∆ c = 0 corresponds to a "perfect" BAE measurement, as can be seen in Fig. 4(g), where a 3 dB decrease in the total mechanical noise relative to∆ m = 0 due to cancellation of measurement backaction is evident. In Appendix B we show experimentally that, at the onset of instability, the effective mechanical frequency (due to the optical spring effect) equals the pump modulation frequency (i.e., the effective mechanical frequency vanishes in the rotating frame). In other words, while the instability occurs for ∆ m = 0, in the experiment the sidebands coincide at the onset of instability.
The predicted decrease in size of the stable region with increasing pumping power is clearly evident in the experimental data depicted in Fig. 4, with∆ c 1/2C required to avoid instability [e.g.,∆ c 0.04 for C = 14 in Fig. 4(c)]. Overall, excellent agreement is obtained between theory and experiment, confirming our theoretical analysis and description of the effect. Thus, optomechanics imposes strict tuning accuracy for a given measurement sensitivity in two-tone BAE measurements. In this case, measurement backaction is due to both quantum and classical noise in the two microwave tones. It is important to emphasize that for BAE measurements that FIG. 4. Investigation of the two-tone instability in a circuit-electromechanical system. (a)-(c) Mapping of the total mechanical noise as a function of∆m and∆c, for cooperativities C = {3.5, 7, 14}, respectively. The total power P in both mechanical sidebands in the output spectra is shown, normalized to the power P0 at (∆m,∆c) = (−18, 0) (no tuning error and far from the BAE regime). Since data points do not align on a regular grid (accounting for small changes in∆c along the horizontal scan), the rasterization was implemented using nearest-neighbor (Voronoi) partitioning. Solid black lines are instability thresholds from Eq.

V. CONCLUSION
We report experimentally and explain theoretically a new type of dynamical instability that was previously unreported in cavity optomechanics. This instability is qualitatively different than the parametric oscillatory instability [10,12,[17][18][19][20]60], and originates from degenerate parametric amplification of the mechanical mode. In the past, parametric oscillatory instability has limited certain single-tone experiments. Our work now demonstrates that the performance of emerging optomechanical experiments, such as backaction-evading measurements aimed at surpassing the standard quantum limit [37,41], generation of quantum squeezing [48,50,61,62], and noiseless single-quadrature amplification [53], will be intrinsically constrained by another instability determined by tuning accuracy and coupling strength. Even below the instability threshold, these new dynamics need to be taken into account.
Data Availability. The code and data used to produce the plots within this paper are available at https://doi. org/10.5281/zenodo.3419929. All other data used in this study are available from the corresponding authors upon reason5able request. Further understanding of the two-tone instability, and its distinction from the parametric oscillatory instability, can be achieved by examining the eigenvalues of the dynamical matrix, similar to the analysis performed in Ref. [63]. The Hamiltonian Eq. (2) leads to the quantum In this domain, the imaginary part of the eigenvalue is nonzero. In reality, the mechanical frequency is positive Ωm > 0, such that the lower right-hand part is unphysical. The red domain that occurs for |∆m| κ (enlargement shown in the inset) is the two-tone instability. In this domain ∆m∆c > 0, and the imaginary part of the eigenvalue is zero. The parameters used are κ = 1, Γm = 10 −2 , and C = 2.

Langevin equations
Ignoring the input noise operators δâ in and δb in , which are irrelevant in the present analysis, the Langevin equations (A1) can be written as a matrix equationẋ = Mx, with We recall that this dynamical equation describes both single-tone pumping, with the equivalence −∆ m → Ω m , and two-tone pumping in the well-resolved sideband regime (Fig. 1). An eigenvalue of the matrix Eq. (A2) with a positive real part leads to an exponentially increasing solution, and thus signals an instability. Figure 5 shows the domains of instability in the parameter space spanned by ∆ m and ∆ c . These domains separate into two classes, corresponding to the parametric and two-tone instabilities. In one class, which corresponds to the conventional parametric oscillatory instability, the imaginary part of the offending eigenvalue is nonzero, corresponding to spiral dynamics [42]. The onset of the parametric oscillatory instability coincides with the transition from a stable to an unstable spiral. This class lies in the vicinity of the diagonal ∆ c ≈ Ω m , as expected (note that the regime ∆ m = −Ω m > 0, although mathematically possible, is unphysical in this case). The second class lies close to the origin, in particular |∆ m | κ, and corresponds to the two-tone instability. In this regime the eigenvalues are well approximated by the eigenvalues of Eq. (5) due to the slow dynamics of the optical field, and the instablity domains are given by the simple condition (6). The eigenvalues are real and of opposite sign, corresponding to a saddle point, as expected from a degenerate parametric amplifier. In this picture, as the power (∝ Σ) is increased, the dynamics change from a stable spiral to a stable node and finally to a saddle point [42]. Figure 3 shows the domains of two-tone instablity for different powers. It is noteworthy to mention that Eq. (5) describes a harmonic oscillator with fixed damping Γ m and power-dependent natural frequency, i.e.,ẍ + Γ mẋ + [Γ 2 m /4 + ∆ m (∆ m − 2Σ)]x = F ext . This system becomes unstable due to vanishing of the frequency (or restoring force).

Appendix B: Microwave experiment details
For the microwave-domain part of this work, we used a system similar to the one described in Ref. 57. Specifically, an overcoupled (κ ex /2π = 2.65 MHz, κ 0 /2π = 0.16 MHz) Al superconducting microwave resonator with a resonance frequency of ω c /2π = 6.43 GHz that was coupled (vacuum coupling rate g 0 /2π = 194 Hz) to a mechanically compliant vacuum-gap capacitor (Ω m /2π = 6.15 MHz and Γ m /2π ≈ 20 Hz). The chip was cooled to about 15 mK in a dilution refrigerator.
In the experiment, we used three microwave sources with a common frequency reference to pump the optomechanical system: two BAE pumps with frequencies ω c + ∆ c ± (Ω m + ∆ m ) and an additional cooling pump tuned to ω c − Ω m − δ cool , where δ cool /2π = 400 kHz. The purpose of the cooling pump was to reduce the thermal occupation of the mechanical oscillator, thereby reducing the fluctuations in power of the mechanical sidebands that originate from fluctuations of the base temperature and vibrations of the dilution refrigerator, since the oscillator is dominantly coupled to the microwave bath. The cooling tone also increased the mechanical damping rate to Γ eff /2π = 110 Hz, allowing us to observe the narrowing of the mechanical sidebands when approaching the instability with better resolution.
To take the data in Fig. 4, we first measured the cavity resonance frequency ω c by applying the cooling tone alone and acquiring the microwave response [64]. We The plot shows the difference between the measured effective mechanical frequency and pump modulation frequency as a function of the detunings. Near the instability domain (shaded in gray), the two frequencies become equal, which corresponds to vanishing of the effective mechanical frequency in the rotating frame, confirming the theoretical treatment. Measurement is the same as in Fig. 4(b).
then placed the BAE pumps symmetrically around the resonance, ∆ c = 0, and measured the mechanical resonance frequency Ω m using the distance between the peaks of the mechanical sidebands. The data of Fig. 4 were taken along horizontal scans, by first setting the required ∆ c and varying ∆ m , acquiring noise spectra as in Fig. 2(b). Prior to taking each horizontal line, we followed a procedure to ensure that no additional mechanical loss or amplification was introduced by the BAE pumps that might lead to parametric oscillatory instability: First, only the red BAE pump was applied and its power optimized such that the width of the mechanical sideband corresponds to the desired single-tone cooperativity C (computed from the additional damping introduced). Second, the blue BAE pump was added, and its power adjusted such that the width of the mechanical sideband narrows back to Γ eff . This procedure was also followed prior to measurement of Ω m . To account for cavity frequency fluctuations, the microwave frequency ω c was also measured after each single point (∆ m , ∆ c ). This results in a slight vertical scatter along the horizontal scan. We used nearest-neighbor (Voronoi) partitioning to present the full two-dimensional image. In addition, we disconnected all pumps for 2 s between data points to let the mechanical oscillator thermalize and cancel any hysteresis effects due to the instability.
In Fig. 6 we plot the difference between ∆ m and the change in effective mechanical frequency of the oscillator δΩ m (due to the optical spring effect), for the same data as in Fig. 4(b). Near the onset of the two-tone instability, the two sidebands coincide, and this difference Here the stable region is smaller, highlighting the difficulty of achieving stable operation due to the inaccuracy in∆c. (c) Total noise power in the mechanical sidebands, normalized to 1 for large ∆m, for the two measurements sequences in (a). The sudden increase in power for the unstable data is evident. The solid lines are theoretical fits using the full Langevin equations. See text for more details.
approaches zero; i.e., the effective mechanical frequency Ω m + δΩ m becomes equal to the modulation frequency of the pump, Ω m + ∆ m . This corresponds to vanishing of the effective mechanical frequency in the frame rotating with the modulation frequency, as predicted theoretically in Sec. III.
Appendix C: Observation of two-tone instability in the optical domain As shown in Fig. 2(a), we also observed the twotone instability in an optomechanical system operating in the optical domain. The system is an optomechanical photonic crystal nanobeam cavity [8,65] with a mechanical frequency Ω m /2π 5.2 GHz, optical resonance frequency ω c 194 THz (wavelength λ 1540 nm), and cavity linewidth κ/2π = 300 MHz (optical Q factor ∼ 6.5 × 10 5 ). The vacuum optomechanical coupling rate is g 0 /2π = 930 kHz. The sample is measured in a 3 He buffer-gas cryostat (Oxford Instruments HelioxTL), as reported previously [41,55]. The buffer gas facilitates the thermalization of the sample, preventing deleterious optical absorption heating and allows strong pumping. The controlled pressure of the buffer gas affects the damping rate of the oscillator. The various measurements reported here were done at temperatures in the range 4.6-4.9 K, and pressures 32-160 mbar, resulting in a damping rate of Γ m /2π = 100-285 kHz. Figure 7 shows examples of the BAE measurements introduced in Sec. II, with ∆ m scanned from positive to negative values, while holding ∆ c approximately con-stant. Two measurements of similar cooperativities C ∼ 2.2-2.6 are shown in Fig. 7(a). In the lower measurement ∆ c is farther from 0, bringing it in the vicinity of the domain of instability (see Fig. 3), which is triggered on the next data point (not shown). This is the same measurement as Fig. 2(a). The separation between the data and the domain of instability is ∼ 2π × 15 MHz, well within the uncertainty in our measurement of ∆ c . Figure 7(b) shows similar data for higher cooperativity C ∼ 4, where uncertainty in measurement of ∆ c precludes discerning between the stable and unstable behavior. Figure 7(c) shows the total mechanical noise in the data of Fig. 7(a), with the theoretical fit obtained from the Langevin equations (A1). The data shown in light red, not encountering the instability, show imperfect, asymmetric BAE behavior (due to ∆ c = 0). The dark red data show the amplified noise prior to the onset of instability.