Quantum sensing and cooling in three-dimensional levitated cavity optomechanics

Typical quantum cavity optomechanics allows cooling and detection of a single mechanical degree of freedom with its motion along the cavity axis. However, a recent breakthrough using cavities populated solely by coherent scattering (CS) allowing quantum ground-state cooling of levitated nanoparticles [U. Delić et al., Science 367, 892 (2020)], is uniquely three dimensional (3D) in character, with coupling along all three spatial axes. We present a reanalysis of current experiments and show that the underlying behavior is far from the addition of independent one dimensional spectral components and that cooling and sensing analysis must consider, to date neglected, nontrivial 3D hybridization effects arising from interferences between the x, y, z modes as well as the optical modes. These lead to new heating and sympathetic cooling channels and modify phonon occupancies. Unique to these systems, we find a close relation between cavity-mediated and direct hybridization terms that can completely suppress the 3D behavior.


I. INTRODUCTION
The coupling of mechanical motion to the optical mode of a cavity permits not only strong cooling, but also ultrasensitive displacement detection, and has led to advances ranging from quantum ground-state cooling of mechanical oscillators [1,2] to detection of gravitational waves by LIGO [3]. Optomechanics employing levitated dielectric particles has recently also experienced rapid development [4,5]. The unique potential of levitated cavity optomechanics in terms of decoupling from environmental heating and decoherence, coupled with the sensitivity of displacement sensing offered by optical cavities, was already recognized in 2010 [6][7][8]. Actual experimental realizations represent a formidable technical challenge: the levitated nanoparticle must be cooled from room temperatures, and is initially millions of quanta above the quantum ground state. Most initial proposals were for self-trapping setups [7,9,10], with trapping and cooling both provided by the cavity modes [11], but this failed to allow stable trapping at high vacuum [10][11][12]. In order to overcome this roadblock, hybrid setups combining, for instance, a tweezer and cavity traps [6,13], or a hybrid electro-optical trap [14,15], or a tweezer and near field of a photonic crystal [16], allowed some progress toward the ultimate goal of quantum ground-state cooling.
This year, an important breakthrough was the realization that the tweezer trapping light coherently scattered (CS) into an undriven cavity offers major advantages [17][18][19][20][21]: the resulting optomechanical couplings along every axis can be * t.monteiro@ucl.ac.uk comparatively large even for modest mean cavity photon numbers, minimizing the deleterious effects of photon scattering [22][23][24]. As a result, quantum cooling of the center of mass of a levitated nanoparticle to phonon occupancies n x < 1 along the x axis (see Fig. 1 for definition of axes) was recently reported [25].
However, although current experiments probe the full three dimensional (3D) displacement PSD (power spectral density) S 3D xx (ω) = |x 3D (ω)| 2 , via heterodyne optical detection, the experimental analysis to date has not fully considered 3D dynamics. We show here the 3D spectra can differ from the one dimensional (1D) case both qualitatively and quantitatively. This has significant implications for thermometry and also for sensing, so here we reanalyze the quantum-cooling experiments in [25] with 3D spectra. We find we can correct the 1D displacement spectra as follows: x 3D (ω) =x 1D (ω) + G xy (ω)ŷ 3D (ω) + G xz (ω)ẑ 3D (ω) (1) by introducing a hybridization function G jk (ω) with a simple form. In particular, we obtain analytical PSDs in excellent agreement with experiment as well as with the stochastic numerics based on the full tweezer and cavity potentials. More generally for any mechanical modeq j (e.g.,q j ≡x,ŷ,ẑ), An interesting consequence of the multimode interference is that the measured PSDs are nonadditive: for instance, the weakly coupled y mode can experience strong sympathetic cooling that lowers the phonon occupancy n y significantly, due to optomechanical intermode correlations. The G jk (ω) combines interference between indirect, cavitymediated coupling and direct intermode coupling terms that have been overlooked in previous studies. A key finding is that there is a close relation between the direct and indirect channels that can lead to exact cancellation, eliminating hybridization.  [25]: a nanoparticle is held by a tweezer trap within a cavity. The cavity is undriven, but is populated by photons coherently scattered from the tweezer. The nanoparticle is placed at a point φ kx (c) 0 from the antinode of the cavity field. In [25] φ = π/2, corresponding to a near "empty cavity" with near zero mean photon occupancy. The pattern of coherent photon scattering [23] (right panel) into the cavity depends on the tilt θ of the tweezer polarization axis. (b) Analysis of the experiments: in [25], measured heterodyne PSDs showed a broad feature (attributed to the strongly coupled, quantum-cooled x mode) and a sharp narrow peak (attributed to the weakly coupled, weakly cooled y mode). Thus, the 1D thermometry considered only the broad sideband features to infer n x = 0.8 at = −380 kHz. The left panel shows calculated heterodyne PSDs (red sideband). However, comparison with S 3D xx (ω) at experimental detuning values −380 and −580 kHz [25] (both blue and red sidebands are shown in the right panels) indicates that in fact the narrow spike mostly corresponds to x motion, responding at the frequency of y. Thus, the spike must also be included in thermometric calculations approximately doubling n x relative to the 1D analysis in [25]. We show that (due to the optical spring effect) at these detunings the x mode has been pushed into a degeneracy with the y mode, enhancing hybridization, even for the "empty cavity." (c) Shows 1D vs 3D phonon occupancies for n x and n y . n 3D y can be almost an order of magnitude lower than n 1D y as a result of the sympathetic cooling induced by 3D coupling. Dotted line plots results from the standard cooling formula of optomechanics that gives perfect agreement for 1D analytics, particularly for the weakly coupled y mode. (b), (c) There is pure cavity-mediated coupling ∝g x g y as g xy 0. We have set gx 2π 80 kHz gy 2π 7.7 kHz for θ = 0.47π . The nanoparticle radius is R 0 = 71.5 nm, the finesse is F = 73 000, and the pressure is P = 10 −6 mbar.

II. TWEEZER-CAVITY SETUP
We consider the experimental CS setup in [22][23][24][25], shown in Fig. 1(a), which involves levitating a dielectric nanoparticle in a tweezer within a cavity. The tweezer and cavity symmetry axis are orthogonal and the tweezer polarization and the cavity axis are tilted at an angle θ . The cavity in these setups is undriven but is populated entirely by light coherently scattered from the tweezer field. The cavity decay rate is κ and the particle moves under the combined effect of the tweezer trapping field and the coherently scattered light. The tweezer ω L is detuned by a frequency = ω L − ω c < 0 from the cavity resonance at ω c .
In [22][23][24][25], the CS system was found to yield the potential (see also Appendix D) where φ ∼ kx (c) 0 , x (c) 0 is the displacement between the tweezer focus and an antinode of the cavity [see Fig. 1(a)], β(z) = kz − arctan(z/z R ), z R is the Rayleigh range, k = ω c /c, c is the speed of light, and E d is the coupling rate determined by the particle polarizability and input power to the tweezer.
Optomechanical coupling strengths were derived in [22][23][24] by the well-known procedure of linearizing Eq. (3) about small displacements from the tweezer origin and the mean cavity amplitudeᾱ. In particular, they have obtained the interactionV int h jĥ Specifically, the optomechanical coupling strengths g j were obtained in terms of the experimental parameters, i.e., g x x zpf sin θ = g y y zpf cos θ = −E d k sin(φ), where q j,zpf = h/2mω j , and q j denotes the mechanical mode. An interesting finding there was that the z mechanical mode is coupled to the phase quadrature and the coupling is of the nonstandard form for optomechanics, i.e., g z i(â † −â)ẑ, where g z = −E d k cos(φ)z zpf .

III. DIRECT COUPLING TERMS AND HYBRIDIZATION FUNCTION
In this work, we take the previous analysis of couplings a step further: a full linearization (quadratic in the Hamiltonian) of Eq. (3) should include additional terms directly coupling the motionsV int h = jĥ (int) j + j<k g jkq jqk . We find g xz are the components of the mean intracavity fieldᾱ. Thus, these (previously overlooked) direct terms are light induced, but where the degrees of freedom of the electromagnetic field do not play an active role. They are thus quite distinct from nonlinear, position-squared coupling terms g j (â † +â)q 2 j which lead to observed sidebands at 2ω j in optically trapped systems at higher temperatures [15,23].
To obtain 3D spectra, we start from a generic quadratic Hamiltonian including both cavity-mediated and direct matter-matter couplings. By writing the equations of motion in Fourier space we solve for the mechanical motions and obtain the 3D couplings (see Appendix A) which were introduced in Eq. (1). The prefactor, where M j (ω) = 1 + g 2 j μ j (ω)η 0 (ω), is peaked around one of the mechanical frequencies, i.e., ω ≈ ±ω j . The μ j (ω) are mechanical susceptibilities, while η 0 , η jk are the optical susceptibilities. However, it is the terms in the square brackets that are of most physical interest. One can see they describe the interference between a direct, ∝g jk , and a cavity-mediated, indirect coupling, ∝g j g k , between any two displacements. In other words, suppressing or conversely enhancing 3D dynamics will involve either suppressing or correspondingly enhancing the 3D coupling via destructive or constructive interference of direct and indirect pathways near ω ≈ ω j .
With these susceptibilities, the G jk are fully specified and Eq. (1) can be solved self-consistently to yield 3D PSDs of all the mechanical and optical modes that can be used to investigate experimental regimes. As all our PSDs are given in terms of the G jk , the PSD expressions do not depend on whether the coupling is direct (g k j g j g k ), indirect (g k j g j g k ), or intermediate. There is no advantage in numerical prediagonalizations to put the mechanical Hamiltonian in normal formal; such a procedure is parameter dependent as g k j depends on , thus complicating experimental comparison. As we will see, Eqs. (1) and (4) give a full quantitative as well as intuitive explanation of the physical mechanisms involved in the quantum-cooled experiments [25].

IV. ANALYSIS OF GROUND-STATE COOLING EXPERIMENTS
Although all calculations presented in the figures employ the full 3D analysis, for physical understanding here we discuss thermometry of the x-y motion, most relevant to the experiments in [25]. The z motion is effectively decoupled as its frequency is not close to the ones of the x and y motions. Those experiments used φ = π/2 so were in the regime of pure cavity-mediated coupling, i.e., "empty cavity," where θ (0.47-0.49)π , and hence g x g y . In Fig. 1(b) our expressions reproduce key experimental features. However, comparison between optical and x-motion PSDs shows that the 1D analysis decomposing heterodyne sidebands into (i) a broad "x-motion" plateau and (ii) a sharp "y-motion" peak, does not yield reliable thermometry if there is hybridization [ Fig. 1(c)]. To establish how much of the heterodyne sideband area belongs to the strongly cooled x mode, we need to consider the full 3D spectra.
For understanding, we can simplify our 3D PSDs for both x and y motion: For the x motion, the important difference between the S 3D xx and S 1D xx arises from the second term in Eq. (5). This latter term is not an interference term, but an additive term, which always results in additional heating, and it provides the sharply peaked feature around ω ω y . For the y motion, on the other hand, the correction in Eq. (6) arises from back-action-induced correlations between the modes: it is an interference effect that can either heat or cool. In [25] it opens a new sympathetic cooling channel for the y motion as shown in Fig. 1(c). In Appendix E, we show these rescaled 1D sidebands in Eq. (5) [and (6)] accurately account for the differences between the full 3D and 1D PSDs.
The heterodyne detection records the y motion with amplitude ∝g 2 y |η 0 (ω)| 2 S 3D yy (ω) (where frequencies are shifted by the appropriate reference oscillator). On the other hand, the y hybridization component, i.e., the x motion responding at the frequency of y motion in the second term in Eq. (5), scales as ∝g 2 x |η 0 (ω)| 2 |G xy (ω)| 2 S 3D yy (ω). The y : x component ratios may be estimated 1 : R where We have g 2 x g 2 y 100 and considering the hybridization region = −380-420 kHz we find R ≈ 3-5; thus, ≈80%-90% of the sharp peak is due to hybridization and hence contributes to n x thermometry. However, for the strongest cooling data at = −300 kHz, we find R = 1.1, and thus in the strongest x 023228-3 cooling region only half the ω ω y peak is due to hybridization.
In [25], a spectrum at −380 kHz was analyzed in detail to yield n x 0.8. Revisiting with the 3D analysis, we estimate a near doubling of the area, thus, n x 1.5. A further experimental data point was reported (without spectral details) for −315 kHz that yielded n x = 0.4 from 1D analysis. In the Supplemental Material of [25] the full sideband area including the peak was given at n x = 0.9. This sideband is completely dominated by the "y" peak. However, for R = 1.1 only half the peak is x motion and one can thus estimate n x ≈ 0.6-0.7.
From the above analysis, we can see that the sharp peak to a good approximation carries the asymmetry of the y motion. If one eliminates the asymmetry introduced by the cavity susceptibility function |η 0 (ω)| 2 , the underlying asymmetry of the sharp ω ω y peak is n y (3D) + 1 : n y (3D). In contrast, the asymmetry of the broad feature is closer to n x (1D) + 1 : n x (1D). This is in sharp contrast to the usual scenario in optomechanics where the red and blue sidebands have exactly the same shape but are simply rescaled by a factor n/(n + 1) where n is the appropriate occupancy.

V. SUPPRESSION OF HYBRIDIZATION
The direct coupling g xy has not previously been considered in the experimental analysis [22][23][24] but we find they can be of great importance. In particular, one can show that g xy −g x g y we then readily find Thus depending on the positioning or κ, the direct couplings' contribution can be similar or exceed the cavitymediated coupling.
In Fig. 2 we compare analytical, closed-form PSDs we obtained with 3D QLT and Eq. (1), with direct solutions of the nonlinear Langevin equations of motion, using the tweezer and cavity potential functions. In the latter, the g j and g jk are not parameters but rather simply emergent properties in the limit of low-amplitude displacements. The symmetrized analytical quantum spectra show excellent agreement with numerics in both quantum regimes as well as thermal (higherpressure regimes) provided the latter are cooled enough so that nonlinearities do not generate additional peaks in the optical spectra [15]. Furthermore, Fig. 2 also demonstrates the importance of the previously neglected g k j terms: in particular, leading to double-peaked structures (x-y hybridization) for φ 0 where cavity-mediated coupling g x g y 0 terms are negligible, as well as φ → π/2, where g xy → 0, but the cavity-mediated coupling from g x g y are strong.
However, the φ = π/4 case is the most interesting and represents a key finding: here, the x-y hybridization almost fully vanishes. Although both direct and indirect contributions are strong, they interfere destructively. We can show that iη (0) (ω) → −2 (κ/2) 2 + 2 if − ω (and we are interested primarily in the region ω ∼ ω j ). Thus, for large − , using FIG. 2. Comparison with full numerics and demonstration of 3D → 1D transitions. Compares displacement PSDs using analytical expressions for the 3D theory (dashed lines) with stochastic numerics using the tweezer and cavity potentials (solid lines). The stochastic numerics do not assume any values for the optomechanical coupling strengths, mechanical frequencies, or equilibrium positions, and include nonlinearities. Agreement is excellent and validates the analytical model for x (black), y (red), and z (blue). The figure also illustrates the transition from (i) 3D dynamics at low φ (top panel; direct coupling ∝g xy ) to (ii) 1D uncoupled dynamics at φ ≈ π/4 (middle panel) back to (iii) 3D dynamics at large φ ∼ 0.4π (bottom panel; indirect cavity mediated coupling ∝g x g y ). Regimes of 3D dynamics have a high degree of hybridization of x, y modes resulting in a prominent double-peaked structure. 1D dynamics results from suppression of hybridization as seen at φ ≈ π/4 (middle panel), where destructive interference between the direct and indirect pathways decouples the modes for − ω j . Parameters similar to the experiment in [23]: input power P in = 0.17 W, = −300 kHz; however, nanoparticle radius R 0 = 100 nm and finesse F = 150 000 are slightly larger, with gas pressure P = 10 −6 mbar, and θ = 0.2π .
Eqs. (4) and (8), we can readily show and the x, y coupling G 3D xy (ω) thus vanishes. The transition from 3D to a near decoupled 1D regime seen above is further illustrated in Fig. 2 (middle panel). The previously overlooked coupling g xy can thus lead to nontrivial consequences. For example, a calculation in [24] [ Fig. 6(a) there] at φ = π/4, − > κ, but without direct coupling terms, shows a x-y hybridization feature in a regime where it should be strongly suppressed.

VI. CONCLUSIONS
We have shown that 3D optomechanical displacement sensing can be far from a trivial sum of PSDs associated to thex,ŷ, andẑ degrees of freedom. CS systems have provided the breakthrough that enabled ground-state cooling in levitated nanoparticles: for reliable thermometry one should either avoid or suppress hybridization regimes or do a full 3D analysis. Although our work focuses specifically on recent experiments on 3D cooling of levitated nanospheres, some of the conclusions are generic. We show one may be able to switch on or switch off some of the additional 3D effects and that these can give advantages in terms of exceeding the usual quantum back-action-limited occupancies for a given coordinate. 3D optomechanics opens the way to new forms of force and displacement sensing, including sensing the direction as well as magnitude.

ACKNOWLEDGMENTS
We are extremely grateful to U. Delić for advice and for sharing with us details of the experimental data. We acknowledge support from EPSRC Grant No. EP/N031105/1.

APPENDIX A: QUANTUM LINEAR THEORY (QLT)
In this Appendix we review 1D QLT and provide details of the derivation of our 3D QLT (quantum linear theory) of optomechanics expressions.

Standard optomechanics QLT
In this Appendix we briefly review the framework of quantum linear theory (QLT) of optomechanics. Optically levitated systems [26,27] generally involve multiple optical and mechanical modes. Such multimode systems (N optical and M mechanical degrees of freedom) are typically described by the well-studied linearized Hamiltonian [1] is the annihilation (creation) operator for optical mode l, andb k (b † k ) for mechanical mode k. l is the detuning between the input laser and the cavity mode l, while ω k is the natural frequency of the mechanical oscillator, and g (l ) k is the light-enhanced coupling strength between an optical mode and a mechanical mode. For simplicity, dissipation is characterized by a single optical damping rate κ and a single mechanical damping rate (though more complex scenarios, for example with multiple mirror losses, can be easily incorporated).
A set of 2(N + M ) quantum Langevin equations of motion are obtained from Eq. (A1) by adding input noises. For example, for the single-mode whereâ in (b in ) is the optical (mechanical) input noise. The above equation even for arbitrary numbers of modes can be cast in matrix formċ where the vector c = (â 1â † T , the matrix A contains the frequencies of the problem, and c in are Gaussian input noises (incoming quantum shot noise in the ideal case in the optical modes and thermal noise for the mechanical noises). Multimode theoretical PSDs are efficiently computed using a the linear amplifier model (LAM) [28]. For the LAM, the first step involves transforming the equations of motion into frequency space. The coupled equations are then manipulated analytically (or even numerically if unavoidable) to recast the matrix equation of the equations of motion in the form where T(ω) = (−iωI − A) −1 and I is the identity. T is a transformation matrix that characterizes the transduction of the input noises into the mechanical and optical field fluctuations, somewhat analogous to the effect of a linear amplifier. The linear amplifier model is very powerful as one may in principle obtain the vector of all PSDs of all modes in one go: where and N is a diagonal matrix of elements: The n k represent the occupancy of the respective baths, thus n k = 0 for quantum shot noise in the optical modes but n k kT /hω k for thermally occupied phonon modes. Typically, one can set γ k ≡ κ for optical modes and γ k ≡ for the mechanical modes. For levitated systems, there is no cryogenic cooling and T = 300 K. The solutionsâ l (ω) of the optical field denote here the intracavity field, while the actual detected cavity output field is then obtained using the input-output relationâ out l (ω) = a in l (ω) − √ κâ l (ω) for the respective optical mode.

1D QLT with amplitude or phase optical coupling
In this section we consider one mechanical modeb j and one optical modeâ with two types of couplings: is the usual optomechanical coupling between the mechanical mode and the amplitude quadrature of light which has been reviewed in Appendix A 1. To obtain the PSD one can restrict the general multimode result in Eq. (A5) to the case of one optical and one mechanical degree of freedom by setting N = M = 1. Alternatively, an explicit calculation of the PSD can be performed by following the steps from Eqs. (A4)-(A7). Specifically, from Eq. (A2) one first moves to the frequency space and solves for the displacement operatorq(ω) = b(ω) +b † (ω), i.e., the displacement operator is expressed in terms of noisesâ in (ω) andb in (ω). The PSD can then be readily obtained by evaluating the expectation value using Eqs. (A6) and (A7). The latter case (ii), where the mechanical mode is now coupled to the phase quadrature of light, can be analyzed using analogous steps. Specifically, one first obtains the quantum Langevin equations with the modified coupling and then follows the steps in Eqs. (A4)-(A7). For both cases (i) and (ii) we can write the 1D displacement operators using the standard notation of optomechanical QLT by expressing them as a sum of thermal and optical backaction noises: where we have the normalization factor the mechanical susceptibilities the mechanical noisẽ the optical susceptibilities, defined in the text as η jk (ω) may be given in a more general form so as to treat the optical, homodyne spectra so we see that η 0 in the main text is ≡ η ( =0) (ω). The optical noiseQ and we have defined The above are (almost) the standard expressions for the 1D quantum linear theory (QLT) of optomechanics. The only difference is that we specify an angle for the optical noise, such that = 0 for standard optomechanical coupling, i.e., coordinates coupled to the amplitude quadrature of light, but = π/2 for the coordinates coupled to the phase quadrature of light (such as the z coordinate in the experiments in [23]).
For a cavity modeâ, homodyne optomechanical displacement sensing in 1D would involve a measurement of some quadrature of the optical fieldQ = e −i â + e i â † , with coupling to a mechanical displacementq: whereQ in represent measurement imprecision, typically from incoming quantum photon shot-noise fluctuations, while κ is the cavity linewidth. Understanding the standard quantum limit of displacement sensing in 1D optomechanics requires analysis of errors in Eq. (A15).
In the 3D case, the measured optical quadrature in general now couples to displacementsq j along all directions j = x, y, z:Q With a simple adjustment to relate the intracavity field to the cavity output field via input-output relations, the corresponding PSD of the measured signal is used to estimate a displacement spectrum SQ Q /|η ( ) | 2 gS 1D qq in the 1D case. A key question is whether one might straightforwardly extend to the 3D displacement spectra by simply considering the sum of the independent contributions, i.e., SQ Q /|η ( j ) | 2 j=x,y,z g j S 1D q j q j . In 3D we find this is not the case. We find that the most significant differences are new 3D back-action terms that redistribute energy between mechanical modes and add additional optical back-action thus attaining the 3D SQL requires suppressing these terms. Nevertheless, even if thê q j were independent, the separate optical back-action contributions are correlated and interfere for the case of singlequadrature homodyne detection leading to differences in ponderomotive squeezing. For the experiments using heterodyne detection (rotating quadrature), ponderomotive squeezing is not relevant.

3D QLT with amplitude and phase optical coupling
In this section we consider three mechanical degrees of freedom that are coupled to both the amplitude and phase quadratures of the optical degree of freedom. Specifically, we consider the interaction Hamiltonian given bŷ whereŶ =â † +â andP = i(â † −â), andq j =b † j +b j denotes the mechanical degrees of freedomx,ŷ,ẑ. The starting points of the analysis are again the equations of motion written in frequency domain: where J jk (ω) = ig jk μ j (ω) (for j = x, y, z, and k = x, y, z, Y, P), J Y j (ω) = i(g j χ (ω, − , κ ) −g * j χ * (−ω, − , κ )), J P j (ω) = (g j χ (ω, − , κ ) +g * j χ * (−ω, − , κ )), and we have defined the complex-valued couplingsg j = g jY + ig jP . Note that in the main text, where we discuss a special case, we use the more common notation of real-valued couplings: g x ≡ g xY , and g y ≡ g yY , and g z ≡ g zP . The input noises are given bỹ whereQ therm j (ω) denotes the mechanical noisesX therm (ω), Y therm (ω),Z therm (ω).
It is instructive to separate the contributions to the spectra ofq j (ω) into two categories: one that contains the terms of a 1D approximation and one that contains additional terms arising in a realistic 3D problem. Specifically, using Eqs. (A19) and (A20) we can rewrite Eq. (A18) aŝ where we made the low-order approximationq 3D j →q 1D j to allow a simplified analysis.
For example, for the 3D coherent scattering discussed in the main text we find (see Appendix C for more details) for j, k = z, thus, for G xy (ω) and G yx (ω), while where the indices j, k denote x or y. We now continue with the general analysis. For numerical accuracy, we here give the exact expressions for the displacements in terms of noises. Specifically, starting from Eqs. (A18)-(A20) we eventually find q j (ω) = A j (ω)Ỹ in (ω) + B j (ω)P in (ω) + C j (ω)X therm (ω) where j denotes one of the mechanical motions where β jY = N j J jY , β jP = N j J jP , β j j = N j , N j = (1 − J jY J Y j − J jP J P j ) −1 . We have defined the coefficients ξ j j = 1 − 1 2 R kl R lk (with l, k = j and k = l), ξ jk = R jk + R jl R lk (with j = k, l = k, and l = j), and R jk = N j (J jY J Y k + J jP J Pk + J jk ). The overall normalization is given by . The PSDs can be readily obtained from Eq. (A28) using the methods discussed in Appendix A 1.

Self-energy, optical spring, and damping
In this section we obtain the analytical expressions for the self-energy, relevant to the experiments of [25] that couple x and y (but not significantly z). We obtain also the resulting optical spring and damping formulas. We start from the coupled equationsx We now focus on the x motion, while the formulas for the y motion can be obtained by formally exchanging x ←→ y in the formulas. Specifically, we solve forx to find (A37) One can then extract the self-energy x , which is given by We note that the numerator is the usual term ∝g 2 x which arises already in the 1D analysis, while the denominator term ∝g 2 y is a new effect which arises in the 3D analysis. In particular, considering the expressions for J xY and J Y x we find from Eq. (A38) Finally, we can find the change of the damping δ and the shift of frequency δω using the following expressions [29]: where ω j denotes the mechanical frequency.

APPENDIX B: SUPPRESSION OF z HYBRIDIZATION
In this Appendix we discuss suppression of hybridization with the z motion. In the main text we found that a remarkable transition from 3D to a near decoupled 1D regime occurs for − ω x,y and φ = π/4, in-between the 3D (direct coupled, φ 0) and 3D (indirect, cavity-mediated φ π/2) regimes. This results from the cancellation between the direct g xy coupling and the cavity-mediated g x g y η (0) terms; an underlying reason for this surprising near exact cancellation is that g xy ∝ Re(ᾱ) whereᾱ is the mean cavity field, which follows the cavity resonance, that in turn determines the form of η (0) .
However, the situation for the G jz (ω) couplings is similar but more involved ( j = z) so the destructive cancellation is less complete. A peculiarity of the system is that the z coupling is of the form g z i(â † −â)ẑ, i.e., the displacement couples to the momentum quadrature of the cavity. In this case, . In other words, G jz = G z j and both couplings cannot be suppressed simultaneously. In any case, using the equation from the main text g xy g x g y 2 cot 2 φ 2 + κ 2 4 , g jz g j g z κ and for large values of − where iη (π/2) → κ−2iω 2 +(κ/2) 2 , we find that G jz (ω) ∝ 2g j g z iω 2 +(κ/2) 2 and G z j (ω) ∝ 2g j g z [κ+iω] 2 +(κ/2) 2 . Thus, even where there is destructive interference, only the real part of G jz (ω) is fully canceled. Nevertheless, all 3D couplings are attenuated for − ω j , κ. Further, since fortunately since ω z ω x,y , hybridization between z and the other two modes is generally weaker than between x and y which are close in frequency. Thus, it is possible to tune quite strongly into the decoupled 1D regime.

APPENDIX C: 3D LEVITATED OPTOMECHANICS IN A CAVITY DRIVEN BY COHERENTLY SCATTERED TWEEZER LIGHT
We consider the 3D optomechanical system such as the coherent scattering cavity levitation introduced in [22][23][24]. As discussed below, some of the optomechanical coupling terms are of the form ig k (â † −â)(b † k +b k ), i.e., the mechanical motion can couple to the phase quadrature of the light, in addition to the more typical coupling to the amplitude quadrature, i.e., g k (â † +â)(b † k +b k ). Specifically, we will consider the case when the z motion has the former type, while x and y motions have the latter one.
In addition, for a truly 3D system, one allows also direct couplings between the mechanical modes, i.e., g kk q kqk , where k, k ≡ x, y, z. Direct couplings are not usually considered in optomechanics: although multimode systems are commonly studied (such as multiple vibration modes of membranes), coupling between mechanical modes is not usually of interest. However, for the considered 3D optical levitation this is not only important, but the g kk are closely correlated with the couplings g k . Specifically, we will find g kk ∝ g k g k , which has important consequences for sensing.
In particular, in a cavity populated only by scattered light as in the recent 3D setups in levitated optomechanics, we need consider only a single light mode, but three mechanical modes including direct coupling terms: whereĤ 0 = − â †â + k=x,y,z ω kb † kb k (see Appendix A 3 where we have developed a generic framework to solve such Hamiltonians within QLT). In order to extract the dynamical parameters, i.e., the frequencies ω k , the optomechanical couplings g k , and the direct couplings g k j , we must first consider the physical tweezer and cavity potentials (Appendix D), and then expand to quadratic order around an equilibrium position (Appendix D).

APPENDIX D: 3D COHERENT SCATTERING HAMILTONIAN
In this Appendix we review details of the potentials in the coherent scattering setup and derive the optomechanical couplings g j as well as direct couplings g jk for j, k = x, y, z. We consider the hybrid tweezer-cavity experiments introduced in [23], which employed setups very similar to those in [22,24]. A nanoparticle is trapped at the focus of a tweezer field and interacts with light coherently scattered from the tweezer field into the (undriven) cavity.
The Hamiltonian describing the interaction between the nanoparticle and the combined fields of the tweezer and cavity is given byĤ whereÊ cav (Ê tw ) denotes the cavity (tweezer) field, α = 3 0 V s R −1 R +2 is the polarizability of the nanosphere, V s is the volume of the nanosphere, 0 is the permittivity of free space, and R is the relative dielectric permittivity.
We assume a coherent Gaussian tweezer field and replace the modes with c numbers to find is the Rayleigh range, w x (w y ) are the beam waist along the x (y) axis, tw = 4P tw w x w y π 0 c is the amplitude of the electric field, c is the speed of light, P tw is the laser power, ω tw is the tweezer angular frequency, t is the time, andr = (x,ŷ,ẑ) is the position of the nanoparticle. e j are the unit vectors: e z is aligned with the symmetry axis of the tweezer field and e y is aligned with the polarization of the tweezer field.
The cavity field is given bŷ where c = hω c 2 0 V c is the amplitude at the center of the cavity, V c is the cavity volume, ω c is the cavity frequency,â (â † )is the annihilation (creation) operator, x (c) 0 is an offset of the cavity coordinate system (centered at a cavity antinode) with respect to the tweezer coordinate system. The cavity x c -y x plane is rotated by an angle θ with respect to the tweezer x-y plane: Note that for θ = 0 the tweezer polarization (y axis) becomes aligned with the cavity symmetry axis (x c axis). In particular, we havex (c) = sin(θ )x + cos(θ )ŷ. Furthermore, we then have the following relation between the cavity and tweezer unit vectors where the terms on the right-hand side are the tweezer term, the cavity term, and the tweezer-cavity interaction term (from left to right). The first (tweezer field) term dominates the trapping and primarily sets the three mechanical frequencies ω x , ω y , and ω z . The second term provides a (typically) small correction to the frequencies and is included only for numerical precision. The third term, which we will denote aŝ V int , is the most interesting and novel form of optomechanical interaction. As discussed in [23,24], time dependencies in this term are eliminated through rotating frame approximations leaving an effective optomechanical Hamiltonian where E d = α c tw sin θ 2h , φ = kx (c) 0 represents the effect of the shift between the origin of the cavity and tweezer. The experiments allow positioning of x (c) 0 with an accuracy of ∼8 nm for λ = 1064 nm. In first approximation one can neglect the Gouy phase . Linearization of the above Hamiltonian to quadratic order yields the 3D optomechanical couplings.

Quadratic Hamiltonian: Unified form
We expand the Hamiltonian in Eq. (D7) around an equilibrium point (x 0 , y 0 , z 0 ,ᾱ) by making the substitution (x, y, z, a) → (x 0 , y 0 , z 0 ,ᾱ) + (x, y, z, a) , where (x, y, z, a) on the right-hand side denotes small fluctuations. To a first approximation, x 0 , y 0 , z 0 represents the origin of the strong tweezer trap. However, there are nontrivial corrections when we have further small offsets δx 0 , δy 0 , δz 0 , for example, when the cavity is strongly populated [24]. These emerge naturally from numerical simulations and can also be well estimated through the linearization analysis. Similarly, the mean cavity photon occupancy number n = |ᾱ| 2 may also need to be corrected from the approximate form [23]ᾱ = (Y 0 + iP 0 )/2 = −iE d cos(φ)/(i + κ/2) to allow for the fact that φ kx (c) 0 + δx 0 cos θ + δy 0 sin θ . We further note that also δz 0 displacements give rise to nontrivial changes for the couplings and the mean values, yet the results for the x-y motions in the main text and in Appendix E remain unaltered as δz 0 z R . A unique feature of these new levitated setups is that the optomechanical coupling can be via the momentum quadrature. In the calculation in [22][23][24], this affected only the z coordinate. However, we note that if there are significant offsets in the fields or misalignment of the cavity and tweezer axes, in general, one might wish to consider both amplitude and momentum couplings to all mechanical modes, so here we introduce a unified notation.
It is convenient to introduce the notationŶ =â † +â, and P = i(â † −â) for the optical field. We also similarly usê zero-point fluctuation lengths are given by x zpf = h 2mω x , y zpf = h 2mω y , and z zpf = h 2mω z , and m is the mass of the levitated nanoparticle. Redefiningx/x zpf →x,ŷ/y zpf →ŷ, and z/z zpf →ẑ we writê where we have omitted the harmonic oscillator terms. We can also rewrite the optical quadratures in terms of the mode operatorâ: where we have introduced the complex-valued couplingsg j = g jY + ig jP (see Appendix A 3 for the resolution of this general Hamiltonian within 3D QLT). However in the following we opt to use the more conventional notation introduced for the special case in Eq. (C1) where all the coupling constants are defined as real valued. Specifically, from Eq. (D7) we find the following nonzero light-matter couplings: In addition, we also have matter-matter couplings The harmonic frequencies are given by where are the typically dominant contributions arising from the tweezer trap. The corrections from the cavity are C x = 2α 2 c k 2 n sin 2 (θ )cos(2φ) and C y = 2α 2 c k 2 n cos 2 (θ )cos(2φ), and C z = 0. The contributions arising from the coupling between the cavity and tweezer are T c . We remark that corrections from C j and T c j can in certain cases become important, e.g., when the cavity has a high photon occupancy, potentially even leading to nanoparticle loss. The cavity-tweezer interaction also changes the cavity detuning from to .

APPENDIX E: ANALYSIS OF GROUND-STATE COOLING EXPERIMENTS
In this Appendix we give further details of our analysis (based on the 3D QLT presented in Appendix A) of the recent experiments reporting ground-state cooling of levitated nanoparticles.
We employ a simplified analysis to aid understanding of the recent experiment reported in [25] which employs the 3D coherent scattering setup discussed in the main text. The experiment places the particle at the node (φ = π/2) and is thus in regime of pure indirect, cavity-mediated coupling, which differs significantly from the regimes where direct/indirect pathways compete and cancel. Nonetheless, there are other novel and important features. The analysis confirms that thex motion is close to the ground state and identifies new effects in thex andŷ displacement spectra stemming from hybridization between thex to theŷ motions. In particular, we find nonnegligible corrections to phonon occupancies in both modes.
We have the following x-y hybridization coupling strengths (see also Appendix A here) However, as the nanoparticle is located at a cavity node φ = π/2, they involve only the indirect cavity-field-mediated coupling terms ∝η 0 (ω)g x g y since the direct coupling coefficients vanish, i.e., g xy = 0. Hence, the x-y hybridization couplings in the case of pure cavity-mediated interactions reduce to where j, k denote the indices x and y. Interestingly, although in this configuration the cavity contains very few photons (the so-called "empty cavity" with only components at the Stokes/anti-Stokes frequencies), the indirect couplings still play a very important role. In addition, the tweezer tilt angle is set to values θ ≈ π/2, thus g y g x , so one expects strong cooling exclusively along the x direction. However, the heterodyne detected PSDs showed prominent peaks at ω ω y (shifted by the reference oscillator), thus, one infers that g y = 0 so θ = π/2. Allowing for an uncertainty in the tweezer tilt of a few degrees, we have thus assumed θ = (0.47-0.49)π to be consistent with the observations. For θ = 0.47 × π , we obtain g x ≈ 2π × 80 kHz and g y ≈ 2π × 8 kHz, thus g x ≈ 10g y .
A further detail of the observed data motivates a very small (10%) adjustment of the tweezer waist dimensions. Figure 3 shows an optical-spring-induced frequency degeneracy between the x and y modes at ≈ −400 kHz, which is a feature of the experiments. In order to get agreement in the x, y frequencies as well as the frequency degeneracy, the tweezer waist values w x = 0.66 μm and w y = 0.77 μm in [25] were reduced slightly to w x = 0.600 μm and w y = 0.705 μm, which is consistent with inherent experimental uncertainties in the tweezer geometry. Figure 3 illustrates key features of the experimental regime in [25], including the optical-spring-induced degeneracy, the showing two notable features: (i) as the detuning approaches − ∼ ω x , the x motion is optomechanically cooled to occupancies close to the ground state; (ii) as is lowered, since g x g y , the optical spring effect reduces ω x , but leaves ω y unperturbed, resulting in a frequency degeneracy ω x ∼ ω y that enhances hybridization effects and 3D heating/cooling channels. In particular, we note that S xx , thex PSD, contains a sharp peak at ω ω y due to the hybridization. (Middle panel)ŷ-motion PSD S yy (ω). The new 3D hybridization results in significant cooling when the mechanical frequencies are nearly degenerate (black line). (Right panel) Phonon occupancies n x and n y as a function of detuning, where 1D (3D) indicates a simplified one dimensional (full three dimensional) analysis. We note that the phonon occupancy for thex motion indicates n x (3D) > n x (1D). Theŷ motion is cooled most effectively at the hybridization point, ω y ∼ ω x . This latter effect is in contrast with the behavior expected from a simplified 1D analysis, where cooling is most effective at − = ω y . Black dashed lines denote results from standard optomechanics cooling formula [see Eq. (E11)]. We take θ = 0.47π . cooling dynamics, and the hybridization. This is the scenario we now analyze. The z motion has a frequency ω z ω x , ω y , and can thus be neglected in the simplified analysis below (but is included in the numerics). In this regime the x and y mechanical motions form a system of coupled equations, which in frequency space take the form The termsx 1D (ω) andŷ 1D (ω) denote the optical and mechanical noises which would be present already in a one dimensional analysis, and the hybridization couplings G jk (ω) are given in Eq. (E2).

Simplified analysis of the y motion
Substituting Eq. (E3) into Eq. (E4) we approximatê where we have for brevity omitted the frequency dependence ω. In other words, we directly used the 1D forms on the righthand side for the simplified thermometry analysis (although we emphasize all calculations for the main text figures used the full self-consistent 3D solutions without approximation as discussed in Appendix A 3).
The above show that the optomechanics introduces correlations between the x and y motions although the corresponding thermal noise fields are uncorrelated. As we are operating relatively far from the back-action limit, we neglect in the first instance the optical noises and hence the optically induced correlations betweenx 1D andŷ 1D . We note, however, that the above optomechanically induced correlations between the x and y modes are somewhat analogous to the well-studied correlations between optical and mechanical modes induced by optomechanical back-action.
As x is strongly cooled, we can in this case neglect thex 1D term. Hence, and thus we arrive at an an approximate expression for the PSD ofŷ: Figure 4 compares the above N -rescaled PSD with the full analytical expressions, showing that the rescaling of the 1D sideband accurately accounts for the differences between the 3D and 1D PSDs including the relative heating and cooling. In summary, around the frequency degeneracy, there is strong (about factor 7) cooling of the y motion due to the x-y correlations and the back-action of y on x, i.e. the y mode is, via the cavity, coupled to x, and in turn the x mode, because of this cavity-mediated coupling, acquires a component correlated with the y thermal noises. We identify this as a new mechanism for "sympathetic cooling" of the y mode, due entirely to the strongly coupled (and strongly cooled) x mode.

Simplified analysis of the x motion
The x motion can be analyzed in similar manner, by substituting Eq. (E4) into Eq. (E3) which readily giveŝ where we have for brevity omitted the frequency dependence ω. Analogously, we find the PSD where we have made the further approximation, based on inspection of the form of N (ω), that S 1D xx S 1D xx |N | −2 ; in other words, the back-action of highly cooled x motion on the PSD of x, arising from its coupling to y, is relatively unimportant. The important difference between the S 3D xx and S 1D xx arises from the second term in Eq. (E9). This latter term is not an interference term, but an additive term, which always results in additional heating, and it provides the sharply peaked feature around ω ω y . This feature has previously been neglected, but its contribution to the sideband area should be included for accurate thermometry.
We note that the x sideband is strongly affected by the optomechanical spring effect. It is straightforward to adapt the usual analysis for this x-y coupled case. One obtains the usual self-energy [29] (see Sec. S3 for more details) x ≡ g 2 x η (0) 1 + g 2 y μ y η (0) (E10) from whence we find the change of the damping δ and the shift of frequency that represents the optical spring effect δω. Specifically, we find [29] δ j = Im[ j (ω j )] ω j , where ω j denotes the mechanical frequency. Note the g 2 y correction in the denominator of the self-energy; setting this to zero yields the standard 1D optomechanical cooling formula. In Fig. 3 we compared phonon occupancies (black dashed lines, right panel) obtained in this way n x n B γ g / x where the thermal bath occupancy n B = kT /(hω x ) for T = 300 K. We note that the effect of this g 2 y correction is small and the significant effects in heating of x arise rather from the hybridization correction [the last term in Eq. (E9)].