This site uses cookies. By continuing to use this site you agree to our use of cookies. To find out more, see our Privacy and Cookies policy. Close this notification
NOTICE: Ukraine: Read IOP Publishing's statement.
Paper The following article is Open access

# Interference effects in hybrid cavity optomechanics

, and

Published 16 January 2019 © 2019 IOP Publishing Ltd
, , Citation Ondřej Černotík et al 2019 Quantum Sci. Technol. 4 024002

2058-9565/4/2/024002

## Abstract

Radiation pressure forces in cavity optomechanics allow for efficient cooling of vibrational modes of macroscopic mechanical resonators, the manipulation of their quantum states, as well as generation of optomechanical entanglement. The standard mechanism relies on the cavity photons directly modifying the state of the mechanical resonator. Hybrid cavity optomechanics provides an alternative approach by coupling mechanical objects to quantum emitters, either directly or indirectly via the common interaction with a cavity field mode. While many approaches exist, they typically share a simple effective description in terms of a single force acting on the mechanical resonator. More generally, one can study the interplay between various forces acting on the mechanical resonator in such hybrid mechanical devices. This interplay can lead to interference effects that may, for instance, improve cooling of the mechanical motion or lead to generation of entanglement between various parts of the hybrid device. Here, we provide such an example of a hybrid optomechanical system where an ensemble of quantum emitters is embedded into the mechanical resonator formed by a vibrating membrane. The interference between the radiation pressure force and the mechanically modulated Tavis–Cummings interaction leads to enhanced cooling dynamics in regimes in which neither force is efficient by itself. Our results pave the way towards engineering novel optomechanical interactions in hybrid optomechanical systems.

Export citation and abstract

Original content from this work may be used under the terms of the Creative Commons Attribution 3.0 licence. Any further distribution of this work must maintain attribution to the author(s) and the title of the work, journal citation and DOI.

## 1. Introduction

Cavity optomechanics [1] has reached a remarkable success in coupling high-quality mechanical resonators and light via radiation pressure. This interaction can be used for measurements of small mechanical displacements and external forces [25], for quantum state transfer between the cavity field and the mechanical oscillator, and for ground state mechanical cooling [6, 7]. Other achievements are frequency conversion between cavity modes [812], generation of two-mode squeezing useful for amplification of the mechanical motion or the cavity field [13, 14], and the creation of photon–phonon or phonon–phonon entanglement [1518]. Many of these applications rely on the simultaneous fulfilment of two requirements: (i) operating in the resolved sideband regime in which the cavity linewidth is smaller than the mechanical frequency and (ii) having a sufficiently strong coupling between photons and phonons. In systems based on optical Fabry–Pérot resonators (such as membrane-in-the-middle optomechanical devices [19, 20]), these two conditions are not independent; using a short optical cavity (leading to a small mode volume and large coupling strengths) results in a large cavity decay rate such that the resolved sideband regime cannot be reached. The sideband resolution is improved by using a long cavity in which, however, the coupling is reduced owing to the large mode volume. It is therefore desirable to investigate alternative approaches that can either relax the conditions on sideband resolution or improve the coupling strength without increasing the decay rate.

In recent years, hybrid optomechanical systems emerged as an interesting platform for novel optomechanical experiments [21, 22]. In these systems, cavity fields interact with mechanical oscillators and few-level systems, such as single atoms or their ensembles [2328], Bose–Einstein condensates [29, 30], colour centres [3134], or superconducting circuits [3540]. For instance, interaction with an atomic ensemble can lead to backaction evading measurements of mechanical motion [41, 42], generation of entanglement between the ensemble and mechanical oscillator [4345], or cooling of the mechanical motion in the unresolved sideband regime [4648].

The interplay of various types of interactions in hybrid quantum systems can lead to interference effects and novel optomechanical phenomena. Several works have pointed out the role of interference in standard and hybrid optomechanics [46, 47, 4952] and shown it to be decisive in obtaining, for example, novel, efficient forms for optomechanical cooling. A particularly interesting situation arises when a vibrating membrane is doped by an ensemble of two-level emitters as shown schematically in figure 1(a). Such a setup has been investigated for the first time in [53] where a poorly reflecting membrane oscillator was considered. Radiation pressure forces thus played a negligible role but, owing to the presence of the dopant, the oscillator experienced an effective optomechanical interaction with the cavity mode. Such a coupling allows for efficient optomechanical cooling in the unresolved cavity limit, enabled by dressing of the cavity field by the narrow-linewidth emitters. A legitimate question, potentially relevant for a wide range of hybrid optomechanical systems, concerns the interplay between this position-modulated Tavis–Cummings interaction and radiation pressure when the mechanical resonator is partially reflecting and radiation pressure can no longer be neglected.

In this work we theoretically investigate the optomechanical effects arising from these two types of interaction. The presence of the dopant results in a Fano resonance in the cavity noise spectrum which can be used to suppress the Stokes scattering (responsible for heating of the mechanical motion) and enhance the anti-Stokes scattering (cooling), leading to improved cooling performance. Radiation pressure can further boost this effect such that the resulting optomechanical forces lead to stronger optomechanical cooling of the mechanics, as compared to the situations in which either the dopant-induced optomechanical force or radiation pressure acts independently. In particular, we demonstrate that efficient cooling is achievable in situations in which neither dopant-induced nor radiation pressure cooling perform well. We focus on the case of a bad optomechanical cavity—a short cavity containing a movable membrane [54, 55]—in which a large optomechanical coupling can be achieved, but the bare cavity linewidth is too large to resolve the mechanical sidebands. To make the discussion simple we focus on the case of a partially reflecting membrane doped with two-level systems that interact with the cavity field via a Tavis–Cummings interaction. Our results could, however, be amenable to other hybrid mechanical resonators doped with single or multiple two-level emitters (such as diamond cantilevers [32, 56, 57], nanowires [58], optically or electrically trapped nanospheres [59, 60], or photonic crystals [61]) and illustrate how interference effects can be exploited for engineering of efficient optomechanical interactions in hybrid mechanical systems.

## 2. Model

We consider the system depicted in figure 1 where a single cavity mode c interacts with a single vibrational mode of a flexible membrane with an embedded ensemble of two-level quantum emitters. Following [53], we consider the limit of weak excitation of the ensemble, such that its collective spin can be described by the bosonic annihilation operator a (with the commutator $[a,{a}^{\dagger }]=1$). The system then follows the Hamiltonian

The bare Hamiltonian ${H}_{0}/{\hslash }={\omega }_{c}{c}^{\dagger }c+{\omega }_{a}{a}^{\dagger }a+{\omega }_{{\rm{m}}}({q}^{2}+{p}^{2})/2$ describes the free evolution of the cavity field at frequency ωc, the dopant spin at frequency ωa, and the mechanical resonator with displacement q and momentum p (obeying the commutation relation $[q,p]={\rm{i}}$) at frequency ${\omega }_{{\rm{m}}}$. The last term, ${H}_{\mathrm{dr}}/{\hslash }=-{\rm{i}}\eta \,c\exp ({\rm{i}}{\omega }_{{\rm{L}}}t+{\rm{i}}\phi )+{\rm{h}}.{\rm{c}}.$, describes driving of the cavity mode with laser light of frequency ${\omega }_{{\rm{L}}}$, amplitude η, and phase ϕ.

The interaction Hamiltonian describes the interaction of the cavity field with the mechanical oscillator via radiation pressure and with the dopant via a mechanically modulated Tavis–Cummings coupling [53], ${H}_{\mathrm{int}}/{\hslash }={g}_{0}{c}^{\dagger }{cq}+(\lambda +{\mu }_{0}q)({a}^{\dagger }c+{c}^{\dagger }a);$ see figure 1(b). Here, the displacement dependence of the Tavis–Cummings interaction arises from the motion of the membrane which shifts the position of the dopant in the standing wave of the cavity mode; for a membrane placed in the middle between a node and an antinode of the field and dopant in the Lamb–Dicke regime, expansion to the first order in mechanical displacement is sufficient to characterize all dynamical effects [53]. The coupling $\lambda =\sqrt{N}d\sqrt{{\omega }_{c}/(2{\epsilon }_{0}{\hslash }{{lS}}_{\mathrm{eff}})}$ stems from the collective interaction of N emitters (with individual dipole moment d) with the zero-point field amplitude of the cavity (inversely proportional to the square root of the quantization volume ${{lS}}_{\mathrm{eff}}$). We assume a Fabry–Perot type of cavity of length l, finesse ${ \mathcal F }$, mode area ${S}_{\mathrm{eff}}$ and resulting mode linewidth $\kappa =\pi {c}_{0}/l{ \mathcal F }$ (c0 is the speed of light). The dipole moment of a dopant emiter is of course directly related to the spontaneous emission rate $\gamma ={\omega }_{a}^{3}{d}^{2}/(3\pi {\epsilon }_{0}{\hslash }{c}_{0}^{3})$ such that the ratio λ2/γ does not depend on the choice of emitter. We can then estimate that the dopant–cavity cooperativity $C=N{\lambda }^{2}/(\kappa \gamma )=3N{ \mathcal F }{\left({\lambda }_{c}/2\pi \right)}^{2}/{S}_{\mathrm{eff}}$ depends mainly on the cavity design and number of dopant atoms; here, ${\lambda }_{c}=2\pi {c}_{0}/{\omega }_{c}$ is the cavity wavelength. In the remainder of the article, we put ℏ = 1 for simplicity.

### 2.1. Linearized dynamics

We linearize the Hamiltonian (1) using the standard approach outlined in detail in appendix A. We start by formulating and solving the classical equations of motion of the system in the steady state. Provided a single steady state solution exists with solutions $\bar{c}$, $\bar{a}$, $\bar{q}$ (i.e. the system is statically stable), we formulate linearized equations of motion for the quantum fluctuations around this steady state, $c=\bar{c}+\delta c$, $a=\bar{a}+\delta a$, $q=\bar{q}+\delta q$. Depending on the strength of the interactions, these linearized equations might become dynamically unstable; we defer discussion of dynamical stability to section 3.3. Assuming the stability criteria are met, the linearization procedure yields the Hamiltonian $H={H}_{0}+{H}_{\mathrm{int}}$, where

Here, ${{\rm{\Delta }}}_{i}={\omega }_{i}-{\omega }_{{\rm{L}}}$ is the detuning of the respective mode ($i=a,c$) from the laser drive frequency and

We also defined the linearized coupling rates $g={g}_{0}\bar{c}$, $\mu ={\mu }_{0}\bar{c}$. Notice that we have dropped the δ and simply denote the fluctuations by c, a, q for simplicity. A simplified diagram of the interactions in the linearized regime is depicted in figure 1(c).

In this linearized regime, the dynamics of the mechanical oscillator are given by the Langevin equations

where ${\gamma }_{{\rm{m}}}$ is the intrinsic mechanical linewidth and ξ the associated bath operator; it has zero mean and correlation function $\langle \xi (t)\xi (t^{\prime} )\rangle ={\gamma }_{{\rm{m}}}(2\bar{n}+1)\delta (t-t^{\prime} )$ with the average thermal phonon number $\bar{n}$. In addition to the thermal bath ξ, the mechanical resonator is also coupled to an effective bath represented by a zero-average noise term with contributions from the atomic and cavity degrees of freedom

To describe the properties of this extra Langevin noise term, we list the equations of motion for the cavity field and the dopant,

The cavity field decays at a rate κ, is driven by the noise operator ${c}_{\mathrm{in}}$ with zero mean and correlation function $\langle {c}_{\mathrm{in}}(t){c}_{\mathrm{in}}^{\dagger }(t^{\prime} )\rangle =\delta (t-t^{\prime} )$, and its output follows the relation ${c}_{\mathrm{out}}=\sqrt{2\kappa }c-{c}_{\mathrm{in}}$. Analogous relations hold also for the dopant for which the decay rate κ is replaced by γ.

To quantify the effect of the extra Langevin noise term (5) on the dynamics of the mechanical resonator, we follow a perturbative approach [62, 63] in which we ignore the backaction of the mechanical resonator on the field and dopant. To zeroth order in the mechanical displacement q, the cavity field and the dopant ensemble in frequency space can be expressed as

where we introduced the bare and dressed susceptibilities

With these solutions, we can rewrite the Langevin force as

We express the spectrum of the Langevin force as ${S}_{F}(\omega )={S}_{\kappa }(\omega )+{S}_{\gamma }(\omega )$ with

Using the force spectrum, we obtain the cooling rate [62]

### 2.2. Overview of cooling strategies

We can use the noise spectra (10a), (10b) to recover existing approaches to optomechanical cooling. First, the standard sideband cooling strategy [62, 64] corresponds to λ = μ = 0; see figure 2(a). In this case, we get the Lorentzian cavity spectrum ${S}_{\kappa }(\omega )=2{g}^{2}\kappa /[{\kappa }^{2}+{\left(\omega -{{\rm{\Delta }}}_{c}\right)}^{2}]$ while Sγ(ω) = 0. It then follows that the best cooling can be achieved with a sideband resolved system, $\kappa \lt {\omega }_{{\rm{m}}}$, driven on the red sideband, ${{\rm{\Delta }}}_{c}={\omega }_{{\rm{m}}};$ final mechanical occupation smaller than unity requires strong optomechanical cooperativity ${g}^{2}/\kappa \gamma \bar{n}\gt 1$. In the following, we will refer to this strategy as radiation pressure cooling.

In the bad cavity regime, $\kappa \gt {\omega }_{{\rm{m}}}$, radiation pressure cooling cannot reach the quantum ground state of the mechanical resonator. To suppress the unwanted Stokes scattering in this situation, one can use an atomic ensemble placed within the same optical cavity. If the atoms are in the resolved sideband regime, $\gamma \lt {\omega }_{{\rm{m}}}$, they will burn a hole in the cavity spectrum; by choosing a suitable set of detunings Δc, Δa, this spectral hole can overlap with the location of the upper mechanical sideband. This modification results in a reduced density of states around the sideband, leading to reduced Stokes scattering such that the mechanical ground state can be reached [46]. This strategy, which we will call dressed cavity cooling, corresponds to the limit μ = 0 and is shown in figure 2(b).

Finally, the regime with g = 0 (see also figure 2(c)) has been studied in [53]; this situation describes a doped membrane with negligible radiation pressure coupling. Here, the dopant provides both the sideband resolution (when in the regime $\gamma \lt {\omega }_{{\rm{m}}}$) and coupling to the mechanical resonator (via the coupling constant μ). The cavity field (which does not couple to the mechanical motion directly) serves only to enhance the intrinsically weak interaction between the dopant and the mechanical resonator. We name this strategy dopant cooling.

In contrast, we investigate a cooling strategy where all three interactions—radiation pressure coupling at a rate g, Tavis–Cummings interaction at a rate λ, and dopant–mechanical coupling at a rate μ—are present in the system at the same time (see also figure 2(d)). This situation might appear identical to the dopant cooling shown in figure 2(c) but these two schemes differ in the effective optomechanical coupling. While the effective optomechanical coupling in the dopant cooling scheme is $-{\rm{i}}\lambda \mu /(\gamma +{\rm{i}}{{\rm{\Delta }}}_{a})$, it is equal to $\tilde{g}=g-{\rm{i}}\lambda \mu /(\gamma +{\rm{i}}{{\rm{\Delta }}}_{a})$ in our model. This latter form of the coupling leads to detuning-dependent interference between radiation pressure and dopant coupling which can further lower the final occupation. Owing to this effect, we denote this strategy interference cooling.

### 2.3. Fano resonance

The noise spectra (10a), (10b) reveal that interference can play an important role in cooling. For example, the cavity input noise can influence the mechanical motion either directly from the cavity field (dressed by the presence of the dopant), or it can be transferred to the dopant and affect the motion from there. These processes are captured by the first and second term in (10a), respectively; since they both stem from the same reservoir, they have to be added coherently. Different interference conditions exist for the Stokes and anti-Stokes scattering, such that strong asymmetry in the two processes is possible even in the bad cavity regime.

In the following, we will consider cooling in the bad cavity regime, $\kappa \gg {\omega }_{{\rm{m}}}$, and assume that the dopant is relatively strongly coupled to the cavity field such that the Tavis–Cummings interaction is in the regime of strong cooperativity, $C={\lambda }^{2}/\kappa \gamma \gt 1$. From the scaling $C=3N{ \mathcal F }{\left({\lambda }_{c}/2\pi \right)}^{2}/{S}_{\mathrm{eff}}$ derived in section 2, one can estimate that in order to reach this regime, a cavity of mode area around 50 μ× 50 μm would require the product $N{ \mathcal F }$ to achieve values of the order 106. As cavities of finesse around ${10}^{4}\mbox{--}{10}^{5}$ are routinely employed in optomechanical setups, one then requires around 100 dopant emitters to be placed in the mode area of the cavity field, thus at relatively low densities (such that the emitters can be considered independent).

The cavity mode and the dopant form polaritons with energies

We can expect the cooling to be optimal when one of the polariton modes is driven on the lower mechanical sideband, ${\omega }_{+}={\omega }_{{\rm{m}}}$ (or ${\omega }_{-}={\omega }_{{\rm{m}}}$), which is achieved for the cavity detuning

Plugging the optimal detuning (13) into the noise spectra (10a), (10b) and assuming $\kappa \gg {\omega }_{{\rm{m}}}$, we can approximate the noise spectra to leading order in ${\omega }_{{\rm{m}}}/\kappa$ as

The spectra describe the hybridization of the cavity mode with the dopant; the emergent polaritonic state is characterized by linewidth Γ and the state energy quantified by Δ:

Note that, for weak coupling, one reproduces the expected result that the polariton exhibits the bare linewidth of the dopant Γ ≃ γ and is positioned at Δ ≃ Δa. For increasing coupling strength the polariton linewidth and energy acquire contributions from both the dopant and the cavity mode. Furthermore, the spectra (14) are characterized by the amplitudes

The amplitude A(ω) is quadratic in frequency so the cavity noise spectrum Sκ(ω) exhibits a Fano resonance [65]; the atomic noise spectrum Sγ(ω), on the other hand, is Lorentzian. The Fano resonance can be further enhanced by the interference between the radiation pressure and the dopant interaction as we discuss below.

## 3. Interference cooling

### 3.1. Dopant-induced cooling

First, we turn our attention to the Lorentzian noise spectrum of the dopant Sγ(ω). It follows from the theory of sideband cooling [62, 64] that the optimum cooling performance is achieved for ${\rm{\Delta }}={\omega }_{{\rm{m}}}$ and ${\rm{\Gamma }}\lt {\omega }_{{\rm{m}}}$. These conditions can be realized using a good dopant $\gamma \lt {\omega }_{{\rm{m}}}$ with detuning ${{\rm{\Delta }}}_{a}={\omega }_{{\rm{m}}}$. The noise spectrum then simplifies to

while the cavity noise spectrum becomes zero, Sκ(ω) = 0. This result is quite natural, since driving the dopant on the red mechanical sideband results (for polariton driving according to (13)) in an infinite cavity detuning. The cavity is thus strongly off-resonant so it decouples from the dynamics which thus obey the Hamiltonian

One might expect that ground state cooling in this regime is possible provided the system exhibits strong cooperativity, ${\mu }^{2}/\gamma {\gamma }_{{\rm{m}}}\bar{n}\gt 1$. This assertion is true in principle, but such a regime would be extremely difficult to reach in an experiment. Recall that the the coupling rate $\mu ={\mu }_{0}\bar{c}$ is obtained from the three-body interaction ${\mu }_{0}({a}^{\dagger }c+{c}^{\dagger }a)q$ enhanced by a strong intracavity amplitude $\bar{c}$. The three-body coupling strength μ0 is, in turn, a perturbative correction to the Tavis–Cummings interaction in the Lamb–Dicke regime so we have μ0 ≪ λ. Moreover, reaching a large cavity amplitude $\bar{c}$ for an effectively infinite detuning Δc would require effectively infinite driving power.

### 3.2. Cooling via Fano resonance

Analysis of the cavity noise spectrum, (14), is more involved. Owing to the frequency dependence of the amplitude A(ω), the cavity noise spectrum exhibits a Fano resonance, which can be used to modify the Stokes and anti-Stokes scattering rates. While a general analysis of these spectra and optimization of the cooling is, in principle, possible, it does not bring a clear physical insight into the system dynamics. We thus only highlight the main features of this approach and defer more detailed analysis to the next section where we study the noise spectra and final mechanical occupation numerically.

The cooling rate is given by ${S}_{\kappa }({\omega }_{{\rm{m}}})$ whereas heating by ${S}_{\kappa }(-{\omega }_{{\rm{m}}});$ to exploit the Fano resonance for suppressing heating and enhancing cooling, we would therefore like the dip of the Fano resonance to fall within the vicinity of $\omega =-{\omega }_{{\rm{m}}}$ while the peak should be close to $\omega ={\omega }_{{\rm{m}}}$ (see figure 3(b) for an illustration). These requirements already put certain conditions on the detuning and linewidth defined in (15a), (15b). Specifically, we need a detuning with magnitude within the mechanical sidebands, $| {\rm{\Delta }}| \lesssim {\omega }_{{\rm{m}}}$ and a linewidth that is not too large either, ${\rm{\Gamma }}\lesssim {\omega }_{{\rm{m}}}$. At the same time, we must not forget that the dopant noise spectrum (14) also contributes to heating and cooling of the membrane. Ideally, we would thus have positive detuning, Δ > 0, such that ${S}_{\gamma }({\omega }_{{\rm{m}}})\gt {S}_{\gamma }(-{\omega }_{{\rm{m}}})$.

The suppression of Stokes scattering via Fano resonance is not unique to our system. The same principle is also used in dressed cavity and dopant cooling [46, 53]. In these two systems, the cavity field and atoms also form two polariton modes, resulting in cavity noise spectra analogous to (14). With interference cooling, however, there is an additional interference between the two types of interaction—the radiation pressure interaction at a rate g and the dopant–mechanical interaction at a rate μ as exemplified by the curly bracket in (16a). This interference can lead to a further suppression of the Stokes scattering (and enhancement of anti-Stokes scattering) and thus a lower final occupation than in any of the previous cooling schemes.

An intriguing consequence of this interference effect is the possibility of cooling with both cavity and dopant driven on resonance, Δc = Δa = 0. In this case, the cavity and dopant noise spectra are not given by (14) (unless $\lambda ={\omega }_{{\rm{m}}}$) but instead by the expressions

(These expressions can be obtained simply by setting Δa = Δc = 0 in (10a), (10b).) The spectra clearly reveal the importance of interference for cooling on resonance: only when both radiation pressure and dopant interaction are present does the numerator of each of the two spectra contain a term linear in frequency. The spectra thus distinguish between positive and negative frequencies, resulting in a net cooling or heating effect. Specifically, we obtain the cooling rate (recall the definition given in (11))

The denominator is always positive so the membrane is cooled as long as gλμ > 0 (i.e. either none or two of the coupling rates are negative).

### 3.3. Numerical simulations

To check our expectations, we perform numerical simulations of the full linearized dynamics to determine the final mechanical occupation. To this end, we formulate a Lyapunov equation for the covariance matrix of the system. We start by defining the quadrature operators ${X}_{c}=(c+{c}^{\dagger })/\sqrt{2}$, ${Y}_{c}=-{\rm{i}}(c-{c}^{\dagger })/\sqrt{2}$ (and similar for the dopant) with the commutator $[{X}_{i},{Y}_{j}]={\rm{i}}{\delta }_{{ij}}$. Together with the mechanical position and momentum operators, we collect these operators into the vector ${\boldsymbol{r}}={\left({X}_{c},{Y}_{c},{X}_{a},{Y}_{a},q,p\right)}^{T}$ and define the covariance matrix with elements

The steady state covariance matrix ${\boldsymbol{V}}$ is a solution of the Lyapunov equation

with drift and diffusion matrices ${\boldsymbol{A}}$, ${\boldsymbol{N}};$ we present these matrices and discuss the dynamical stability in appendix B. We obtain the mechanical occupation in the steady state from the variance of the mechanical position and momentum,

Note that since the dynamics are linear, the (initially Gaussian) state of the system remains Gaussian throughout the evolution and the covariance matrix is sufficient to fully describe the correlations in the system.

We plot the results of such a simulation in figure 3(a) where we show the final occupation ${n}_{{\rm{f}}}$ as a function of the cavity and dopant detunings. Particularly, driving the upper polariton with energy ω+ on the lower mechanical sideband (shown as the dashed red line in the lower left quadrant) leads to substantive cooling and even makes it possible to reach final occupation ${n}_{{\rm{f}}}\lt 1$. Driving the lower polariton in the same way (upper right quadrant), on the other hand, leads only to moderate cooling or even becomes unstable (when entering the dark blue region).

We further elucidate this difference in figures 3(b)–(e) where we plot the spectra at four different points of the 2D plot. On the lower sideband of the upper polariton (figure 3(b)), the cavity noise spectrum (dashed orange line) exhibits a clear Fano resonance which reaches a minimum around $\omega =-{\omega }_{{\rm{m}}}$ and maximum close to $\omega ={\omega }_{{\rm{m}}};$ the Stokes scattering is thus suppressed while the anti-Stokes scattering is enhanced, which leads to a final occupation ${n}_{{\rm{f}}}\simeq 0.74$. On the lower sideband of the lower polariton (panel (c)), the Fano resonance is still present but not ideally oriented (the minimum is to the right of the maximum) so the final occupation is much higher (${n}_{{\rm{f}}}\simeq 19.4$). A smaller final occupation than on the lower sideband of the lower polariton can, in fact, be achieved also far detuned from the lower sideband of the upper polariton (such as at the point (d) in figure 3, where the final occupation ${n}_{{\rm{f}}}\simeq 10$). Finally, when the Stokes scattering is stronger than the anti-Stokes scattering, the system becomes unstable; see figure 3(e). Together, these results reveal the importance of Fano resonance for efficient cooling: the Fano minimum suppresses the Stokes scattering while the maximum enhances the anti-Stokes scattering. These requirements limit the suitable dopant detuning $| {{\rm{\Delta }}}_{a}| \lesssim {\omega }_{{\rm{m}}}$ (see (15a), (15b)), leading to optimal cooling around the lower sideband of the upper polariton.

We study the final occupation along the lower sideband of the two polariton modes in more detail in figure 4(a). Two observations are crucial here: first, the minimum final occupation reached along the lower sideband of the upper polariton (${n}_{{\rm{f}}}\simeq 0.74$) is very close to the absolute minimum in figure 3 (${n}_{{\rm{f}}}\simeq 0.73$) indicating that the lower sideband of the polariton mode is near-optimal for cooling with moderate cooperativity (we have λ2/κγ = 4). Second, interference cooling (shown as the solid blue line) performs better than any other of the cooling schemes; the best results can otherwise be achieved with dressed cavity cooling, which reaches a final occupation ${n}_{{\rm{f}}}\simeq 1.1$.

We present further comparison of the four cooling schemes in figures 4(b)–(d). There exists a broad range of system parameters—generally in the bad cavity regime—where interference cooling can outperform existing cooling strategies (panels (b), (c)). In these cases, one can reach optimum cooling for blue-detuned cavity drive, Δc < 0, corresponding to rather small dopant detuning (e.g. in panel (c), the optimal dopant detuning ${{\rm{\Delta }}}_{a}\simeq -{\omega }_{{\rm{m}}}$). This observation further confirms our assertion that the Fano resonance in the cavity noise spectrum is responsible for the suppression of Stokes scattering and enhancement of anti-Stokes scattering. We also note that in the good cavity regime (panel (d)), the performance of radiation pressure, dressed cavity, and interference cooling is comparable; admittedly, radiation pressure cooling is, from the experimental point of view, the simplest of these methods to implement.

Finally, we study the final occupation for interference cooling with driving on resonance, Δc = Δa = 0, in figure 5. Remarkably, final occupation ${n}_{{\rm{f}}}\lt 1$ is possible even in the bad cavity regime (panel (a)). In the sideband resolved regime (panel (b)), the final occupation can be lower than in the bad cavity regime, but resonant interference cooling cannot outperform radiation pressure cooling; here, the minimum final occupation is ${n}_{{\rm{f}}}\simeq 0.8$ whereas radiation pressure cooling can reach ${n}_{{\rm{f}}}\simeq 0.14$ with the same sideband resolution. Nevertheless, resonant driving (as used for interference cooling) requires smaller driving power than a sideband drive (necessary for radiation pressure cooling) to achieve the same coupling strength; interference cooling might thus have an important advantage over radiation pressure cooling even in the good cavity regime.

In both regimes, there is an optimal range of coupling rates for which ground state cooling is possible. This effect is a consequence of the interference in the noise spectra (19a), (19b). Take, for instance, the cavity noise spectrum Sκ(ω) (note, however, that a similar argument holds also for the dopant spectrum Sγ(ω)): here, the second term in the numerator, $\propto {\left(g{\gamma }^{2}+\lambda \mu \omega \right)}^{2}$, is responsible for cooling. More specifically, it is the term linear in frequency, $\propto g\lambda \mu \omega$ that gives rise to cooling; the remaining two terms, $\propto {g}^{2},{\lambda }^{2}{\mu }^{2}$ do not affect the cooling rate but still affect the final occupation since they contribute to the backaction that the cavity field and dopant exert on the mechanical oscillator. We therefore need to maximize the interference term relative to the latter two, which gives rise to an optimal range of coupling rates as can be seen in figure 5.

## 4. Summary and outlook

In conclusion, we investigated cooling of a mechanical resonator doped by an ensemble of two-level quantum emitters. The interplay between radiation pressure and mechanically modulated Tavis–Cummings interaction between the cavity field and the dopant gives rise to a Fano resonance in the cavity noise spectrum. This resonance can lead to a suppression of Stokes and enhancement of anti-Stokes scattering, leading to ground state cooling in regimes where none of the effects alone can efficiently cool the motion. An additional signature of the interference between these two types of interaction is the possibility of ground state cooling when the cavity and dopant are driven on resonance. Our results are not limited to the particular architecture considered here; similar results can be expected for any mechanical oscillator with embedded two-level quantum emitters and experiencing a direct radiation pressure force.

This work highlights the importance of interference effects in hybrid optomechanical systems for studying novel phenomena and developing new applications. The interference can also result in a lowered instability threshold, which can have profound implications for the generation of ponderomotive squeezing of light [66] or for observing mechanical limit cycles [67, 68]. Further improvements and new effects may occur when the dopant ensemble is prepared in a super- or subradiant state [69] or with quadratic optomechanical coupling [19, 20].

Looking forward, these devices will enter a new domain once they reach the regime of near-unit reflectivity around the dopant resonance [7074]. Such membranes could then be used as end mirrors in Fabry–Pérot resonators, where their strongly frequency dependent reflectivity can reduce the cavity linewidth [75] and lead to the observation of non-Markovian optomechanical dynamics in the resolved sideband and strong coupling regimes.

## Acknowledgments

We gratefully acknowledge financial support from the Max Planck Society and the Velux Foundations.

## Appendix A.: Linearization of the three-body dynamics and static stability

For completeness, the equations of motion obtained from the full Hamiltonian (1) and the linearization around the semiclassical steady state are detailed here. We start by adding dissipation to the Hamiltonian (1) and obtaining the Langevin equations

where the cavity and dopant dynamics is expressed in the rotating frame with respect to the driving frequency; moreover, we defined ${\eta }_{\phi }=\eta {{\rm{e}}}^{-{\rm{i}}\phi }$.

Next, we separate each operator into its classical amplitude and quantum fluctuations, $o=\bar{o}+\delta o$. The classical amplitudes obey the steady state equations

the solutions are

Introducing $g={g}_{0}\bar{c}$ and $\mu ={\mu }_{0}\bar{c}$, we recast (1.3c) as

the solution of which is the intracavity field amplitude $\bar{c}$, implicitly contained in g and μ.

Without dopant (λ = μ = 0) one retrieves the usual dispersive Kerr bistability equation for the intracavity field

and, in the absence of a dynamical instability, the motion-induced nonlinear phase-shift leads to optical bistability when the Kerr dephasing is of the order of κ. With dopant, however, the bistability threshold can be lowered (or highered) owing to interference between various terms in (1.4). We assume here that the system is stable and a single solution $\bar{c}$ exists.

Linearized fluctuations around the steady state obey the Langevin equations

where, to simplify the notation, we absorbed the term $g\bar{q}$ into Δc, redefined λ to include the term ${\mu }_{0}\bar{q}$, introduced $\tilde{g}=g-{\rm{i}}\lambda \mu /(\gamma +{\rm{i}}{{\rm{\Delta }}}_{a})$, and set the driving phase ϕ such that $\bar{c}\in {\mathbb{R}}$. We can associate the coherent dynamics in these equations with the linearized Hamiltonian given in (2); for simplicity of notation, we drop the δ in the linearized Hamiltonian (2) and the following calculations from the operators.

## Appendix B.: Lyapunov equation and dynamical stability

The drift and diffusion matrices ${\boldsymbol{A}}$, ${\boldsymbol{N}}$ in the Lyapunov equation (22) can be obtained from the Hamiltonian and the jump operators [76]. For the Hamiltonian in (2), the assumed decay of the cavity field and the dopant, and the thermal noise acting on the mechanical resonator, one gets

here, we defined $\eta =\lambda \mu /({\gamma }^{2}+{{\rm{\Delta }}}_{a}^{2})$. The system remains dynamically stable if the real parts of all the eigenvalues of the drift matrix ${\boldsymbol{A}}$ are nonpositive.