Instability of multi-mode systems with quadratic Hamiltonians

We present a novel geometric approach for determining the unique structure of a Hamiltonian and establishing an instability criterion for quantum quadratic systems. Our geometric criterion provides insights into the underlying geometric perspective of instability: A quantum quadratic system is dynamically unstable if and only if its Hamiltonian is hyperbolic. By applying our geometric method, we analyze the stability of two-mode and three-mode optomechanical systems. Remarkably, our approach demonstrates that these systems can be stabilized over a wider range of system parameters compared to the conventional rotating wave approximation (RWA) assumption. Furthermore, we reveal that the systems transit their phases from stable to unstable, when the system parameters cross specific critical boundaries. The results imply the presence of multistability in the optomechanical systems.


I. INTRODUCTION
Quantum information processing and communication (QIPC) has developed rapidly in the past few decades.Optomechanical systems, composite systems of light and mechanical modes interacting by radiation-pressure force, have been proposed for QIPC applications [1][2][3].The theoretical and experimental studies on effects by radiation pressure on large objects were conducted in the context of interferometers [4].The optical bistability in a Fabry-Perot resonator was experimentally demonstrated to vary by radiation pressure [5].Various quantum phenomena effected by the optomechanical interaction have been observed, including squeezed optomechanics [6], cooling of the mechanical mode to its motional ground state [7], generation of optomechanical entanglement [8], optomechanical induced transparency [9], and optomechanical transduction [10].As a prerequisite for QIPC applications, the stability of optomechanical systems has received attention [6][7][8][9][10][11][12][13].
Most studies on the optomechanical systems have focused on stable regions, and the physical mechanism of instability remains unraveled in more general ranges of physical parameters.The stability condition was investigated, when the optical cavity is driven by a coherent field; the strength of driving field determines the effective coupling between the cavity and mechanical modes [2].In particular, it has been studied in the assumption of the rotating wave approximation (RWA), with the redsideband laser (ω L +Ω ≈ ω cav , where ω L , Ω, and ω cav are laser, mechanical, and cavity frequencies, respectively) and/or blue-sideband laser (ω L ≈ ω cav + Ω) [6][7][8][9][10][11][12][13].In RWA, the red-detuned laser stabilizes a two-mode optomechanical system, while the blue-detuned laser renders it unstable [12].For a three-mode optomechanical system, the power of red-detuned laser needs to be stronger than that of blue-detuned laser to stabilize in RWA [13].In this paper we generalize and extend the * hyoung@hanyang.ac.kr stability analysis of optomechanical systems to the full range of detuning frequencies and effective optomechanical couplings, going beyond RWA.We also explore the physical mechanism of instability.
Methods of stability analysis on a quantum system include Routh-Hurwitz method [14,15].This method can be applied even to a system of nonlinear differential equations.It requires, on the other hand, sophisticated tools to determine stability conditions, and hard to unravel the underlying physics on the stability [16].An alternative method can be employed when a system's Hamiltonian is given in a quadratic form with constant coefficients.It has been applied well to a classical system [17].In this work we extend the method to a quantum quadratic system [18] by employing a geometric picture.The geometric picture is motivated by the observation: A singlemode orbit diverges on phase space if it is governed by a hyperbolic equation.We generalize the geometric picture for multi modes and find a geometrically unique structure of Hamiltonian for a time-independent quantum quadratic system.
In this paper, we derive an instability criterion for a quantum quadratic system with constant coefficients by the geometric approach (Sec.II).The criterion reveals the underlying geometric perspective of instability: A time-independent quantum quadratic system is dynamically unstable if and only if its Hamiltonian can transform to be hyperbolic.We then apply the geometric method to the stability analysis of two-mode and three-mode optomechanical systems (Sec.III).Expanding the range of physical parameters beyond RWA, we show that the twomode system can be controlled dynamically stable even in the blue-detuning regime, and the three-mode system can be stabilized irrespective of the relative power of the red-detuned laser to the blue-detuned.We also show that the system transits its phase from stable to unstable, as the system parameters cross the critical boundaries, i.e., the parameter values at which the Hamiltonian changes its geometric type from circular to hyperbolic.Additionally, we show that the Hamiltonian of an N -mode unstable quantum quadratic system is transformed to the hyperbolic Hamiltonian by some unitary operation followed by some symplectic operation in Appendix A, giving the explicit forms for a two-mode (three-mode) optomechanical system in Appendix B (C).

II. HYPERBOLIC HAMILTONIANS AND UNSTABLE QUADRATIC SYSTEMS
We start with the observation that a single-mode orbit diverges on phase space of position x and momentum p if it is governed by a hyperbolic equation, i.e., αp 2 − βx 2 = const with α > 0 and β ≥ 0 [19].If β < 0, the orbit becomes elliptic and is bounded in a finite region on phase space.From this observation, we employ a geometric approach of orbits on phase space with Hamiltonian diagonalized under symplectic transformations, so that we analyze the instability of a quadratic system.Here, the diagonalization means that Hamiltonian is given by squares of canonical variables with constant coefficients, or equivalently, its equation-of-motion matrix is diagonalized.
To introduce the principal idea for a quantum system, we consider the simplest case of a single-mode unstable system, whose Hamiltonian is given as where α, β are the positive real coefficients, p and x the momentum and position operators, respectively.The observable operators satisfy the canonical commutation relations [x, p] = i in unit of ℏ = 1.The solution to the Heisenberg equation, governed by the Hamiltonian in Eq. ( 1), is given by x(t) = x(0) cosh (2t αβ) + p(0) sinh (2t αβ), (2) p(t) = p(0) cosh (2t αβ) + x(0) sinh (2t αβ), (3) that grow indefinitely as a function of time.In other words, the orbit diverges on the phase space.The divergence (or the instability) originates from the hyperbolic structure of Hamiltonian in Eq. (1).The Hamiltonian in the hyperbolic form of Eq. ( 1) is said hyperbolic.That the Hamiltonian is hyperbolic is a sufficient condition for the system to be unstable.
We prove that it is also a necessary condition when the Hamiltonian is quadratic, in other words, an unstable single-mode system is governed by a hyperbolic Hamiltonian.The most general form of a single-mode quadratic Hamiltonian is given by where ξ = (x, p) T , and α 1 , β 1 and γ 1 are real numbers.On one hand, the stability analysis method in Ref. [17] shows that the system is unstable if and only if On the other hand, we apply some symplectic transformation S to diagonalize V in Eq. ( 4), as in Ref. [20], into a geometric form of Here, Ξ = ( X, P ) T are the new canonical variables transformed from the original ξ = (x, p) T by the symplectic matrix where tan 2θ = 2γ 1 /(α 1 − β 1 ).The geometric form of Hamiltonian Ĥg in Eq. ( 6) is hyperbolic with . This condition coincides with the one in Eq. (5) that the system is unstable.It is seen that the hyperbolic Hamiltonian is the necessary condition for the system to be unstable, as well as sufficient.A special case is considered that α 1 β 1 = γ 2  1 , for which β ′ = 0 and α ′ = α 1 + β 1 .In the case, the system is 'free' with Ĥg = α ′ P 2 , which we also call 'lineal' in the geometric perspective.We say the lineal (or free) Hamiltonian belongs to the hyperbolic, for a sake of simplicity, as its orbit also is unbounded on the phase space.It is remarkable that the hyperbolic Hamiltonian is the necessary and sufficient condition for the instability of a single-mode quadratic system.
We generalize the equivalence between the hyperbolic Hamiltonian and the instability, for N -mode quadratic systems.Let us consider Hamiltonian given in general by where α jk , β jk , γ jk are time-independent real coefficients with α jk = α kj , β jk = β kj , pj and xj are of mode j = 1, ..., N , and ξ = (x 1 , ..., xN , p1 , ..., pN ) T .To this end, we transform V in Eq. ( 8) by some symplectic matrix S and take an interaction picture by some unitary transformation ÛI (t), so as to obtain a geometric form of Hamiltonian ĤG .By the geometric Hamiltonian we investigate the instability of N -mode quadratic system as testing whether it contains any modes of local hyperbolic Hamiltonians.Quadratic Hamiltonians in the form of Eq. ( 8) can be classified into 6 types of Jordan normal forms with respect to eigenvalues of J V (see Append.A and Refs.[18,21]), where J is a skew symmetric matrix for N modes with elements J jk = −i[ ξj , ξk ].We show in Appendix A that, for all the types, Hamiltonian (8) is rewritten by where ĤG is a geometric Hamiltonian decomposed into a sum of modal geometric Hamiltonians, where Ĥk is the geometric Hamiltonian of mode k, α ′ k and β ′ k are its real coefficients, and new canonical variables Pk and Xk result from pk and xk by the symplectic transformation S. The form of interaction Hamiltonian ĤI varies, depending on the type, while ĤI always commutes with ĤG , i.e., [ ĤI , ĤG ] = 0, as in Appendix A. The commutation leads us to take the interaction picture by unitary transformation ÛI (t) = exp(−it ĤI ), so that the Hamiltonian in the picture is given by It is worth noting that the Hamiltonian ĤG (t) = ĤG in the interaction picture, i.e., independent of time, thanks to the commutation of [ ĤI , ĤG ] = 0. Thus, the general quadratic system is governed in the interaction picture by the geometric Hamiltonian ĤG in Eq. ( 10), consisting of independent modes, regardless of its instability.
Looking further into the geometric structure of Hamiltonian (10), we see that ĤG contains at least one mode of Ĥk hyperbolic when the system is dynamically unstable (see Appendix A).For a sake of simplicity, we say a multi-mode Hamiltonian is hyperbolic when it contains at least one mode of hyperbolic Hamiltonian.
The equivalence between the instability and the hyperbolic form of Hamiltonian induces an instability criterion: A (time-independent) quantum quadratic system is dynamically unstable if and only if its (transformed) Hamiltonian is hyperbolic, which is one of our main results.This criterion allows us to analyze the instability in terms of the geometric Hamiltonian and also provides us the physical insight for stabilizing quantum systems.

III. INSTABILITY OF OPTOMECHANICAL SYSTEMS
Our geometric method is applied to dynamical stability of an optomechanical system.In one case of two modes (Sec.III A), a mechanical oscillator of one mode is a mirror to build an optical cavity of the other mode, as in Fig. 1.In the other case (Sec.III B), one mode is a middle mirror separating two optical cavities of the other two modes, respectively, as in Fig. 4. The optomechanical system has been studied most at sideband driving frequencies, where it is stabilized by the red-detuned pumping laser, whereas it becomes unstable by the bluedetuned laser [12].These opposite effects of the reddetuned and blue-detuned lasers can cooperate as they simultaneously affect a three-mode optomechanical system.For a three-mode optomechanical system, thus, one may require that the power of the red-detuned laser is stronger than that of the blue-detuned laser [13].We show that these constraints are relaxed beyond the sideband interaction limit.We consider an optical Fabry-Perot cavity consisting of one mirror firmly fixed and the other movable, as depicted in Fig. 1.The movable mirror is modeled as a quantum harmonic oscillator b with annihilation operator b and frequency Ω.As it is affected by the radiation pressure of cavity, the mechanical oscillator b is coupled to the optical cavity a with annihilation operator â and frequency ω cav .The cavity is pumped by a laser field with strength κ in and frequency ω L .The Hamiltonian in the rotating frame of the laser frequency is given [2] by where ∆ ′ = ω cav − ω L is the detuning between cavity mode and pumping laser, and κ 0 is the optomechanical coupling constant.We "linearize" Hamiltonian in Eq. ( 12) by assuming the limit of strong pumping, |α s | 2 ≫ ⟨δâ † δâ⟩, where α s is the steady-state amplitude and the deviation δâ = â − α s .Similarly, we consider the deviation δ b = b − β s from the steady-state amplitude β s for the mechanical mode.The approximated Hamiltonian up to the second order of the deviations is given in a quadratic form of two modes by Here the amplitudes of stead state are given by α s = κ in /∆ and β s = κ 0 |α s | 2 /Ω [2].The vacuum states of the deviation modes δa and δb are given in terms of the original modes a and b by TABLE I. Geometric kinds of modal Hamiltonians Ĥ1,2 and stability for two-mode optomechanical system with respect to critical parameters KR = Ω|∆|/4 and KB = (∆ 2 − Ω 2 ) 2 /16Ω|∆|.Cases (a)-(c) are in the red-detuning regime (detuning ∆ > 0), cases (e)-(g) in the blue-detuning regime (∆ < 0), and the other case (d) of resonance (∆ = 0).A modal Hamiltonian Ĥk is of a geometric kind either circular, hyperbolic, or lineal (free), when it is in a form of Ĥcircular = λ( P 2 + X2 ), Ĥhyperbolic = λ( P 2 − X2 ), or Ĥlineal = λ P 2 for some real number λ, whose explicit expression is given in Appendix B.
where |α s ⟩ and |β s ⟩ are coherent states of modes a and b.
The linearized Hamiltonian Ĥcm-lin derived above governs the quantum dynamics of the system near a given steady-state point {α s , β s }.The analysis of stability against small fluctuations near the steady state is based on the equations of motion for the deviation modes (associated with Ĥcm-lin ) in the presence of noises [6][7][8][9][10][11][12][13].The requirement for stability can then be derived by applying, e.g., the Routh-Hurwitz criterion [16].However, the analytic expressions are quite cumbersome and they can just be practical in numerical works, in finding the threshold values for a given set of system parameters.In this work, adopting the geometric approach, we explore the mechanism of instability that arises from the optomechanical interaction, neglecting the noise effects, and we analyze the quantum dynamics of fluctuations from the linearized Hamiltonian Ĥcm-lin .
To apply our geometric method of instability, we rewrite Hamiltonian (13) in terms of quadratures, where x1 = (δâ , and We then find the geometric Hamiltonian ĤG from the original Hamiltonian Ĥcm-lin in Eq. ( 15), as in Sec.II for N = 2 (see Appendix B).
Table I summaries 7 cases of geometric Hamiltonians ĤG and their stabilities in terms of parameters ∆, K R = Ω|∆|/4, and , where three cases (a)-(c) are in red-detuning regime with detuning ∆ > 0, other three cases (e)-(g) are in blue-detuning regime with ∆ < 0, and the other case (d) is of resonance ∆ = 0.Here each modal Hamiltonian Ĥk is said either circular, hyperbolic, or lineal (or free), when it is in a form of Ĥcircular = λ( P 2 + X2 ), Ĥhyperbolic = λ( P 2 − X2 ), or Ĥlineal = λ P 2 for some real number λ, whose explicit expression is given in Appendix B. The system is stable in case (a) or (e) as its geometric Hamiltonian ĤG consists of two circular modal Hamiltonians, whereas the other cases are unstable as ĤG contains at least one hyperbolic (or lineal) modal Hamiltonian(s).The stability conditions are given in case (a) by ∆ > 0 and Ω∆ − 4|κ| 2 > 0, and in case (e) by ∆ < 0 and We prove the inequalities ( 16) hold in both cases of (a) and (e): For ∆ > 0, the first inequality is trivial and the second inequality holds by the stability conditions in case (a).For ∆ < 0, the second inequality is trivial and the first inequality holds by the stability conditions in case (e).The stability conditions in Eq. ( 16) coincide with those in Ref. [17].
To explain the origin of hyperbolic Hamiltonian, we transform the linearized Hamiltonian (13) to, in the interaction picture, where and The beam-splitter term Ĥbs , which is resonant at ∆ = Ω, describes the exchange of excitation between the optical and mechanical modes, where excitation is created in one mode while being destroyed in the other.On the other hand, the two-mode squeezing term Ĥsq , which is resonant at ∆ = −Ω, represents the down-conversion process that generates or destroys excitations in both modes simultaneously.While the beam-splitter term Ĥbs can cause an exchange of energy between the modes, the twomode squeezing term Ĥsq can lead to an unbounded increase in energy for both modes, potentially triggering a dynamical instability in the system.Indeed, using the geometric approach, we find that the geometric structure of Ĥbs is circular for all values of ∆, Ω, and κ, indicating that the system governed by Ĥbs is always stable.Whereas, the geometric structure of Ĥsq varies with the parameters, appearing circular when 2|κ| < |∆ + Ω|, hyperbolic (lineal) when 2|κ| > |∆ + Ω| (2|κ| = |∆ + Ω|).That the hyperbolic Hamiltonian governs the system and causes the instability when 2|κ| > |∆ + Ω|, is in line with our earlier assumption about the effect of the squeezing interaction Ĥsq on the energy of the system at resonance ∆ = −Ω.This fact suggests that the squeezing interaction is the origin of the hyperbolic Hamiltonian.As a result, the system is dynamically unstable in a parameter region where the squeezing interaction contributes dominantly.We show in the following that this provides an intuitive interpretation for all cases in Table I.
For ∆ > 0, the counter-rotating (squeezing) term Ĥsq can be neglected by RWA under the condition of weak coupling (|κ| ≪ Ω, ∆) as |∆ − Ω| ≪ |∆ + Ω|, so that the system is stable [case (a)].When the coupling |κ| increases and becomes comparable to Ω or/and ∆, the effect of Ĥsq becomes significant, leading to instability in the system [cases (b) and (c)].For ∆ < 0, we focus on the weak coupling (|κ| ≪ Ω, |∆|) even though our method also works in the strong coupling.In the week coupling, H sq is dominant rather than H bs , as |∆+Ω| = ||∆|−Ω| ≪ ||∆| + Ω| = |∆ − Ω|.Now we take two extremely cases: near resonance of ||∆| − Ω| ≪ |κ| and far-off resonance of ||∆|−Ω| ≫ |κ|.In the near resonance of ||∆|−Ω| ≪ |κ|, it is clear that H sq dominates over H bs , so that the system is unstable [cases (f) and (g)].On the other hand, in the far-off resonance of ||∆| − Ω| ≫ |κ|, both of H sq and H bs are small and it is necessary to employ the second-order perturbations [22].Then, the effective Hamiltonian for the dispersive far-off resonant interactions is given by When turning back to Schrodinger picture, the effective Hamiltonian is transformed to ) Thus, in the far-off resonance, the effective Hamiltonian is that of free harmonic oscillators, so that the system is stable [case (e)] .
We discuss two special situations of the red and blue sidebands by that the cavities are driven, included in Table I.While the former belongs to case (a), the latter is part of case (g).Therefore, the known assumption that the system is stabilized by the red-detuned pumping laser while becoming unstable by the blue-detuned laser, is no longer valid in general.More precisely, the squeezing term of Hamiltonian triggers the instability in optomechanical systems.The effect is not solely determined by the type of detuning alone: The system can become unstable in the red-detuning regime [as shown in cases (b) and (c) and illustrated in Fig. 2(a)], whereas it can be controlled stable in the blue-detuning regime [case (e), Fig. 2(b)].Fig. 2(a) shows that the average numbers diverge of the cavity and mechanical modes when the cavity is driven by a red-detuned laser with ∆ = 1.5Ω > 0. In contrast, Fig. 2(b) demonstrates that the average numbers remain bounded in the blue-detuning regime with ∆ = −1.5Ω< 0. In addition, we present the diagrams of stability.Fig. 3(i) displays the stability diagram in terms of dimensionless parameters ∆ := ∆/Ω and |κ| := |κ|/Ω.This diagram encompasses all the cases described in Table I.Particularly noteworthy are the critical boundaries representing cases (b), (d), and (f).These boundaries consist of the critical points {∆ > 0, |κ| = K R }, {∆ = 0, κ ̸ = 0}, and {∆ < 0, |κ| = K B } for a given Ω.At the critical points the Hamiltonian changes its geometric type from circular to hyperbolic.In other words, the system transits its phase from stable to unstable as the system parameters cross the critical boundaries.Consequently, the system can be controlled stable in both the red-detuning and blue-detuning regimes by adjusting the strength and frequency of the pumping field.We now focus our attention on the stable areas.In Fig. 3(ii), we depict a stability diagram illustrating cases (a) and (e) in terms of ∆′ := ∆ ′ /Ω and |κ in | := |κ 0 κ in |/Ω 2 .The blue and yellow areas in Fig. 3(ii) correspond to two distinct stable steady states of the system, respectively, while the dark green area represents their overlap.These two stable steady states conform to the nonlinearity of the optomechanical interaction, where the system exhibits three possible steady states: two stable and one unstable [2].Furthermore, both steady states found stable in the dark green overlapping region demonstrates the multi-stability characteristic of an optomechanical system [24].
It is remarkable that we analyze the stability criteria of a two-mode optomechanical system by its geometric Hamiltonian in a wide range of system parameters.The extension beyond the limitation of ∆ = ±Ω enables us to explore the general stability condition, also to understand more insight into the mechanism causing the instability of the system.An unstable two-mode optomechanical system is governed by the hyperbolic Hamiltonian, which originates from the two-mode squeezing interaction term.Although focusing on the dynamics in the linearization approximation, the resulting geometric Hamiltonian provides an excellent background for a more general analysis, including noise and nonlinear coupling terms [25,26].

B. Three modes
We consider a three-mode optomechanical system consisting of two fixed mirrors and one movable mirror in between them, where the movable mirror is one mode of mechanical oscillator and two modes of optical cavities formed by the movable mirror and the two fixed mirrors, respectively, as in Fig. 4. The cavities are driven by two pump lasers, respectively.The Hamiltonian of system Ĥcmc is linearized, as done in Sec.III A, where δâ j (δ b) is the annihilation operator of deviation cavity mode j (deviation mechanical mode), κ j the effective optomechanical coupling constant between cavity mode j and the mechanical mode, and ∆ j the Lambshifted detuning between cavity mode j and input laser j.
FIG. 4. Schematic of a three-mode optomachanical system.
The stability condition and geometric kinds of modal Hamiltonian Ĥ1,2,3 for a general three-mode optomechanical system is shown in Appendix C. Here, we focus on a simple case of ∆ 1 = s∆ 2 = ∆, where the sign s = ± ('+' means that both cavity modes are driven by either blue-detuned or red-detuned lasers, whereas '−' indicates that one cavity mode is driven by a red-detuned laser while the other is driven by a blue-detuned laser).The choice of ∆ 1 = ±∆ 2 enables us to achieve a simple explicit expression for stability condition.Even so, it is general enough to go beyond RWA for any chosen value of ∆ (rather than at the sideband interaction limit |∆| = Ω).
By symplectic transformation S c (shown in Table II), Hamiltonian Ĥcmc-lin ( 22) is transformed to (except for the case s = −1 and Here, the term Ĥ Â1-δ b describes a subsystem of two modes, Â1 and δ b, coupling each other by the linearized optomechanical interaction where Because the harmonic oscillator mode Â2 is stable and completely decouples from modes Â1 and δ b, the analysis of stability relies on the two-mode optomechanical Hamiltonian Ĥ Â1-δ b, enabling us to apply the results obtained in Sec.III A. Accordingly, the stability condition for the system is which is the same as the condition in Eq. ( 16) with ∆ and |κ| replaced by ϵ∆ and κ s , respectively.Geometric Hamiltonian of the sub-system including modes Â1 and δ b falls into one of the cases listed in Table I conditional on the values of s, ϵ, ∆, and κ s , as shown in Table III.Mapping between the system parameters and cases (a)-(g) tells us specific kinds of modal geometric Hamiltonian for a given set of parameters.For instance, if ϵ∆ < 0 and κ s > K B , the system is unstable in the manner of the case (g), where the geometric Hamiltonian of the system composing of two hyperbolic modal Hamiltonians associated with modes Â1 and δ b beside a circular Hamiltonian of mode Â2 .
Similarly to the two-mode system, the system can become unstable even when both cavities are driven by red-detuned lasers (s = +, ϵ∆ > 0, κ s > K R ), while it can be stable when both cavities are driven by bluedetuned lasers (s = +, ϵ∆ < 0, κ s < K B ). Besides, the system can be stabilized when s = − and ϵ∆ < 0, which corresponds to one cavity driven by a red-detuned laser while the other is driven by a blue-detuned one and |κ red | < |κ blue |.Here, κ red (blue) represents the interaction coupling strength between the cavity mode driven by a red-detuned (blue-detuned) laser and the mechanical mode.This fact implies that the power of the reddetuned laser stronger than that of the blue-detuned one is no longer a necessary condition for stability as the system goes beyond RWA.To illustrate these interesting cases, Fig. 5 demonstrates the time evolution of average numbers for deviation modes δâ 1 , δâ 2 , and δ b of a three-mode system with ∆ 1 = ±∆ 2 .
In addition, the transition of stability from stable to unstable as the system parameters cross the critical boundaries is observed in a three-mode optomechanical system.For ∆ 1 = ±∆ 2 , the critical points are the same as ones of a two-mode system with ∆ and |κ| replaced by ϵ∆ and κ s , respectively.For a general system with ∆ 1 and ∆ 2 arbitrary, the stability phase transition is shown by stability diagrams in Fig. 6, where the critical boundaries between stable and unstable areas appears in all three situations: (a) both cavities are red-detuned, (b) both cavities are blue-detuned, and (c) one cavity is red-detuned while the other is blue-detuned.

IV. REMARKS
We present a novel geometric approach for analyzing the stability of a quantum quadratic system, which serves as a fundamental component in stability theory.Our investigation reveals that a time-independent quantum quadratic system is dynamically unstable if and only if its  (diagonalized) Hamiltonian is geometrically hyperbolic.Applying this geometric method to optomechanical systems, we derive comprehensive stability criteria that remain valid across the entire range of system parameters for both two-mode and three-mode configurations.Our findings demonstrate that the system transits its phase from stable to unstable as the system parameters cross the critical boundaries.As a result, the system can be controlled stable in all regimes, surpassing the previous understanding that the system is stabilized by the reddetuning laser for the two-mode case or when the power of the red-detuned laser is stronger than that of the bluedetuned laser for the three-mode case.In addition, we explore the mechanism of the instability in optomechanical systems.An unstable optomechanical system is governed by the hyperbolic Hamiltonian, which originates from the two-mode squeezing interaction term.
condition is satisfied if and only if the equation-of-motion matrix A in Eq. (A2) is diagonalizable and has only purely imaginary eigenvalues [17], i.e., all eigenvalues of A belong to type III with D = 1, which is the only case that the geometric Hamiltonian contains no hyperbolic (nor lineal) modal Hamiltonian(s).Thus, the system is dynamically unstable if and only if its geometric Hamiltonian contains at least one hyperbolic modal Hamiltonian.
Appendix B: Geometric Hamiltonian of a two-mode optomechanical system Here we apply the general results obtained in Appendix A to the two-mode optomechanical system described by Hamiltonian Ĥcm-lin Eq. ( 15) in order to derive the geometric Hamiltonian listed in Table I.

A
FIG.1.Schematic of a two-mode optomachanical system.

TABLE III .
Geometric Hamiltonian and stability of the subsystem governed by Hamiltonian Ĥ Â1 -δ b in comparison with the cases from (a) to (g) described in TableI.

TABLE IV .
Geometric kinds of modal Hamiltonians Ĥ1,2,3 and stability for a three-mode optomechanical system.C: circular, H: hyperbolic (including lineal and zero modes).