ournal of C osmology and A stroparticle hysics Stabilization of linear higher derivative gravity with constraints

. We show that the instabilities of higher derivative gravity models with quadratic curvature invariant αR 2 + βR µν R µν can be removed by judicious addition of constraints at the quadratic level of metric ﬂuctuations around Minkowski/de Sitter background. With a suitable parameter choice, we ﬁnd that the instabilities of helicity-0, 1, 2 modes can be removed while reducing the dimensionality of the original phase space. To retain the renormalization properties of higher derivative gravity, Lorentz symmetry in the constrained theory is explicitly broken.


Introduction
It is well known that non-degenerate higher derivative theories suffer from Ostrogradski's instability [1][2][3][4][5]. For example, consider an action with a quadratic 2nd order time derivative term The equation of motion is 4th order, and hence its phase space is 4 dimensional. We can define the two canonical coordinates and their conjugate momenta to be (Q 1 , P 1 ), (Q 2 , P 2 ), and the Hamiltonian is hence while P 1 only appears linearly in the Hamiltonian as P 1 Q 2 . The linearity of the P 1 in this term renders the Hamiltonian unbounded from below and the theory is thus unstable. Because

JCAP05(2014)010
of this undesirable property, non-degenerate higher derivative theories are often viewed as taboo and avoided in the literature. There are several classes of higher derivative theories in the market, which evade this instability. A higher derivative theory may be degenerate, which means that the theory is constrained. For example, in f (R) gravity [6][7][8], the naive unstable degree of freedom is rendered harmless by a gauge constraint. Furthermore, some theories are secretly 2nd order despite the appearance of higher derivative terms in the action due to a clever cancellation of the higher derivative terms in the equation of motion -as seen in the Galileon theory [9][10][11]. 1 On the other hand, generic non-degenerate higher derivative theories are inevitably unstable. Theories with curvature invariants such as R µν R µν , R µνσρ R µνσρ [14][15][16][17][18][19][20][21][22][23], or the Weyl invariant C µνσρ C µνσρ [24,25] 2 suffer from the sickness of Ostrogradski's instability.
One way to deal with the instability is to impose boundary conditions in such a way that the unstable modes vanish. For example, in [24,27] the modes with the wrong sign of the kinetic terms are "turned off" by imposing suitable boundary conditions. However, this is only valid at the quadratic level. In the presence of higher order interaction terms beyond the quadratic power of the field, the vacuum states will rapidly decay (even classically) into states with positive energy modes and negative energy modes by the entropic argument [4,5,28,29]. The "removed" instability is thus revived. 3 We will consider the following action first investigated by Stelle [14] This action with mass dimension −2 parameters α and β in general contains eight degrees of freedom [31], two of them corresponding to the massless graviton in general relativity, five corresponding to the massive graviton, and the last one is a massive scalar. Among them, the helicity-2 sector is a non-degenerate higher derivative theory and thus suffers from Ostrogradski's instability. Nevertheless, this action is interesting as it is power-counting renormalizable [14] the presence of higher derivative terms in the action means that there exist higher spatial derivatives in the propagator of the graviton modes. These spatial derivatives suppress the UV divergences in the loops, rendering the theory naively renormalizable. The price we pay for this is the presence of the higher time derivative terms which leads to Ostrogradski's instability.
One way to take advantage of this insight is to impose different scaling dimensions to the time and space coordinates -a stratagem utilized by Hoȓava [32][33][34][35][36][37]. The low energy limit of this theory is then a generic 1st order time derivative graviton action with higher order Lorentz violating spatial derivative terms, which is both stable and power-counting renormalizable. 1 There are some more exceptions, the higher detivative theories can be made degenerate by interacting with extra fields [12], or the theories with infinite order time derivatives (i.e. the nonlocal theory) are free of Ostrogradski's instability [13]. 2 In 4D, the Weyl invariant CµνσρC µνσρ can be written as 1 2 (Rµν R µν − 1 3 R 2 ) because the Gauss-Bonnet term √ −g(RµνσρR µνσρ −4Rµν R µν +R 2 ) is a total divergence if the coupling is fixed by demanding a condition on the asymptotic curvature [26]. The boundary term does not contribute to the classical equations of motion. 3 However, it is claimed that with special initial conditions the self-interacting higher derivative theory (Pais-Uhlenbeck oscillator) is stable [30]. 4 Here we have turned on the bare cosmological constant since the theory admits constant curvature background solution with Rµν = Λgµν . In this paper, we pursue a different track. We ask whether we can selectively remove the linear instability by imposing constraints on the theory. This idea is motivated by our recent proof [5] that the linearly unstable phase space can be excised from the theory by a judicious choice of additional constraints (i.e. the final dimensionality of the phase space will be smaller). We will show that, at least in the linear theory, we can add by hand to the theory additional constraint terms which will render the theory stable, while simultaneously preserving the improved renormalizable features of it. Roughly speaking, we add a constraint where the higher time-like derivative terms in the equation of motion is constrained to some lower time-like derivative or higher order spatial derivative term, i.e. (1.4) We will show that the final form of this constrained theory is, at least linearly, that of a second order equation of motion of higher order spatial derivatives very similar in spirit to the Hoȓava model, at the price of giving up Lorentz invariance. Such artificial addition of constraints changes the the dynamics and thus the general theory. However, as we have simply worked in linear theory, we do not know what is the non-linear completion of the theory. A possibility which we plan to explore in a future work is that the full non-linear theory has no such technical issue or at least they appear in a more natural way.
Our strategy is as follows. In section 2 we show how to perturb the action up to second order in metric perturbation in general background, which will be used in Minkowski/de Sitter backgrounds. In section 3 we obtain the action quadratic in the metric fluctuation by parameterizing the metric fluctuation in Minkowski background. Since up to quadratic order, the action can be separated into helicity-0, 1, 2 sectors, we demonstrate how the instabilities appear in each sector. In section 4, we show that, how the helicity-0, 1, 2 instabilities can be rendered stable by introducing suitable constraints. In section 5, 6, we study the behavior and how to remove the instabilities in de Sitter background. We hasten to add that we have chosen different constraints for the different background in this work -while it is possible to stabilize the theory with the same constraints, in the helicity-0 mode in the de Sitter background we run into the difficulty that the resultant action is non-local. This non-locality could be a pathology that arises because we are dealing with linear theory and may be resolved when a full non-linear completion is obtained -however we will postpone its resolution to future work. We conclude and conjecture the possible way to make the procedure viable in the full non-linear theory in section 7.
Because of the nonlocal terms in the helicity-0 sector of the unconstrained theory, we choose a different pair of fields which do not appear in the Minkowski case. Although this choice might make the stabilization procedure hard to be applied to the full non-linear theory, one should notice that we can still remove the instabilities even the nonlocal terms appear and it is totally possible that there is no nonlocal terms in the full unconstrained non-linear thoery.

Higher derivative gravity: quadratic action
In order to study how do the instabilities appear in action (1.3) at the quadratic order in the metric fluctuation, we will need to expand every curvature invariant up to second order in the metric perturbation h µν , which is defined by

JCAP05(2014)010
whereḡ µν at this stage can be general background metric and h µν ḡ µν [38]. The inverse metric up to second order in h can be written as Assuming a constant curvature background of either Minkowski (Λ = 0), de Sitter (Λ > 0), or Anti-de Sitter (Λ < 0), we compute the second order action where is d'Alembert operator and the linearized Ricci tensor, Ricci scalar, and Einstein tensor are defined by 5 Note that the indices are raised and lowered by background metricḡ µν .

Quadratic action around Minkowski background
In this section we want to study how the instabilities appear in the action at the quadratic level of perturbation around Minkowski background Λ = 0. We parameterize the metric fluctuation by where E ij is symmetric, traceless tensor and the index i, j are raised and lowered by δ ij . We can further decompose B i and E ij into helicity-0, 1, 2 modes, where B and B T i are longitudinal and transverse parts of vector B i , E T i is transverse, and E TT ij is symmetric, trace-free and transverse, and the angled bracket indices component is trace-free. By this decomposition, we can separate the action into helicity-0, 1, 2 sectors, since at the quadratic level there is no mixing between different helicities. 5 see, for example, [39].

Helicity-2 sector
The second order action of helicity-2 modes is which describes two massless helicity-2 degrees of freedom originating from the massless graviton and two massive helicity-2 degrees of freedom coming from the quadratic invariant term βR µν R µν . Since there is no first class (i.e. gauge) constraint in the helicity-2 modes, there are four helicity-2 degrees of freedom in the theory. Notice that only the β term enters in this expression.
Ostrogradski's choice of canonical coordinates is the pair of canonical variables (E ij , π ij ) and (q ij , p ij ) defined by One might notice that in Ostrogradski's formalism the two canonical variables E ij , q ij have different dimensionalities, the field E ij is dimensionless while q ij has mass dimension 1 and thus the dimension of canonical momenta are different. The dimensionality is not particularly important -in principle one can rescale q ij = M −1 PĖ ij to make the two canonical variable at the same footing.
Using the Legendre transform, we construct the Hamiltonian as usual It is easy to check that the Hamiltonian (3.7) generates the equations of motion for the 4 canonical variables via the Poisson Bracket d(·)/dt = [·, H]. The important point here is that the Hamiltonian is linearly dependent on π ij in the second term and hence the Hamiltonian is unbounded from below -the π ij q ij term can be arbitrarily negative when q ij > 0, π ij → −∞ or vice versa. This instability is often called a "ghost", i.e. a dynamical degree of freedom with the wrong sign kinetic term. To see this, we can explicitly diagonalize the Hamiltonian by the following canonical transformation where the (ψ, p ψ ) pair is ghostlike. In the classical theory, the unboundedness of the Hamiltonian leads to instabilities as the phase space for negative energy higher frequency modes become unbounded below. 6 In the quantum theory, while this instability does not prevent us from identifying a vacuum state and then constructing the Fock space of many particle states, imposition of positivity in the energy of all particle states will lead to some states possessing negative norms, i.e. ghosts. One can further excise these unphysical negative norm states from the Fock space, but this generically leads to violations of unitarity. For a review of the quantization issues with such theories, see appendix A.

Helicity-1 sector
The second order action of helicity-1 modes can be written by the gauge invariant variable The action describes a vector with mass m −2 = −β and the sign of β also decides the overall sign of the action, i.e. if β < 0, the helicity-1 modes are ghostlike. The Euler-Lagrange equation of action (3.10) is which can be solved by Fourier transform, and the solutions are harmonic oscillators with frequency w 2 p = p 2 − 1 β . The canonical momentum conjugate to v i is as usual defined by and since we use the gauge invariant variable to write the action, there is no constraint in helicity-1 sector and the Hamiltonian is If we choose β > 0, then m 2 < 0, which means the theory is tachyonic. On the other hand, if we choose β < 0 in eq. (3.13), the Hamiltonian will be negative definite and thus ghostlike.
6 This is to be contrasted with tachyonic instability -which is the "exponential blowing up" of each individual mode. 7 One should not be unduly worried by the appearance of the non-local square root of Laplacian operator. Recall that the Laplacian operator −∇ 2 has zero or positive eigenvalues λ k , e.g.
k φ k (as long as both u and φ k vanish at the boundary), i.e. u = k c k φ k .

Helicity-0 sector
The second order action for helicity-0 modes is more complicated. With the help of two gauge invariant variables the action can be written as There are two scalar functions in the action, and because of the second order time derivatives on Ψ, there are three naive degrees of freedom. 8 One degree of freedom will eventually be removed by gauge constraint and the helicity-0 modes sector are in general consist of two degrees of freedom. Notice that all the second order time derivatives appear on the second line with the coefficient (β + 3α) -this is the well-known fact [14] that if we choose β + 3α = 0, the massive scalar will be frozen and removed from the theory because of its infinite mass. The only degree of freedom in this sector is the helicity-0 mode of massive graviton.
On the other hand, we know that β = 0 is simply an f (R) type theory which is degenerate and hence is also ghost-free -this fact is not manifest in the eq. (3.16) above if we simply set β = 0. However, when β = 0, the action can be rearranged as where we have suggestively written the second line in the action (3.17) as a complete square. By varying Φ, we obtain

JCAP05(2014)010
Inserting eq. (3.18) back into the action (3.17) we obtain the action of a single non-ghostlike massive scalar field as we would expect for f (R) type theories. It is clear that since the action is only dependent on Ψ andΨ, there is only one ghost-free d.o.f.. Notice that if α < 0 this scalar is a tachyonic unstable d.o.f., which is consistent with the general f (R) gravity, where we require f (R) > 0 to avoid tachyonic instability [7,40]. By setting α → 0 means that the mass term blows up and rendering this d.o.f. non-dynamical, i.e. it reduces to simple General Relativity.
Harking back to the action for general α and β, eq. (3.16), Ostrogradski's choice of canonical coordinates is (3.20) where the choice β + 3α = 0 means that p χ = 0 becomes a primary constraint instead of an additional d.o.f.. The Hamiltonian can be expressed by the canonical coordinates The primary constraint is ϕ 1 : p Φ = 0 and all the constraints can be generated by the consistency relation where ≈ means "weak equality" (i.e. only satisfied when the variables are on-shell) -see [41] for a discussion on this point. Since ϕ 1 , ϕ 2 are second class, 9 we can use them to reduce the phase space (Φ, p Φ ), and the reduced Hamiltonian is

JCAP05(2014)010
The linear dependence of p Ψ again renders the Hamiltonian unbounded from below. In order to see the mass content of helicity-0 modes, we will need to further diagonalize the Hamiltonian by the following canonical transformation: The diagonalized Hamiltonian is then The reduced Hamiltonian of helicity-0 sector contains two massive degrees of freedom. One is a massive ghost comes from massive graviton with mass m −2 1 = −β and the other is massive scalar with positive definite kinetic energy, with mass m −2 2 = 2(β + 3α). Let us combine the result from all sectors. In 3.1, we saw that there are four helicity-2 degrees of freedom and two of them suffer from ghost-like instabilities. In 3.2, the two helicity-1 degrees of freedom are either ghostlike or tachyonic, depending on the sign of β. In 3.3, one of two scalar degrees of freedom is ghostlike. With β < 0, one can see the unstable modes in helicity-0, 1, 2 sectors are massive with mass m −2 = −β, which corresponds to the massive graviton. This result is derived by the Stelle in his seminal work on higher derivative gravity [31] using an auxiliary field methodology. Here we rederived the results using the usual Hamiltonian formalism.
There are two special choices of parameters in the linearized theory. With α = 0, β = 0, the massive graviton sector gains an infinite mass and hence becomes non-dynamical. In this case the theory consists of one massless graviton with one massive scalar field (i.e. an f (R) theory). On the other hand, by taking the limit β + 3α = 0, the massive scalar field becomes infinitely massive and hence non-dynamical. In this case, the theory's particle content reduces to one massive and one massless graviton. With the latter choice and a total minus sign, at the linear level one can have a theory with a healthy massive graviton [27], since this choice is consistent with the Fierz-Pauli tuning. However, one should expect that the Boulware-Deser ghost [42] will enter at the nonlinear level.

Stabilization by constraints in Minkowski background
In this section, we will demonstrate how to remove the unstable degrees of freedom by introducing constraints via auxiliary fields. As shown in [5], this will result in the effective dimensionality of phase space being reduced. Roughly speaking, we impose the constraints such that the auxiliary fields are related to second order time derivative of the unstable fields, resulting in the final equations of motion being second order in time derivatives yet -9 -JCAP05(2014)010 up to fourth order in spatial derivatives. The advantage of preserving spatial part of the "higher derivative" component is that we retain the improved renormalization properties of such theories, at the price of giving up Lorentz invariance.
One might ask what if we remove the instabilities without explicitly breaking Lorentz invariance? Here we emphasize that we can equally insert constraints to remove the higher spatial derivatives, with the end result being a stable 2nd order theory both in space and time derivatives. For example, the unconstrained helicity-2 action (3.5) can be written as Without the full theory, we do not know how to impose constraints by introducing the auxiliary field λ into the action without breaking Lorentz invariance while removing the highest time derivative in the equations of motion. The best thing we can do is to couple λ ij with E ij , and the Lorentz invariance is not explicitly broken by extra terms. We can modify the action as If a = 1, we force λ ij coupling to every E ij and if a = 0 we only force λ ij coupling to those E ij where the cannot be removed by an integration by part. The equations of motion of the theory are The equation is either trivial if a = 1 or a Klein-Gordon equation with mass m 2 = a 2 /4β(1−a) if a = 1. In both cases, the equations of motion will have same order of time derivatives and spatial derivatives and the improved renormalization properties will not be retained. For notational simplicity, from now on we drop the traceless notation B T i , E TT , which should be clear from the context.

Helicity-2 sector
We begin by introducing a helicity-2 auxiliary tensor field λ ij into the action (3.5) where λ ij is transverse traceless, which also explicitly breaks Lorentz invariance. The canonical coordinates are The Poisson bracket of a pair of transverse traceless canonical coordinates can be found as whereΛ ij,kl is the transverse traceless projection operator defined byΛ ij,kl is the transverse projection operator. Since the equations of motion in the Hamiltonian picture are generated by Poisson bracket, the projection operator will preserve the transverse traceless characteristic.
It is clear that p λij = 0 is a primary constraint as it is an auxiliary field. Via the consistency relation, we can generate further (traceless and transverse) secondary constraints as follow ϕ 1 :p λij = 0, We can use the constraints ϕ 1 , ϕ 4 to eliminate the degree of freedom (λ, p λ ), and use ϕ 2 , ϕ 3 to eliminate (q, p). The coefficients in the action (4.1) are chosen such that there are at least four constraints in the theory and there is no ∇ 2 in ϕ 3 which will generate nonlocal terms in the reduced Hamiltonian. Using the constraints, (q ij , p ij ) can be written as follow and the reduced Hamiltonian becomes To check whether the reduced Hamiltonian is bounded from below, we will explicitly quantize the theory. Similar to QED in the Coulomb gauge, one can follow Dirac's method to quantize the constrained system. We first write down the generalized version of Poisson bracket (i.e. Dirac bracket), which generates time evolution of any fields in constrained theory while preserving all the constraints. We then promote all the fields to operators and the commutators of two fields now become i times the Dirac bracket of them.

JCAP05(2014)010
To write down the Dirac bracket, we first define a matrix C ab ≡ [ϕ a , ϕ b ] P B , whereâ andb are two operatorsâ ≡ 2(−1 + 2β∇ 2 ) andb ≡ 2(β∇ 2 ∇ 2 + ∇ 2 − 1/β). The inverse of C ab is and the Dirac bracket of two field X, Y is defined by Equipped with Dirac bracket, one can use the reduced Hamiltonian to write down the equations of motion of this systemĖ Using eq. (4.8) and eq. (4.9), we find which is the Euler-Lagrange equation of the action (4.1). We can solve eq. (4.10) by taking the Fourier transform For any p,Ẽ ij (p, t) is a harmonic oscillator with frequency In order to quantize the theory, we write E ij , π ij as linear summation of creation and annihilation operators a r † p , a r p , where the coefficients are chosen in such a way that they solve the equations of motion eqs. (4.8), (4.9), and the superscript r labels the polarizations. The symmetric transverse traceless tensor r ij satisfies p i r ij = δ ij r ij = 0, and is normalized as , (4.14) with the completeness relation .
The operator Λ ij,kl (p) is defined by replacing all the ∇ 2 in the transverse traceless projection operator by −p 2 . One can calculate the Dirac bracket of (E ij , π kl ) and the commutator of the two operators is thus With the normalization eq. (4.14) and the completeness relation eq. (4.15), the commutation relation eq. (4.16) is equivalent to One can thus rewrite the reduced Hamiltonian (4.7) as creation and annihilation operators The energy spectrum is real and bounded from below if w 2 p is positive definite as long as β > 0.

Helicity-1 sector
We now turn to the attention of the helicity-1 unstable modes. As shown in 3.2, this sector is tachyonic if β < 0 and ghostlike if β > 0. As usual, we will remove it by modifying the action (3.10) with the introduction of a helicity-1 field λ i (4.20) and the Hamiltonian is There are four constraints in the theory, which can be found as If we use the four constraints to eliminate (v i , p i v ), (λ i , p i λ ), the physical phase space will be zero dimensional and the reduced Hamiltonian vanishes.

Helicity-0 sector
Finally we introduce a helicity-0 field λ into the action (3.16) where A is some arbitrary real constant. There are four naive degrees of freedom, whose canonical variables are The Hamiltonian can be written as where the constraints of this theory are We use the six constraints to eliminate three pairs of canonical coordinates (Φ, p Φ ), (λ, p λ ), and (χ, p χ ), reducing the Hamiltonian to become which generates the evolution of a single dynamical variable Ψ. It is clear that it can be made positive definite with some parameter choice, for example, (A = −8, α = 0, β > 0). We can check that the quantum theory is also stable in the following manner. First we find the Dirac bracket of the theory as usual, which can be used to find the equations of motioṅ

JCAP05(2014)010
Following the usual quantization rules, we define the commutator of Ψ and p Ψ to be "i" times their classical Dirac bracket, i.e.
[Ψ, p Ψ ] ≡ (12 + A)i 2(6 + A) + 32(β + 3α)∇ 2 δ (3) (x − y). (4.32) By expanding Ψ, p Ψ as linear summation of creation and annihilation operators a † p , a p (4.33) we obtain the usual result that the commutator eq. (4.32) is consistent with the Fock space commutators Using these operators, the reduced Hamiltonian can then be written as which is bounded from below as long w 2 p is positive definite, a condition which is satisfied by suitable choices of the parameters A, α and β.

Quadratic action around de Sitter background
In this and next section, we will show how to remove the unstable degrees of freedom in each helicity sector in de Sitter background. In order to separate the action into helicity-0, 1, 2 sectors, we first parameterize the metric fluctuation as where t is conformal time in this and next section, and in de Sitter background we have a(t) = − 1 Ht , Λ = 3H 2 . We can again decompose B i , E ij into

Helicity-2 sector
The second order action of helicity-2 modes in de Sitter background is  1.3), we note that in de Sitter space R ≈ H 2 , and hence we generally expect that |α|H 2 1 and |β|H 2 1 if we reasonably suppose that the higher derivative terms are corrections to usual General Relativity. This means that generically we should expect c > 0 unless the higher derivative terms dominate. As an aside, note that in the special case β + 3α = 0, c = 1.
Since there is no constraint in the helicity-2 modes, there are four helicity-2 degrees of freedom in the theory. Ostrogradski's choice of canonical coordinates is 6) and the Hamiltonian is As in eq. (5.7), the linear dependence of π ij is the signal of Ostrogradski's instability. The π ij q ij term can be arbitrarily negative when q ij > 0, π ij → −∞ or vice versa and hence the Hamiltonian is thus unbounded from below.

Helicity-1 sector
The action up to quadratic level of helicity-1 modes can be written using the gauge invariant 8) and the Euler-Lagrange equation of eq. (5.8) is By Fourier transform, we find that the solutions are harmonic oscillators with frequency The canonical momentum conjugate to v i is then defined by Since we use the gauge invariant variable to write the action, there is no constraint and the Hamiltonian is

JCAP05(2014)010
There is a subtle but important difference between de Sitter and Minkowski backgrounds for the scalar modes. In the Minkowski case, eq. (3.13), the helicity-1 mode is either tachyonic or ghostlike since the sign of v i v i is always negative. However, in de Sitter background, one may choose c < 0 to render the coefficient of v i v i in eq. (5.11) to be positive. Nevertheless, as we have argued that generically c > 0 unless the higher derivative terms dominate, we will not consider this case further. The helicity-1 sector is thus either tachyonic or ghostlike, depending on the sign of β.

Helicity-0 sector
The second order action for helicity-zero modes in dS space is much more complicated. We can use the usual two gauge invariant variables 12) to write the action as whereupon the action becomes much shorter, i.e.
The two set of gauge invariant variables eqs. (5.12) and (5.14) are related by the following field redefinitions

JCAP05(2014)010
One can see that there is no corresponding field redefinitions in Minkowski background, because in Minkowski case, aH = 0. We will use the new pair of fields (Φ, B) to show how to remove the instabilities in this and the following sections. We made this choice to evade the possible issues with the physical interpretation of the nonlocal terms -we emphasise that in principle one can use the old pair of fields (Φ, Ψ) and adding the same set of auxiliary field terms to remove all the instabilities, similar to the helicity-1, 2 cases. 10 Since there might not be any nonlocal terms in the full non-linear theory, this procedure could be a better starting point -however, since the non-linear theory is beyond our scope, here we only want to show the same idea can be applied to the de Sitter case. Using these new set of variables, the canonical momenta can be written as (5.17) and the Hamiltonian thus becomes To make the dynamics explicit, we perform a final canonical transformation B, (5.19) such that the Hamiltonian becomes

JCAP05(2014)010
Since there is no constraint in this theory, this is the Hamiltonian describing two physical degrees of freedom. Although the instability is not explicitly shown, one can see in some limit the Hamiltonian is unbounded from below. In order to have stable kinetic terms, we require that (β + 3α) > 0 and β > 0. On the other hand, in the high frequency limit in Fourier space, one should expect those terms with the highest spatial derivatives to dominate. This requires that (β + 2α) < 0 so that the B 2 term is stable (from the third line in eq. (5.20)), which cannot be satisfied at the same time. We thus conclude the Hamiltonian is unbounded from below with any parameter choice in the high frequency limit.
6 Stabilization by constraints in de Sitter background

Helicity-2 sector
Similar to the Minkowski case, we first rewrite the action (5.4) by introducing a helicity-2 auxiliary tensor field λ ij where λ ij is transverse traceless and the Lorentz invariance is explicitly broken by λ ij . Ostrogradski's choice of canonical coordinates is and the Hamiltonian is The Poisson bracket of a pair of transverse traceless canonical coordinates is identical to their Minkowski counterparts To find the constraints, we apply the Dirac Bracket formalism as usual. It is clear that p λij = 0 is a primary constraint, and the rest of the (transverse and traceless) constraints of this theory are generated by the consistency relation

JCAP05(2014)010
Armed with these, we can use ϕ 1 , ϕ 4 to eliminate the degree of freedom (λ, p λ ), and use ϕ 2 , ϕ 3 to eliminate (q, p). The coefficients in the action (6.1) are again chosen such that there are at least four constraints in the theory and there is no ∇ 2 in ϕ 3 which will generate nonlocal terms in the reduced Hamiltonian. Using the constraints, (q ij , p ij ) can be written as follow 6) and the reduced Hamiltonian becomes which is positive definite if β > 0, c > 0.

Helicity-1 sector
In section 5.2, we showed that the helicity-1 modes are only stable if β > 0, c < 0. However, our imposition of the constraints to restore stability of the helicity-2 sector requires that c > 0 in addition to the usual arguments on subdominant higher derivative terms. We thus choose c > 0 and remove the unstable helicity-1 modes altogether as follows. Similar to the Minkowski case, we modify the action (5.8) by introducing a helicity-1 field λ i (6.9) and the Hamiltonian is There are four constraints in the theory, which can be found as If we use the four constraints to eliminate (v i , p i v ), (λ i , p i λ ), the physical phase space will be zero dimensional and the reduced Hamiltonian vanishes. Finally, we deal with the helicity-0 instability. We modify the action (5.15) by introducing a helicity-0 field λ As now must be familiar, the canonical coordinates are p B =4(β + 2α)(Ḃ − λ + ∇ 2 Φ) + (β + 3α)(8a 2 H 2 Φ + 8aHΦ) (6.13) and the Hamiltonian is There are four constraints in the theory, which can be found as ϕ 1 : p λ = 0 If we require H R > 0, which means every term in eq. (6.16) needs to be positive definite, there are two possibilities:
Choosing either possibility will result in a stable helicity-0 sector.

Conclusion and future direction
We investigate the instabilities in higher derivative gravity models with quadratic curvature invariant R 2 , R µν R µν by expanding action to the quadratic level of metric fluctuation around Minkowski/de Sitter background. We show how the instabilities in the helicity-0, 1, 2 sectors can be removed by some choices of additional constraints. With help of the constraints, the degrees of freedom are reduced from two helicity-0, two helicity-1, and four helicity-2 to one helicity-0, zero helicity-1, and two helicity-2 modes. The fact that the phase space has to be reduced -i.e. it is impossible to modify the theory via constraints such that the instabilities are "made stable" -is an expression of the theorem proven in [5] that Ostrogradski's instability can only be removed if the original theory's phase space is reduced. We emphasize that adding constraints to remove instabilities is only valid in the linear theory. A full non-linear extension of this methodology is beyond the scope of this paper, and we have made no attempt at a covariant formalism. However, even in the linear theory, some features of a stable higher derivative gravity can be gleaned. First, it is clear that a general higher derivative theory which is stable and possesses the desirable renormalization properties breaks Lorentz invariance. Indeed, the "stabilized" theory has the form of a low energy effective limit of a Lorentz violating, much like that of Hoȓava gravity.
Second, the stable higher derivative theory has no helicity-1 modes, at least in the Minkowski case since this mode is unstable in the original theory and hence need to be removed. The de Sitter case is less clear-cut -the helicity-1 sector may be made stable by the curvature term although we have chosen to remove it to be consistent with the stability of the helicity-0 sector.
Although the full non-linear extension of this procedure is not yet known, we would like to point out several directions one can work on. First, one could expand the action eq. (1.3) around constant curvature background to the next order, the action would be cubic in metric perturbation h µν and one can check if the stabilization procedure still work. If the procedure works at the next order, one can see whether there is a pattern that we can use to generate the full non-linear extension. The second possible thread of study is to stabilize the linear action covariantly, although the theory would not have improved renormalization properties, it would be easier to find the pattern and thus generalize the methodology to the full non-linear theory. It will be very interesting to check whether this result can be extended to the full non-linear regime.