Quantifying the impact of changing the threshold of New York City heat emergency plan in reducing heat-related illnesses

The adverse impact of high heat on population health is widely documented, particularly in the United States (US) and Europe, with growing evidence for other regions (Anderson and Bell 2011, Anderson et al 2013, Guo et al 2017, Green et al 2019). Days of high heat have a substantial public health burden by impacting different organs (heart, kidneys, brain or lungs) through various mechanisms (heat cytotoxicity, inflammatory response or ischemia for example). Health consequences range from heat exhaustion to death (Bouchama et al 2007, Mora et al 2017). Notably, untreated heat exhaustion can progress to heat stroke which is a life-threatening condition characterized by the manifestation of different symptoms (i.e. elevated body temperature, rapid pulse, headache, dizziness, nausea, or unconsciousness). In the US, an average of 658 heat-related deaths are estimated to occur each year (Fowler et al 2013). Yet, it also important to consider heat impacts on morbidity outcomes that may be a more relevant early indicator to monitor heat-related health impacts. Particularly, it is possible to use heatstroke and related ICD diagnoses as indicators of the many heat-related health outcomes (Bobb et al 2014, Peiris et al 2017), although many other health outcomes can occur from extreme heat (see: http://blr.hcpro.com/content.cfm?content_id=307085)

Heat-related illnesses are mostly preventable in western countries, however extreme heat remains the main weather contributor to the burden of disease in the United States (National Weather Service (NWS), 2017). Furthermore, heat-related morbidity and mortality are likely to increase under climate change (Li et al 2013, Sanderson et al 2017). Overall temperatures are anticipated to increase, and extreme heat days or heat waves are expected to occur more often, last longer, and be hotter. The need for adaptation measures for climate change has been emphasized by many US agencies including the US EPA, NASA, and others (O'Neill et al 2010, McGregor et al 2015). Therefore, the implementation of effective adaptation measures are urgently needed.

Growing evidence related to the dangers of heat has led to the development and implementation of preventive measures in many jurisdictions across the US. Heat early warning systems are based on forecasted periods of high heat and aim to provide notice to public health and emergency management offices (McGregor et al 2015). These early warning systems also target the public through local media by providing specific recommendations for individual protective behaviors to adopt (Boeckmann and Rohn 2014, Hess and Ebi 2016). In most US cities, local National Weather Service (NWS) offices issue heat advisories in advance of forecast periods of high heat (White-Newsome et al 2014, Hess and Ebi 2016). Such heat alerts are based on heat index forecasts but the criteria used to trigger heat alerts varies across NWS offices. However, their effectiveness in preventing heat related illnesses may be limited. In a recent paper, Weinberger et al evaluated if such NWS heat alerts were associated with lower rates of mortality across 20 US cities between 2001 and 2006 and did not find strong evidence of the effectiveness of early NWS heat alerts (Weinberger et al 2018).

In addition, such NWS heat alerts can also trigger additional local interventions (local heat emergency plan or heat action plans) that include additional community-based actions to protect the public from the health impacts of heat. These local heat emergency plans typically include additional messaging targeting identified vulnerable populations, intensification of surveillance of signs and symptoms of heat-related illness, daily contacts by telephone and home visits to home care patients or the opening of cooling centers (White-Newsome et al 2014, McGregor et al 2015). Various cities in Arizona (Kalkstein and Sheridan 2007), Philadelphia (Ebi et al 2004) Wisconsin (Weisskopf et al 2002) and Montreal, Canada (Price et al 2013) have developed such local heat emergency plans. Most existing local heat emergency plans have been implemented in the last 10 years (White-Newsome et al 2014) and it is interesting to highlight that health departments are more frequently engaged in heat response coordination in recent years (White-Newsome et al 2014) while traditionally (Bernard and McGeehin 2004) such responses were coordinated by public safety or emergency management offices.

New York City (NYC), a densely populated urban center with landscape characteristics that exacerbate heat through micro-heat islands (Hamstead et al 2016), is a location where the burden of high temperatures is high (Metzger et al 2009, Matte et al 2016). While NYC authorities (including the NYC Health Department and NYC Office of Emergency Management, OEM) traditionally relied on NWS heat alerts criteria to activate their local heat emergency plan, such criteria used were not derived from local epidemiologic analysis of heat-dependent health effects, and instead were based on national criteria (see: https://weather.gov/bgm/heat). In the Weinberger paper (Weinberger et al 2018), during the period 2001–2006, no changes in mortality attributable to the NWS heat alerts have been observed in NYC.

In 2008, the NYC OEM changed their local heat emergency plan by lowering the threshold for triggering heat advisories based on local epidemiological studies that identified that heat related burden occurred at temperatures below the NWS criteria (Ito et al 2018). The initial threshold of 105 °F (40.6 °C) for 1 d was thus changed to the forecast maximum heat index of 100 °F (37.8 °C) for any period of time (i.e. one day or more), or 95 °F (35 °C) for at least two consecutive days.

When heat emergencies are activated, NYC provides residents with a multitude of services to reduce the health effects of extreme temperature. NYC OEM disseminates detailed information on the different tiers of heat wave alerts, the characteristics of the environment that contribute to heat waves, and the potential deleterious effects to human health through their online resources (see examples in supplemental material). This plan also includes homeless outreach, and protection of water and power supply. Additionally, NYC opens cooling centers in air-conditioned facilities, such as libraries, community centers, senior centers and NYC Housing Authority facilities, to offer people relief from the heat. Individuals who have no access to a cool environment, and particularly those at risk for heat-related illness, are encouraged to use the cooling centers during a heat wave (www.nyc.gov/beattheheat). Residents can further sign up to receive heat wave alerts directly to their mobile phone, using the NotifyNYC system. Despite these efforts made in NYC to prevent heat-related illnesses during extreme heat days, evidence regarding the effectiveness of the NYC local heat emergency plan is lacking. Previous studies (Weinberger et al 2018) only considered the potential influence of NWS heat alerts before 2008 and analyzed mortality outcomes only. The set of interventions provided by NYC OEM were not altered in 2008 when the threshold was reduced.

The aim of this study is to quantify the quantifying the effectiveness of changing the threshold for triggering heat advisories and operation effectiveness of the NYC Heat Emergency Plan. Specifically, we estimate the change in heat-related risk of hospital admissions for the elderly population before and after 2008, when the threshold for triggering heat advisories and operation was lowered. We applied a quasi-experimental study design using the Difference-in-Differences (DID) method coupled with the propensity-score matching (PSM) that exploited the specific eligibility criteria for triggering the NYC local heat emergency plan.

Data

We used a DID approach to evaluate the effectiveness of the threshold change in the NYC Heat Emergency Plan in reducing heat-related risk of hospital admissions for an older population. This approach exploited the particular choice of the heat index criteria defining days that were eligible or not for triggering the NYC Heat Emergency Plan (i.e. extreme hot days or not). We classified each day as eligible for the Plan or not based on the following criteria: 100 °F heat index or higher for one day or more, or ≥95 °F (for each day) for at least two consecutive days. We did not quantify the specific impact of heat waves duration. Our study period for the main analysis is 2006–2010 throughout the summer season (May–September). We chose a short interval of time (2 years before (2006 and 2007) and 2 years after (2009 and 2010) initiation of NYC Heat Emergency Plan), to limit potential confounding due to population acclimatization and urban infrastructure changes (Petkova et al 2014) that might influence the impacts of hot days on health after 2008, as well as other potential long term trends. By focusing on such a short term interval of time, we also aim at isolating the change potentially attributable to the threshold change as the set of actions did not change between 2008 and 2010. We also ran an additional analysis in which we restricted days to <105 F so we can focus on newly eligible days after 2008. We also included years before 2006 (2003 and 2004) as a sensitivity analyses.

We compared the difference in daily rates of heat-related illnesses (defined as heat stroke and related illness) between eligible and non-eligible days before and after implementing the policy (2006–2007 versus 2009–2010). Our counterfactual quantity of interest is the difference in the daily rate of heat-related illnesses between eligible and non-eligible days that would have occurred during 2009–2010 had the threshold modification of the NYC Heat Emergency Plan not been made. The key assumptions of DID analysis are that the trend in the control group represents a good approximation for the counterfactual trend of the treated group in the absence of the treatment and the absence of other interventions contemporaneous with the implementation of the policy of interest. We also assume that no other policy potentially influencing heat related illnesses had been implemented at the same time. We augmented the DID with PSM to achieve exchangeability between the eligible and non-eligible days and thus comply with DID assumption related to pre-policy similar trends between both groups. PSM techniques aim at achieving covariate balance in regard to measured background covariates (i.e. confounders) (Austin 2009). We also visually inspected pre-policy trends of maximum heat index and other environmental daily level variables between on eligible and non-eligible days.

Using a logistic regression, we estimated the propensity score for each observed day to predict the probability that a day was eligible or not (extreme heat day [EH = 1]). We used in this propensity score model the following measured covariates: PM_2.5, Ozone, day of week (DOW: weekend versus week days). To incorporate the possible lag effects of all covariates, we included ${X}_{i-j},$ a vector of observed predictors for the $j{\rm{t}}{\rm{h}}$ day prior to event $i$ where $\left.\,j\in j,5\right].$ The following model was used to a derive propensity score for each day during the study period (except 2008): $\mathrm{log}\,it\,P\left[EH=1| X\right]={\beta }_{0}+{\beta }_{1}\left({\rm{D}}{\rm{O}}{\rm{W}}\right)+\displaystyle \sum _{n=0}^{5}{\beta }_{2n}X{n}_{{{\rm{lag}}}_{n}}$. Here, X_n includes PM_2.5 and Ozone. We thus obtained a propensity score for each day which was used to match eligible days to similar non-eligible days. We used 'nearest-neighbor' matching (Dehejia and Wahba 2002, Austin 2011) to match eligible days to similar non-eligible days with optimizations assessed for choice of matching ratios and caliper. We chose the ratio and caliper that provided the lowest average imbalance and the lowest average standardized difference across covariates after matching. A ratio of 1:1 and a caliper of 0.2 have been used for our final model. Then, conditional models considering this matching were used in the DID models. We used a DID model accounting for temporal pattern such as year and month, to estimate the program impact on daily events, as follows:

$$\begin{eqnarray*}&&g\{E\left(Y\right)\}={\beta }_{0}+{\beta }_{1}E+{\beta }_{2}P+{\beta }_{3}E\,* \,P+{\beta }_{4}Xn.\end{eqnarray*}$$

Here, $g\left\{.\right\}\,\,$is a generic link function. We used Poisson regression with identity and log link function. $Y$ denotes daily heat illnesses rates. $E$ represents a binary variable indicating whether the days were eligible or not, whereas $P$ indicates whether the days were before or after 2008. $Xn$ represents all other time-varying covariates included in the model (e.g. calendar year and month). The DID estimate represents the product interaction term (${\beta }_{3}$). We then quantified the marginal number of daily heat illnesses. A positive value of the DID estimate represents an estimated reduction of daily heat illnesses attributable to the NYC Heat Emergency Plan modification. We estimated 95% confidence intervals (CIs) for the adjusted DID estimate by bootstrapping (1000 samples).

We also included some placebo tests by conducting the same analyses as above, but falsely defining the threshold change of the NYC Heat Emergency Plan in 2005. We thus compared differences in daily heat illnesses on eligible and non-eligible days between 2003–2004 and 2006–2007. We then tested a different criterion to classify days as eligible days. As a second sensitivity analysis using the actual years (comparing 2006–2007 to 2009–2010) and generated a false definition of hot days as any day above 90 °F (and below 100 °F) for one day or more, expecting no difference between eligible and non-eligible days defined in this way.

We estimated a total of 158 Heat-related Illnesses (ICD-9-CM, category 992 including heat stroke and related illness) during the study period (May–September between 2006 and 2010 (including 2008), 488 in total). The average number of daily Heat-related Illnesses was 0.26 (SD: 1.25) with a minimum of 0 cases per day, and a maximum of 20. During days defined as non-hot days, the average number of illnesses was of 0.07/day (SD: 0.26) before the implementation of the modification of the NYC Heat Emergency Plan and 0.03 (SD: 0.18) after its implementation. During days defined as hot days, this estimate changed from 1.49/d (SD: 3.62) before the implementation of the modification of the NYC Heat Emergency Plan to 0.65/d (SD: 1.49) after its implementation.

Table 1 presents descriptive statistics for heat index and other covariates (ozone, PM_2.5, etc) before and after NYC Heat Emergency Plan modification for non-hot days (n = 383) and hot days (n = 105). We observe that the distribution of covariates between the two periods (before and after implementation of the Plan) were similar except for PM_2.5 for which we observed a reduction after 2008. The proportion of hot days (25%) was higher in the post-NYC Heat Emergency Plan period than before its modification (18%).

In order to ensure covariate balance (exchangeability) between hot and non-hot days, we used a PSM approach, where hot days, before and after the plan modification, were matched to non-hot days that are similar in regards to covariates. Our visual comparison of the standardized mean differences of each covariate confirmed that PS matching considerably improved balance (see figure S1 is available online at stacks.iop.org/ERL/14/114006/mmedia). We can observe that trends in Heat-related Illnesses between hot and non-hot days were parallel before 2008 (see appendix).

We found that the modification of threshold of the NYC Heat Emergency Plan is associated with reduced Heat-related Illnesses during hot days as compared to a situation if the threshold change had not been made. The final DID estimate when comparing 2006–2007 to 2009–2010 was 0.80 (95%CI: 0.27; 1.33), which means that on average 0.80 fewer Heat-related Illnesses per day were observed during hot days after the modification of the NYC Heat Emergency Plan, which corresponds to a total of ~50 prevented Heat-related Illnesses during the summers of 2009 and 2010. Considering the total number of Heat-related Illness cases in our study period (i.e. 158), this estimate represents an important proportion of prevented cases. When restricting the analysis to days <105 F we found a DID estimate of 0.29 (95%CI: 0.06; 0.52).

Table 2 presents final DID estimates from sensitivity analyses, which supports our hypothesis that the Heat Emergency Plan may have contributed to reduce heat-related illness. When falsely defining the modification of the NYC Heat Emergency Plan in 2005, we did not find evidence of a difference in the effect of hot days on Heat-related Illnesses after the false NYC Heat Emergency Plan. We also failed to find evidence of an effect of a false plan when an untrue criterion to classify days as eligible days (any day above 90 °F and below 100 °F).

In this paper, we found that the 2008 modification of threshold of the NYC Heat Emergency Plan contributed to a reduction in the rate of heat-related illnesses during days defined as heat waves. This result is particularly relevant to highlight the importance of using local epidemiological evidence to inform thresholds for triggering heat advisories and operation that are accompanying NWS heat alerts. Indeed, the study by Weinberger et al did not identify any health benefits in relation to the activation of NWS heat alerts (Weinberger et al 2018) before 2008. When focusing on the period before 2008 (see sensitivity analyses in table 2), we had similar conclusions on heat related illnesses which highlights the specific role of the threshold change. There are two main reasons for which such changes in the NYC local heat plans can be expected to be more effective in reducing heat-related burden than NWS heat alerts alone. First, the changes in the NYC Heat Emergency Plan led to its activation based on temperature thresholds resulting from local epidemiologic analysis (Ito et al 2018) while NWS heat alerts thresholds were typically based on regional or national criteria. Second, while NWS heat alerts are mainly aiming at communicating recommendations for individual protective behaviors to adopt during a heat event, the NYC Heat Emergency Plan includes several additional actions (e.g. homeless outreach, home visits to home care patients or the opening of cooling centers) in junction with local government and nongovernmental organizations based on the identification of local populations at high risk for heat-related illness.

NWS heat alerts, as currently implemented, mainly aim at increasing heat awareness for the population as a whole and implicitly encourage individual actions to protect their health during extreme heat events. As described by Sheridan, while the general population appears to be aware of such heat alerts, relatively few people take concrete action to protect their health during extreme heat events (Sheridan 2007). Such phenomenon has been widely documented for other public health campaign-type interventions (Yadav and Kobayashi 2015). In a survey conducted by Lane et al about the awareness of heat alerts, it appeared that most New Yorkers are indeed aware of heat warnings when activated (Lane et al 2014).

Interestingly, in their paper, Weinberger et al identified that NWS heat alerts were effective to reduce mortality (by approximately 45 deaths each year) in Philadelphia, which had an extensive Heat Emergency Plan (labeled heat response plan) in place during the study period and also based on local epidemiological evidence. Activities in Philadelphia's Heat Emergency Plan include similar activities to the NYC's plan including for instance home visits to vulnerable individuals or increasing the number of emergency medical service personnel (Sheridan and Kalkstein 2004). While it is unrealistic to attribute such difference to the existence of the Philadelphia heat response plan only, this finding is interesting and resonates with our postulate above about the importance of having local interventions in parallel to NWS heat alerts.

Besides, it has been shown that some population subgroups are particularly vulnerable regarding the heat impacts on health (Åström et al 2011, Benmarhnia et al 2015) and relying only on a single intervention to encourage self-behavior change is expectably not having significant health benefits. In addition, such identified vulnerable populations are neither vulnerable to heat for the same reasons nor requiring the same types of interventions (Abrahamson et al 2009, Benmarhnia et al 2018). For example, the Montreal Heat Emergency Plan (labeled Heat Action Plan), implemented in 2004, includes important efforts to target different vulnerable populations during heat waves (Price et al 2013, Price et al 2018). In this plan, there are different messages and levels of interventions for different population sub-groups, based on individuals' conditions (e.g. presence of risk factors like social isolation, lack of air conditioning, medical conditions). This strategy has been shown to be effective in reducing inequalities in heat-related mortality (with more benefits for elderly and individuals living in low SES neighborhoods) (Benmarhnia et al 2016). Yet, some recent evidence (Benmarhnia et al 2018) also shows the important heterogeneity regarding needs and challenges during a heat wave that may exist between different groups (i.e. individuals diagnosed with schizophrenia, and one was composed of individuals who have alcohol or drug addictions) both identified as vulnerable in the Montreal Heat action plan. This suggests that a 'one size fits all' approach for reducing heat-related illnesses does not appear to be a successful strategy in this context and that actions that go beyond communication and educational efforts are still needed. In the future, it would be interesting to explore to which extent some population subgroups are benefiting more (or less) from the NYC Heat Emergency Plan activities when activated and if some spatial heterogeneity exists in its effectiveness.

Other communities in the US (White-Newsome et al 2014) and in other locations (Pascal et al 2006, Knowlton et al 2014) have implemented policies to prevent heat-related illnesses and deaths with a large heterogeneity in the included activities. Evaluating the effectiveness of such programs is crucial especially when only communication actions aiming at encouraging self-behavior change are included. As discussed by Weinberger et al while the majority of published work on this topic showed evidence of a protective effect regarding policies to prevent heat-related health effects, it is likely that null or negative findings from other locations (or other studies at the same locations) would not be accessible due to publication bias.

There are some limitations in our study that are important to acknowledge. We a priori focused on one diagnosis only (i.e. heat-related illnesses) as we aimed to identify some effects for which the mechanisms would be unambiguous. It is possible that the probability of diagnosing heat-related illnesses has increased because of the NYC Heat Emergency Plan itself, but as shown in table 1, this is unlikely as the average number of such diagnoses was lower in the post-implementation period in both hot and non-hot days. While our inference is only valid for such diagnosis, it has been shown that different diagnosis can be attributable to exposure to extreme heat (Bobb et al 2014, Hopp et al 2018) so our results can be seen as very conservative regarding the public health benefits. Indeed, there are different morbidity outcomes that can be exacerbated by heat such as renal failure or fluid and electrolyte disorders. Thus, we can reasonably expect the benefits to be larger through these other mechanisms. It is also important to highlight that the threshold used for triggering heat advisories and operation in NYC is binary by nature and does not vary directly in proportion to the severity of extreme heat events. In our analysis, we were not able to consider such pattern in heat severity (Matte et al 2016). Then, we used daily maximum heat index observed values. Yet, such plans are activated based on forecasts and false positives and negatives are possible (Åström et al 2015). We did not have access to any information regarding the forecasted temperatures but is has been shown that short-term forecast (e.g. 1 d), which are typically used in this context tend to be accurate when compared to observed weather conditions (Zhang et al 2014). We also can reasonably assume that such discrepancy between forecasted and observed weather data remained unchanged before and after the threshold modification. Then, we did not explore specific mechanisms of the NYC Heat Emergency Plan that contributed to reduce heat-related illnesses during hot days. Finally, by using a single daily heat index value, we were not able to get within-city temperature variability. Given the important spatial heterogeneity regarding micro-heat islands that exist in NYC (Hamstead et al 2016) and the effect measure modification role such within-city temperature variability can have on health (Schinasi et al 2018), it would be interesting to explore such research question in future studies.

With this study, we conclude that the modification of the threshold of the Heat Emergency Plan occurring in NYC 2008 may have been effective in reducing heat-related illness. This suggests that emergency heat plans that are activated based on temperature thresholds from local epidemiological data and include a range of actions that target populations at high risk are valuable in decreasing heat-related illness. Heat action plans should continue to be implemented and evaluated in various climates and contexts to determine the full scope of their protective effect, and to further understand which specific activities have maximal impact and which populations benefit from them. Overall, the results suggest local heat action plans to be a promising strategy to decreasing the health impacts of high heat.

We would like to thank Kazuhiko Ito from New York City Department of Health and Mental Hygiene, New York, NY for his valuable inputs about the New York City Heat Emergency Plan and critical revision of the manuscript.