Comparing methods for assessing the effectiveness of subnational REDD+ initiatives

Reducing emissions from deforestation and forest degradation and enhancing forest carbon stocks (REDD+) has emerged as a key climate change mitigation strategy within the United Nations Framework Convention on Climate Change (UNFCCC). Through the Paris agreement, the necessity for supporting and implementing REDD+ was reconfirmed and the role of forests as carbon sinks emphasized (UNFCCC 2015). So far, approximately 40 countries⁷ mention either REDD+ or forests as part of the mitigation strategy in their Nationally Determined Contributions (NDCs). This importance makes it critical to monitor and evaluate the carbon effectiveness of REDD+.

The measurement, reporting and verification (MRV) of carbon stocks and emissions is a vital part of national REDD+ schemes (Herold and Skutsch 2009, UNFCCC 2015). Carbon emissions are calculated by multiplying activity data—the area of land use/cover change due to human activity– by its corresponding emission factor (Verchot et al 2012). While national forest monitoring systems have progressed, e.g. with PRODES from the Brazilian Institute for Space Research (INPE), capacities in developing and operationalizing these MRV systems vary widely among countries (Romijn et al 2015). In the last decade, technical innovations in remote sensing and forest-relevant monitoring techniques resulted in a plethora of national and global datasets with increasing levels of coverage, detail (spatial and temporal) and accuracy. Examples include the Landsat-based Global Forest Change 2000–2014 (Hansen et al 2013), global pan-tropical biomass datasets (Baccini et al 2012, Saatchi et al 2011, Avitabile et al 2016), and national carbon maps using LiDAR (Asner et al 2013).

Meanwhile, at the subnational level, hundreds of REDD+ projects and programmes are led by a diversity of actors including private non-profit organizations, for-profit companies and government agencies (Simonet et al 2015). The implementers of these initiatives are applying a range of REDD+ interventions from enabling measures (such as tenure clarification) to command-and-control measures (disincentives) to direct payments and livelihood improvements (incentives). While data-driven developments facilitate forest and carbon monitoring, it remains unclear how to align information on subnational performance with national level reporting related to NDCs. The implementers of several of these subnational REDD+ initiatives state that 'vertical integration or nesting of MRV systems is important, but has been elusive' (Ravikumar et al 2015, p 919).

Any effectiveness assessment needs to compare an observed outcome with a hypothetical counterfactual (business-as-usual scenario, baseline or reference level). In the face of dynamic contexts globally (e.g. commodity prices), nationally (e.g. macroeconomic policies), and locally (e.g. newly constructed roads), simple retrospective 'before-after' (BA) reference level assessments fail to properly attribute factors of change, and consequently misjudge the impacts of REDD+ interventions. Establishing a counterfactual that discriminates these confounding effects is the key in assessing true policy impacts. The quasi-experimental Before-After-Control-Intervention (BACI), or differences-in-differences (DID), approach aims to control for these contextual changes. It is applied in ecological studies to assess the effect of a stress or treatment on a given population (Smith 2002) and in econometrics and social sciences for program evaluation (e.g. Imbens and Wooldridge 2009, Jagger et al 2010). The unit of interest is measured at (a minimum of) two points in time (before and after the treatment) and in (at least) two different locations, that is, an area subjected to the 'treatment' (intervention area) and an area that is not (control area), to identify changes that are additional. The BA approach corresponds to using a conventional reference level, i.e. the average historical deforestation (e.g. past ten years). Hence, unlike BACI, it does not account for changes in drivers during the intervention period. This paper explores the application of both methods to measuring the performance of subnational REDD+ initiatives. The purpose of the comparison is to increase our understanding of conditions under which the more complex and costly BACI approach is essential, and those conditions under which BA might be acceptable.

Here, we (1) develop a BACI method to assess the effectiveness of these REDD+ initiatives; (2) compare the results at the meso (initiative) and micro (village) scales; and (3) compare BACI with BA results. We focus on comparing the results of different methods and scales, rather than on explaining individual performance scores of the REDD+ initiatives.

2.1. Study area

Our study includes 23 subnational REDD+ initiatives in Brazil, Peru, Cameroon, Tanzania, Indonesia and Vietnam from CIFOR's Global Comparative Study on REDD+ (GCS) (figure 1). They differ greatly in terms of proponent type (government, NGO, private sector), size (ranging from 28 to approximately 160 000 km²), environmental context (from dense primary rainforest to dry miombo woodlands) and interventions applied (Sills et al 2014). While specific interventions differ across sites, most proponents use customized combinations of enabling measures, disincentives and incentives to reduce deforestation and degradation (Duchelle et al 2017).

Zoom In Zoom Out Reset image size

2.2. Tree cover data

2.3. Performance assessment framework

For both approaches, we aggregate the time series data on annual tree cover loss into two periods (before and after) (figure 2). To compare assessment approaches, we simultaneously apply BA and BACI approaches. Correspondingly, we calculate relative performance scores to allow for comparison across sites and countries.

Zoom In Zoom Out Reset image size

REDD+ initiatives' starting years differ, ranging from 2006 to 2013 (Sills et al 2014, appendix 6⁹), thus the number of years in the after period ranges from two to nine (see table 1). The BA score α is calculated as follows:

$$\begin{equation} \begin{array}{*{20}c} {\hbox{BA}} & {\hbox{score}\,\alpha = \overline x _{\hbox{AI}} - \overline x _{\hbox{BI}} } & {} \\ {\hbox{with}} & {\overline x _{\hbox{AI}} = \frac{1}{{n_a }}\sum\limits_{i = 1}^{n_a } {x_i \;\hbox{and}} } & {\overline x _{\hbox{BI}} = \frac{1} {{n_b }}\sum\limits_{i = 1}^{n_b } {x_i } } \\ \end{array} \end{equation} \tag{ 1 }$$

Where $\overline x_{\hbox{AI}}$ represents the average annual deforestation rate in the intervention area in the period since the intervention started, as a percentage of the total forest area in 2000; $\overline x _{\hbox{BI}}$ represents the average annual deforestation rate in the intervention area in the period from the start year of measurement (here: 2001) up until the intervention started, n_a and n_b the number of years in respectively the after and before period. A BA score of −2 thus means that the average annual deforestation rate in the intervention area decreased by 2% points when compared to pre-intervention years.

When including control areas in the assessment, the BACI score β is calculated as follows:

$$\begin{equation} \begin{array}{*{20}c} {\hbox{BA score }\beta } & = & {\hbox{(}\overline x _{\hbox{AI}} - \overline x _{\hbox{BI}} \hbox{)} - (\overline x _{\hbox{AC}} - \overline x _{\hbox{BC}} )} \\ {\hbox{with }x_{\hbox{AI}} } & = & {\frac{1}{{\hbox{n}_a }}\sum\limits_{i = 1}^{n_a } {x_i, \ldots,\hbox{etc}\hbox{.}} } \\ \end{array} \end{equation} \tag{ 2 }$$

Here, $\overline x _{\hbox{AC}}$ and $\overline x _{\hbox{BC}}$ represent the average annual deforestation rates in the control areas in the after and before period, respectively. β thus scores performance in the intervention area as compared to its control area. A negative β indicates a greater reduction or lower rise in deforestation in the intervention area than in the control area, and thus a positive REDD+ impact. We calculate the BACI scores β at both meso and micro levels (see next section and figure 3).

Zoom In Zoom Out Reset image size

2.4. Levels of analysis: initiative and villages

3.1. General results

Table 1 shows¹¹ the summary statistics of the main variables introduced in section 2.3.

The results of the BA α and BACI β performance scores were grouped into good, neutral and poor¹², where a good score means a relative reduction in tree cover loss over time (BA, BACI) and/or compared to the control area (BACI) (figure 4).

Zoom In Zoom Out Reset image size

First, we compare results from the two aggregation levels. At the meso (initiative) level, the median scores for both approaches (BA and BACI) are close to zero (table 1), meaning that there is no substantial change in deforestation rates between the two periods across the sample as a whole. At the micro (village) level, however, the scores are typically lower when compared to the results at meso level (i.e. better scores in terms of reduced deforestation rates)¹³. Apparently, the interventions thus had less impact at the more aggregated level. This finding could be due to interventions targeting only a few villages (including the ones studied here) within the site or within-site leakage from treated to untreated villages, which would lower the scores at the meso level.

Table 1. Summary statistics.

Level	Variable	Explanation	n	min.	max.	mean	median
both	start year	start year of the initiative	23	2006	2013	2009	2009
both	na	years in after period	23	2	9	6	6
both	nb	years in before period	23	5	12	8	8
meso	α	BA score (in initiative area)	23	−0.903	0.588	0.043	0.083
meso	β	BACI score	23	−1.184	0.315	−0.085	0.002
micro	α	BA score (in intervention villages)	16	−2.139	0.669	−0.271	0.048
micro	β	BACI score	16	−2.277	2.827	−0.449	−0.466

Second, we compare the two assessment methods. The BA scores (α) range from −2.139 (good performance) to 0.669 (poor) and the BACI scores (β) range from −2.277 (good) to 2.827 (poor). The BACI scores are typically lower than the BA scores at both meso and micro levels. Hence, the intervention areas tend to outperform the control areas, regardless of the overall trend in annual deforestation rates over time. Yet, median micro deforestation declines more in intervention than in control areas (median BACI score of −0.466), indicating slightly better REDD+ performance at lower aggregations. In turn, most good BACI scores at meso levels represent cases of increased deforestation trends though these increases were generally lower than in control areas.

3.2. Individual BA and BACI scores

To better understand the methodological differences, in this section we examine specific scenarios. Table 2 shows the occurrences of the prevailing factors that affect the BA and BACI scores, which we explain in more detail below.

Table 2. Occurrences per analysis level of factors affecting the BA and BACI scores.

Level	No. casesa	Bias in before periodb	Low absolute deforestationa	Peak yearsa	Outperforming control areab	Clear comparative performancea
Meso	23	7	9	16	1	5
Micro	16	2	8	13	1	1

^aRelevant for both BA & BACI. ^bRelevant for BACI only.

3.2.1. Bias in the before period

To confidently attribute changes (or lack thereof) to REDD+ activities in the after period, tree-cover loss patterns for intervention and control areas should have been similar in the before period (figure 2). Yet, two sample t-tests show that in five meso cases, and in two sites at both levels, significant differences in the before period influenced the resulting BACI scores (table D1). One such case is shown in figure 5 where meso-level before deforestation rates in the initiative area exceeded those in the corresponding control districts.

Zoom In Zoom Out Reset image size

3.2.2. Low absolute deforestation

For four meso-level cases, three micro cases, and five sites at both levels, median annual deforestation was less than 100 ha in absolute terms. Here, small year-to-year deviations in deforestation can determine the BA and BACI scores. Furthermore, many of these cases correspond to forest change maps where marked tree cover loss speckles may reflect degradation, climatic effects, or input data errors. We should thus be cautious in drawing conclusions from the corresponding scores, which might be driven more by tree cover data uncertainty than factual changes in deforestation dynamics.

3.2.3. Peak year

Single years of exceptionally high tree-cover loss (for intervention or control, before or after) can heavily influence our target variable of mean annual deforestation for BA and BACI scores alike.

A peak is defined as an observation above the upper quartile. A post-intervention peak might flag failure to target big driver(s) of deforestation, but could also have natural causes. A peak in the control area in the before period and a peak in the intervention area in the after period (and vice versa) can cancel each other out when having the same magnitude. Only seven meso-level cases and three micro-level cases showed no peaks in the intervention or control areas in the period 2001–2014. We checked the robustness of the BA and BACI scores by recalculating the scores without peak years and recorded the shifts from one category (good or poor) to the opposite (table 3, in bold). The majority of the scores do not shift categories (grey numbers). In one case (meso level, BACI approach), the performance score would change from good to poor if the peak years were excluded from the analysis.

Table 3. . Evaluating BA and BACI score robustness to peak year influence^a.

^a Bold numbers indicate highly sensitive cases where the particular score shifts from one extreme category (good or poor) to the opposite. Grey numbers indicate robust scores that are not influenced by the peak year.

3.2.4. Control area outperforms intervention area

Using the BACI method, good REDD+ performance can only be achieved if deforestation is reduced more in the intervention than in the control area(s). One meso-level (figure 6) and one micro-level case show good BA scores, but poor BACI scores, because control areas improved even more. In those cases, the slowdown in deforestation might have occurred even without the REDD+ intervention (e.g. due to commodity prices or national policies).

Zoom In Zoom Out Reset image size

3.2.5. Clear comparative performance scores

Clear comparative performance is defined as a score where we found no bias in the before period; no low absolute annual deforestation (median); and where the presence of peak years –if any– did not determine the category of the score. We found three meso level cases, three micro level cases and three sites at both levels with clear comparative performance scores (BA and BACI).

For these clear meso level scores, there were two with good, two with neutral, and two with poor BACI scores. In one site, deforestation increased in its corresponding control area, while deforestation decreased in the intervention area, yielding a good BACI score. One other site had poor BA, but good BACI scores, meaning that deforestation increased during the intervention phase, but less so than in control areas. Yet, arguably, it may be difficult to celebrate this latter case as a victory, since there was still more deforestation in the intervention area in the after period than before the REDD+ initiative started.

For the clear micro level scores, there were four with good, and two with poor BACI scores. At one site, deforestation decreased in the intervention area, while it increased in the control site, yielding a good BACI score. At another site, deforestation also decreased in the intervention area, while there was a less substantial decrease in the control area, resulting in another good BACI score. The other two good BACI scores represent cases where there was an increase of deforestation in the intervention areas, but less so than in the control areas. The two poor BACI scores represent cases of outperforming control areas similar to those explained in the previous section. That is, one denotes a case where deforestation increased in the intervention areas, while deforestation in the respective control areas increased less. The other is a site where deforestation decreased in the intervention villages (good BA score), but the decrease in the control villages was even stronger.

We applied BA and BACI approaches at meso and micro levels to assess subnational-level REDD+ performance. Both approaches and levels of measurement have advantages and disadvantages for effectiveness assessment (table 4). While the BA approach only considers trend shifts in local deforestation as an indicator for REDD+ performance, the BACI approach adds comparative performance in control areas. In principle, the BACI approach thus enables us to control for changes in deforestation that are unrelated to REDD+ interventions. Where BA measures the direction of change, BACI intends to measure attributive change. This approach, however, requires careful ex ante control site matching and selection. The high sensitivity of the results to matching procedures is clear from our results. At seven sites in the meso-level analysis, the jurisdiction used as the control area for the initiative had a significantly different pre-intervention deforestation rate compared with the initiative. Although meso-level assessment puts forest changes observed in the initiative area in a wider context, selecting a suitable control area (i.e. districts, region, or country) is not straightforward, since ideally these control areas should be subject to all of the same time-varying factors as the intervention areas.

Assessing performance at the micro level allows for more precise comparison between targeted and non-targeted villages. Yet, as the notion of village is not universal, delineating village boundaries can turn out to be a subjective process, and small (absolute) forest changes at the village level may wrongfully be interpreted as equivalent to large (absolute) forest changes at higher levels. Moreover, matching intervention and control villages is challenging. At two sites, in our micro-level analysis, baseline deforestation rates in the intervention villages and their control areas were significantly different, which resulted in uninformative BACI scores. For the village matching in GCS, our matched samples of intervention and control villages had statistically similar means across a range of characteristics as later measured in a village survey (Sills et al 2017). Still, the percent forest cover variable used in the matching was based on reported and not observed values, because global comparative satellite data for all sites was not available when the initial matching was performed in 2010. This choice clearly had implications for outcomes subsequently measured through the use of spatial data. Due to recent developments in the remote sensing domain, ex ante village matching could now be based on annual tree cover loss data from satellite data instead of reported forest cover loss from cost- and labour-intensive field studies. Although the BACI approach has strong analytical advantages, the sensitivity of results to control selection cannot be overstated.

Independent of approach, we found slightly better performance at the micro level compared to the meso level, possibly reflecting both a higher local treatment intensity, and more occurrence of confounding factors at higher scales, as well as leakage (relocated deforestation activities) from the intervention to control areas. Still, only four sites¹⁴ had both a good BACI score and were not influenced by factors like control area bias, low absolute deforestation and peak years.

The overall underwhelming performance of the studied initiatives could be due to a host of factors. First, performance scores are highly sensitive to cases with a late start year, and one could question how much REDD+ impact is reasonable to expect in the early years of initiative implementation. That is, multiple sites only had a couple of years of after observation. Furthermore, funding has been a major constraint for REDD+, meaning that interventions may not have been rolled out in the intensity originally planned (Sunderlin et al 2015). Short time spans combined with limited funding would naturally lead to less effective 'treatment', which may explain underperformance. Second, we did not consider forest degradation, which contributes to forest-based emissions considerably (Lambin et al 2003, Putz et al 2008, Nepstad et al 1999) and is the focus of REDD+ interventions at many sites (e.g. improved cooking stoves in Tanzania, sustainable forest management in Peru, etc.) (Sills et al 2014). While removals due to selective logging, undergrowth fires and fuelwood collection cannot yet be clearly detected by remote sensing based methods (Wertz-Kanounnikoff et al 2008), substantial progress has been made in recent years for measuring areas affected by forest degradation (De Sy et al 2012, GOFC-GOLD 2016). The dataset used in this study is unable to identify (reductions in) forest degradation, so any success regarding the second 'D' of REDD+ would have been missed here. Third, we only considered change in forest loss as proxy for the carbon impact of REDD+ and did not include forest gain, i.e. carbon stock enhancements that are integral to REDD+. Indeed, at several sites in the sample, restoration activities are a key part of the overall REDD+ strategy, but would also need more time to become significant and measurable. Finally, possibly the REDD+ proponents did not always effectively target the main driver(s) of deforestation at their sites, which may genuinely affect deforestation outcomes. For instance, most focus their efforts on smallholders, but sometimes these are not the main agents of deforestation, such as in some sites in Brazil and Indonesia (appendix 5 of Sills et al 2014, Sunderlin et al 2015). This prioritization of interventions targeting smallholders could also explain why we found slightly better results at the village than at the site level. However, as a general caveat, both BA and BACI methods work better with longer timeframes, and with before and after periods that are approximately equal. Future analysis is thus needed to understand the longer-term impacts of REDD+ at these sites and to better understand why impact varies across initiatives, taking into account the variation in both treatment and context.

Much early REDD+ progress has been through the implementation of subnational initiatives, yet we know very little about their carbon effectiveness. In this paper, we compared two approaches for assessing the effectiveness of 23 REDD+ initiatives in six countries through: (1) analysing trend development (BA approach); and (2) including control areas to correct for confounding factors (BACI approach).

We conclude that the more local the scale of performance assessment, the more relevant is the use of the BACI approach. Although BA is a good starting point for assessment, it is not able to distinguish between the REDD+ effect and confounding factors. BACI allows getting closer to attribution by removing the confounding influence of background dynamics, yet the results are only as good as the choice of control areas. While this remains a key challenge, new global forest datasets allow for improved control area matching and selection.

Nevertheless, there may be local situations where a BA approach, with its focus on the direction of change, is useful. For instance, in cases where BA scores flag poor and BACI scores good performance, due to increases in deforestation being higher in control areas than in intervention areas, the BA score makes clear that deforestation is still increasing, just less rapidly than would have occurred in the absence of REDD+. The poor BA score flags that the goal to reduce deforestation has become more distant (change has overall gone into the wrong direction); the good BACI score reflects that under a 'no intervention' counterfactual things would have been even worse (positive attribution). Conversely, in situations of generalized positive changes, BA scores alone risk painting a rosier picture than what could reasonably be attributed to the REDD+ intervention.

The BA and BACI assessment approaches used in our research both highlight overall minimal impact of REDD+ in reducing deforestation thus far. This could be due to the slow implementation of REDD+ interventions and low treatment density; proponents focussing primarily on smallholders instead of other important drivers; and/or our analytical focus on deforestation only, without examining degradation or reforestation. Furthermore, we did not examine specific REDD+ intervention mixes and strategies applied at different sites. To better understand what works (or not) in which contexts, linking the performance assessment results to the (types of) interventions would be an important next step.

Results-based payments for REDD+ will use conventional reference level approaches at the national level, yet there is clearly a need to understand the carbon effectiveness of local REDD+ interventions. Indications of which combinations of intervention mixes have shown to be more or less effective under variable contextual circumstances may provide valuable pointers for selective upscaling options to national REDD+ policies. Countries should seek ways to incorporate results from local level monitoring into their national reporting systems, since overall REDD+ impact depends on land use decisions on the ground.

This research is part of the Global Comparative Study (GCS) on REDD+ of the Center for International Forestry Research (CIFOR) (www.cifor.org/gcs) with funding support from the German International Climate Initiative, Norwegian Agency for Development Cooperation, Australian Department of Foreign Affairs and Trade, European Commission (EC), UK Department for International Development, and CGIAR Research Program on Forests, Trees and Agroforestry (FTA) Programme.

We would like to thank all CIFOR researchers and affiliates who helped defining, measuring and compiling village and initiative boundaries. We are grateful to Louis Verchot for helpful discussions throughout the process and thank two anonymous reviewers for their helpful comments.

Appendix A. Village boundary delineation

Appendix B. General results extended

Table B1. General results table extended.

Level	Variable	Explanation	n	min.	max.	mean	median
both	start year	start year of the initiative	23	2006	2013	2009	2009
both	na	years in after period	23	2	9	6	6
both	nb	years in before period	23	5	12	8	8
meso	$\overline x _{\hbox{AI}}$	Average annual deforestation rate in intervention area during after period	23	0.037	1.84	0.522	0.43
meso	$\overline x _{\hbox{BI}}$	Average annual deforestation rate in intervention area during before period	23	0.021	1.62	0.479	0.37
meso	$\overline x _{\hbox{AC}}$	Average annual deforestation rate in control area during after period	23	0.065	1.93	0.664	0.605
meso	$\overline x _{\hbox{BC}}$	Average annual deforestation rate in control area during before period	23	0.048	1.62	0.536	0.465
meso	α	Before-After score (in intervention area)	23	−0.903	0.588	0.043	0.083
meso	β	BACI score	23	−1.184	0.315	−0.089	−0.008
micro	$\overline x _{\hbox{AI}}$	Average annual deforestation rate in intervention area during after period	16	0.073	3.933	0.928	0.605
micro	$\overline x _{\hbox{BI}}$	Average annual deforestation rate in intervention area during before period	16	0.068	4.514	1.199	0.489
micro	$\overline x _{\hbox{AC}}$	Average annual deforestation rate in control area during after period	16	0.106	2.479	1.023	0.862
micro	$\overline x _{\hbox{BC}}$	Average annual deforestation rate in control area during before period	16	0.073	4.993	0.845	0.486
micro	α	Before-After score (in intervention area)	16	−2.139	0.669	−0.271	0.048
micro	β	BACI score	16	−2.277	2.827	−0.449	−0.466

Appendix C. BA and BACI classified scores for intensive sites only

Figure C1 reports results at both the meso and micro level for the 16 'intensive' sites only, which as described in section 2.4 include both intervention and matched control villages. These results are mostly consistent with the results presented in figure 4, confirming our finding (presented in section 3.1) that performance generally looks better at the micro than at the meso level (i.e. evaluating REDD+ at the micro level makes it appear more effective in terms of reducing deforestation). Figure C1 confirms that this finding is not due to the difference in sample size for the meso and micro level analysis reported in figure 4.

Zoom In Zoom Out Reset image size

Appendix D. Test results for bias detection