Improving uncertainty in forest carbon accounting for REDD+ mitigation efforts

Forests are a critical part of the solution to address the global climate crisis (IPCC 2018, 2019) because of their natural ability to remove carbon dioxide from the atmosphere (Turner et al 2009, Lewis et al 2019) and the magnitude of carbon dioxide emissions from deforestation and forest degradation (van der Werf et al 2009, Baccini et al 2012, Pearson et al 2017). One strategy for decreasing carbon emissions and enhancing forest carbon sinks is to provide developing countries with results-based financial incentives to reduce deforestation. Nearly two billion dollars are currently committed to such programs. Unfortunately, forest carbon accounting is fraught with uncertainties, making it challenging to evaluate carbon mitigation efforts in the forest sector (Pan et al 2011). Countries participating in these incentive programs must document the uncertainty in their estimates along with their claimed emission reductions or forest sequestration relative to a baseline (Pelletier et al 2013, FCPF 2016; Green Climate Fund 2017, BioCarbon Fund 2020). This paper highlights common problems and presents solutions to strengthen uncertainty estimates in forest carbon accounting. Uncertainty should not be seen as a handicap; it should be embraced as an important element of tracking progress on addressing climate change.

The strategy to provide results-based payments for reducing forest carbon emissions and enhancing forest carbon sinks was agreed upon in a series of decisions under the United Nations Framework Convention on Climate Change (UNFCCC), and is referred to as REDD+ (Reducing Emissions from Deforestation and forest Degradation, conservation, sustainable management of forests, and enhancement of forest carbon stocks) (Houghton et al 2010). Currently, there are three multilateral programs—largely donor government funded—that offer such financial incentives to developing countries, including the Forest Carbon Partnership Facility's (FCPF) Carbon Fund (around $900 million committed), the BioCarbon Fund Initiative for Sustainable Forest Landscapes (ISFL, $360 million) and the Green Climate Fund's REDD+ Results-based Payments Pilot Programme ($500 million). In addition, several donor governments have provided or committed results-based payments to countries on a bilateral basis (over $2 billion for Brazil and $1 billion for Indonesia) or through the REDD Early Movers program (which has disbursed EUR 234 million).

Countries seeking payments for forest-related mitigation efforts must establish reference levels to use as benchmarks for assessing REDD+ performance. Currently, most countries focus on deforestation, commonly estimating emissions based on carbon stocks per unit area and the area of land changing from forest to non-forest (FAO 2019). Some countries also estimate emissions from forest degradation, and some are estimating carbon sequestration from reforestation or forest management. Forest reference levels are usually set by calculating an historical average level of emissions (Maniatis et al 2019) and using this as a proxy for expected future emissions. To date, 50 countries have submitted forest reference levels to the United Nations Framework Convention on Climate Change (2020).

To receive results-based payments, countries must report how they reduced emissions below their forest reference level. To avoid overcrediting, a conservative estimate would use values or assumptions that are more likely to underestimate, rather than overestimate, emission reductions (WRI & WBCSD 2005). Uncertainties in estimates of net emissions in the forest sector can be quite large, making it difficult for countries to demonstrate performance statistically distinguishable from the reference level (Grassi et al 2008, Köhl et al 2009, FAO 2017). Most results-based payment programs include procedures that reduce payments to account for uncertainty in emissions estimates (i.e. payments are made for a portion of estimated emission reductions) (Angelsen 2017; appendix A). However, to maintain incentives for countries to continue mitigation efforts while promoting honest reporting, such programs will make payments even when uncertainties are large (figure 1). In some cases, the goal of the penalties is simply to encourage efforts to reduce uncertainties.

Zoom In Zoom Out Reset image size

The forest reference levels submitted to date report many uncertainties that are too low to be credible (FAO 2019). These low uncertainty estimates could be a response to the incentive system or they could be due to confusion on how to calculate uncertainty correctly. Regardless, improving the accuracy and transparency of uncertainty estimates of forest reference levels and the reporting of results (Birigazzi et al 2019) is critical not only for the Paris Agreement's global stocktake, but also for the efficient use of scarce financial resources and, importantly, if these credits enter into carbon markets, to ensure that any such offset is real.

2.1. Omitting major sources of uncertainty

2.2. Mistakes in combining uncertainties

Combining multiple sources of uncertainty to provide a single estimate, as is needed for country-level carbon accounting, can be challenging (JCGM/WG1 2008, Couto et al 2013), in part because there are so many ways to make mistakes (Pappenberger and Beven P2006, Birdsey et al 2013).

One way to combine the effects of uncertain inputs is to randomly sample from their assumed distributions. In this approach, called Monte Carlo simulation (for its similarity to gambling), a carbon accounting calculation is iterated hundreds or thousands of times, with the inputs varying randomly to mimic the uncertainties in their values (Metropolis 1987). The distribution of the resulting hundreds or thousands of outputs reflects the net effects of the input uncertainties.

A common mistake in interpreting this output is to report the uncertainty in the mean or median of that distribution as an indicator of the uncertainty in the output (e.g. Mcmurray et al 2017). It is not, for the simple reason that in the real world, we will not get all the Monte Carlo trials; we will only get one of them, and we do not know which one. Each trial has the same chance as all the other trials of corresponding to reality, as far as we know. This mistake is a big one, typically underreporting uncertainty by a factor of 100, because calculating uncertainty in the central tendency (e.g. the standard error of the mean) involves dividing the standard deviation by the square root of the number of 'observations,' which is commonly 10 000 trials.

An example of this mistake is shown in figure 2, which is based on a report submitted to the FCPF in 2019. The uncertainty reported to the FCPF should have been 8.9 Mt yr⁻¹, the range of the middle 90% of the distribution of Monte Carlo estimates. Instead, the uncertainty in the median of the Monte Carlo estimates was reported, which was very small (0.11 Mt yr⁻¹), because the number of estimates was very large (10 000). To accurately characterize the distribution, it is important to make a large number of estimates, but making more estimates does not diminish the uncertainty. The confidence in the best estimate (sometimes reported as the standard error of the mean, or in this case the 90% confidence interval around the median, obtained by sampling from the estimates) could be made arbitrarily small by increasing the number of Monte Carlo iterations, but the 90% confidence interval of the increased number of estimates would remain just as wide.

Zoom In Zoom Out Reset image size

Using the Monte Carlo approach requires defining the distributions of the inputs; commonly, in the absence of information to the contrary, normal distributions are used, based on the standard deviation of the observations, which may not be realistic. An alternative to describing the distribution of the inputs is to resample the data. This approach requires no assumption of a distribution and is thus most true to the measured population. The drawback to this approach is that the representation of the population is only as good as the data, and if the data set is small, it may not accurately capture the range of potential values.

An analytical approach to combining uncertainty sources is easier to implement and avoids the mistake of reporting the wrong property of a distribution of Monte Carlo trials. For example, when adding uncertain estimates, the uncertainty in the sum can be calculated as the so-called 'sum in quadrature' of the uncertainty in the inputs, a formula that can be traced back to Gauss (1823). The uncertainty in a sum can be estimated by squaring the uncertainties of the component quantities, adding them together, and taking the square root of the sum—as long as the uncertainty sources are independent (sometimes they are not; see below). This approach, referred to as Approach 1 in the IPCC guidelines (IPCC 2006), was commonly used in early submissions to the UNFCCC. Since 2013, the Monte Carlo approach (Approach 2 in IPCC parlance) has been required for participation in the FCPF. Using multiple approaches to combine uncertainties could be a good strategy for identifying some of these common mistakes.

2.2.1. Independent vs. shared sources of uncertainty in space

For both the analytical approach and the Monte Carlo approach to combining sources of uncertainty, it is easier to treat the various sources as independent than to account for the relationships among them. Misrepresenting these relationships, either by assuming that uncertainties are independent when they are shared, or assuming that they are shared when they are independent, is another common source of mistakes in uncertainty accounting.

When combining uncertainties from multiple land areas, treating all uncertainty sources as independent can result in very low combined uncertainty. For example, in 2015, one country reported a reference level for carbon emissions with an uncertainty of only 1.5%. This calculation involved 18 forest types, with uncertainties based only on sampling error, which averaged 19% (and ranged from 2% to 92%). Combining values that range from 2 to 92 to obtain a summary of 1.5 seems counterintuitive, because we are more familiar with averaging observations than uncertainties. A weighted average of the carbon emissions would lie between the highest and lowest values. But the uncertainties in carbon emissions, if independent, do not add directly—they sum in quadrature—because any estimate could be either too high or too low, and combining opposing errors can give an answer closer to the truth. However, if the uncertainties were shared, over- and under-estimates would coincide, and they would sum normally.

Another report submitted in 2019 used a national-scale sampling error of 10%, assigned this to all the forest types, and then combined them as if they were independent. If there were two evenly distributed forest types, this would give a combined uncertainty of 7% (using analytical uncertainty propagation, the square root of the sum of squared uncertainties in this case is $\surd {{{\left(( {10\% \times 0.5} \right)}^2} + {{\left( {10\% \times 0.5} \right)}^2)}}$; if there were 20 forest types, each 5% of the land area, the combined uncertainty would be 2.2% $\surd({20 \times {{( {10\% \times 0.05})}^2}})$, and so on, down to vanishingly small uncertainties with very large numbers of forest types. Thus it appears to be advantageous for uncertainty reporting to consider as many forest types as possible.

However, these reports of small uncertainties, although correct for sampling error alone, fail to account for many other sources of uncertainty. Those sources that are shared are not diminished by dividing up the landscape. For example, the belowground biomass of forests is often estimated from aboveground biomass, using a ratio (root:shoot) measured at another location. Measurements of root biomass are rare, because they are costly and destructive; thus, the same root:shoot ratio is commonly used across many forest types, because the true root:shoot ratios are unknown. Similarly, although forest inventory data are collected independently across forest types, they are usually converted to carbon stores using common allometric relationships, tissue density, and carbon fraction. These sources of uncertainty, which may amount to 5%–10%, are not diminished by applying them to multiple forest types (Martin and Thomas 2011, van Breugel et al 2011). To the degree that they are in error, they are in error in the same direction in every instance, and for this reason it is a mistake to combine them as if they were independent (Hill et al 2013).

The two examples described above did not incorrectly combine uncertainties, they merely omitted sources that should have been treated as shared. Summing in quadrature is incorrect when used to combine shared uncertainties, as illustrated in a 2017 report, in which carbon emissions of 19% from the area in rainforest and 24% from the area in mesophytic forest were combined to give a country-level uncertainty of 15%. Summing in quadrature was not appropriate for the sources that were shared, which included allometry, tissue density, and carbon fraction. Rather than using addition in quadrature, they should be combined with proper attention to correlation among the sources, which can be done analytically (JCGM/WG1 2008, Kirchner 2020). Monte Carlo sampling of shared errors, such as tree allometry, can be done correctly by assigning the same error to all trees at each Monte Carlo trial (Yanai et al 2010). It is easy to make the mistake of assigning error independently for each observation, which results in unrealistically small errors at large spatial scales.

2.2.2. Independent and shared sources of uncertainty in change over time

While shared sources of uncertainty in assessing carbon stocks over multiple forest types in a country give higher uncertainty than if the sources were independent, they reduce uncertainties in change over time, compared to independent sources of uncertainty. If, for example, the same root:shoot ratio is used in two successive carbon inventories, an error in that ratio will lead to similar errors in both inventories and thus to relatively small uncertainty in the net change between them. Changes over time in carbon stocks and carbon emissions are what matter to mitigating climate change (Birdsey et al 2013).

Similarly, revisiting the same plots over time to assess forest biomass gives a more precise estimate of change over time, by reducing the effects of plot-to-plot variability (Magnussen et al 2014, Mcroberts et al 2018a). Every plot on the landscape is different; thus, if we conducted a new sample each time, this source of variation (sampling error) would be independent at each time. The value of resampling permanent forest inventory plots is illustrated with data from the US Forest Service Forest Inventory and Analysis (FIA) Program (Bechtold and Patterson 2005) for the state of Minnesota, USA (figure 3). A pairwise comparison of estimates has about half the uncertainty (0.8 Mg C/ha) of an unpaired comparison (1.5 Mg C/ha).

Zoom In Zoom Out Reset image size

Other examples of correlated error over time, besides sampling error, include allometric models. Allometric relationships are uncertain (Chave et al 2004, Breidenbach et al 2014), as are conversion factors such as wood density and root-to-shoot ratio. However, although these parameters are not perfectly known, if they are incorrect at both time periods in the same way, they do not detract as much from our confidence in change over time as if they were incorrect in different ways at the two time periods (Yanai et al 2012). For this reason, it is important to use the same models and methods of calculation over time (Picard et al 2015b). There is also uncertainty in determining the land area undergoing transitions such as deforestation or degradation, and these uncertainties are less likely to be correlated over time than the factors that contribute to estimates of carbon per unit area. When maps are used to compare rates of change between two periods, there can be correlations in inaccuracies in classifying land cover change, which are commonly ignored (Muchoney and Strahler 2002, Pontius Jr and Lippitt 2006, Sexton et al 2015). Again, accounting for these correlations would reduce the combined uncertainties.

2.3. Other common mistakes

When the first reference levels were submitted to the UNFCCC in 2014, it was acceptable to submit estimates with no quantification of uncertainty (figure 4). Since then, standards of uncertainty reporting have come a long way. The FCPF's Carbon Fund requires countries to identify sources of uncertainty, quantify their contribution to overall uncertainty, and estimate the uncertainty of their reported emissions reductions. Starting in 2019, the Green Climate Fund, which uses UNFCCC submissions as a basis to determine the amount of payments to countries for REDD+ results, has required countries to provide uncertainty estimates in forest reference levels and emission reductions to be eligible to receive payments (Green Climate Fund 2017). More broadly, there is a move towards greater overall transparency, with calculations of reference levels and their uncertainties made available to review panels and, sometimes, also made publicly available. This transparency makes it possible to identify mistakes in uncertainty accounting and recommend improvements.

Zoom In Zoom Out Reset image size

In fact, we found considerable room for improvement. As of April 2020, our examination of 60 forest reference level submissions, including resubmissions, to the UNFCCC (figure 5) and 18 Emission Reductions Program Documents to the FCPF (which contain proposed reference levels) found that very few correctly combined the uncertainty of individual components to estimate overall uncertainty. There were 19 submissions that combined uncertainties by summing in quadrature, commonly reporting a total error smaller than the individual component errors. This result would be correct if the errors of each component were independent, but often they are not, as described above. At least five countries incorrectly reported the uncertainty in the mean or median of Monte Carlo estimates, rather than the dispersion of the estimates (figure 2), resulting in reported uncertainties as low as 0.1%.

Zoom In Zoom Out Reset image size

Although countries have made efforts to discuss sources of uncertainty and account for some of them, many sources are still omitted, which effectively assigns them, incorrectly, zero uncertainty. Of the 18 countries that have submitted reference levels to the FCPF Carbon Fund, none of them reported measurement uncertainty in land-use change (deforestation, degradation, afforestation), which, admittedly, is not straightforward to quantify (Mcroberts et al 2018b). Some countries (7) quantify at least some sources of measurement uncertainty in forest carbon density (carbon per unit area). The most widely quantified source of uncertainty is spatial sampling error, with all but 1 country reporting this for both land-use change and forest carbon density. Ten countries accounted for uncertainty in allometric models and carbon concentrations and 14 countries accounted for uncertainty in belowground biomass estimated using root-to-shoot ratios. Finally, describing the procedures used to assure the quality of the data (QA/QC) is accepted in lieu of reporting measurement uncertainty: 14 countries took this approach for forest carbon density and 7 for land-use change. Unfortunately, meeting this requirement does not ensure that the QA/QC data are used to improve measurements; many countries collect the information necessary to quantify measurement uncertainty but have not analyzed the results.

We applaud the increasing effort devoted to uncertainty reporting, thanks in part to programs that encourage or require uncertainties to be reported. As uncertainties are more correctly and consistently calculated, payments will be better justified in terms of performance and more fair in terms of penalties taken (figure 1).

There is still room for improvement in the reporting of uncertainty. Uncertainties are often reported without adequate discussion of the sources or recognition of bias in the estimates, and the calculations themselves are not always completely documented. Additionally, decisions are made to ignore data that would contribute high uncertainty without describing and justifying these decisions—or the justification given is that the uncertainties are high.

The benefits of repeated forest inventories, discussed above (figure 3), require accurate location of plots on the ground. Many countries have yet to establish repeatable forest inventories, and others are encountering difficulties with implementation, often arising from a lack of funding. Some countries have relied on GPS coordinates to identify plots, rather than permanently marking them, resulting in poor repeatability. Other countries have found their plot markers to be less than permanent, especially in heavily populated areas. Tradeoffs among multiple inventory objectives and increasing costs may result in decisions that increase uncertainties. For example, FAO has suggested that permanent transect plots 250 m in length would produce better estimates of biodiversity and land use (Saket et al 2010), but such long transects marked at only one end have resulted in poor overlap in remeasured areas. Finally, even well marked plots can be impossible to revisit due to ecological or political conditions. Currently, few of the countries eligible for REDD+ payments have completed more than one inventory of permanent plots, but more such continuous forest monitoring systems are being developed.

Clearly, there is a need for better understanding and implementation of error quantification, especially error propagation (Chave et al 2004, Yanai et al 2010, Magnussen et al 2014). Guidelines should be more specific as to how to obtain uncertainties from Monte Carlo simulations; the IPCC chapter on uncertainties (IPCC 2006, section 3.2.3.2) specifies how to conduct a Monte Carlo analysis but not how to interpret the results, which has allowed the mistake documented in figure 2. The US Forest Service International Programs is currently supporting efforts to provide guidance and tools for correctly calculating and reporting uncertainties through QUERCA (Quantifying Uncertainty Estimates and Risk for Carbon Accounting). Conducting a survey of authors of REDD+ reports will identify obstacles to properly estimating uncertainty; trainings and workshops will help to build capacity, avoid mistakes in quantifying uncertainty, and enable better decision-making to improve monitoring and ultimately reduce uncertainty.

REDD+ uncertainty reporting to date has not always received adequate scrutiny. The technical review of submitted forest reference levels should be more rigorous. The reviewers may lack knowledge of the typical levels of uncertainties that are realistic, as they have often accepted unrealistically small uncertainties without comment. The review process is more prescriptive for the FCPF Carbon Fund than for the UNFCCC; for example, the UNFCCC technical assessment provides only a list of 'areas of improvement' whereas the FCPF Carbon Fund can require that certain standards be met before a country can proceed in the pipeline to receive payments.

Quantifying uncertainty should be embraced, not avoided. There are many benefits of uncertainty analysis besides meeting the requirements for reporting on carbon mitigation efforts. Understanding the effect of multiple uncertainty sources on any estimate helps identify where best to direct monitoring effort towards improving confidence in those estimates. For example, an examination of uncertainty in quantifying carbon in coarse woody debris in the US FIA program revealed that estimates of wood density were the greatest source of uncertainty, and thus investments in better measurements of wood density would have the greatest payoff in terms of improved estimates of this carbon stock (Campbell et al 2019). In this vein, many submissions to the FCPF have identified areas for improvement, based on sensitivity analyses and an assessment of the main sources of uncertainty, and defined work plans to address them prior to the first monitoring period (FCPF 2019). Costs as well as benefits may be considered; the cost of reducing uncertainty through more intensive monitoring may exceed the benefits in carbon credit values (Köhl et al 2020). Smarter monitoring, such as accurate location of plots, might be achieved at lower cost than adding more plots. Fears that uncertainties would prevent payments for emission reductions (Köhl et al 2009) except in the case of very high deforestation (Plugge et al 2012) have been assuaged by programs allowing high uncertainties with small payment discounts (appendix A).

International REDD+ payment programs have spurred investment in building the capacity of countries to inventory, monitor, and report on the extent and state of their forests and changes in carbon storage and sequestration. This investment in capacity building is paying off (Romijn et al 2015, Neeff and Piazza 2019). More and more countries are including uncertainty in their reference level submissions (figure 4) and using more sophisticated methods for error propagation. These improvements provide for better informed forest management as well as improved confidence in the efficacy of results-based payments.

To date, REDD+ results-based finance has largely been through donor government payments to developing countries. However, in the past two years there has been increasing interest by the private sector in purchasing 'nature-based' carbon offsets. If these credits are to be traded for fossil fuel emissions, they must represent real and robustly quantified carbon offsets. While there are other characteristics of forest carbon credits that must be assured—such as their permanence, avoidance of leakage, and credibility of baseline selection (Chagas et al 2020) —correctly quantifying the uncertainty of emission reductions is essential to safeguarding the environmental integrity of these assets. Investments now in better uncertainty quantification will allow for more targeted financing in the future to improve emission reductions and to protect the global environment.

Rabia Munsaf Khan, Joe Nash, Lalita Adhikari and Thomas Mann reviewed REDD+ submissions to UNFCCC and FCPF. Hunter Stanke helped us use rFIA to obtain FIA data. Till Neeff and an anonymous reviewer provided helpful feedback on the manuscript. This paper is a product of QUEST (Quantifying Uncertainty in Ecosystem Studies) and QUERCA (Quantifying Uncertainty Estimates and Risk for Carbon Accounting). QUERCA is funded by the US Department of State and US Agency for International Development. Please visit our website at https://www.quantifyinguncertainty.org for papers, sample code, presentations, tutorials, and discussion.

No new data were created or analysed in this study.