Errors and uncertainties associated with the use of unconventional activity data for estimating CO₂ emissions: the case for traffic emissions in Japan

We estimated monthly traffic CO₂ emissions using fuel consumption data for the first six months of 2020 and all of 2019 (total 18 months). We used the monthly fuel consumption data for automobile use collected by the Ministry of Land, Infrastructure, Transport and Tourism (MLIT) (MLIT 2021). The monthly fuel consumption data are reported for four fuel categories, such as gasoline, diesel, liquified propane gas, and liquified natural gas. The automobile road transport emissions accounts for approximately 90% of the total transportation sector (IPCC sector code: 1A3) emissions in Japan (GIO and MOE 2020). Following the IPCC guidelines (IPCC 2006), we calculated monthly traffic CO₂ emissions as follows:

$$\begin{equation}{\text{Emission}} = \sum\limits_a {\left[ {{\text{Fue}}{{\text{l}}_a} \times {\text{E}}{{\text{F}}_a}} \right]} \end{equation} \tag{ 2 }$$

where Fuel is the amount of the fuel (fuel type a) consumed, and EF_a is the emission factor for the fuel type a. We used country-specific EFs provided by the Ministry of the Environment, Japan (see values presented in table S1 in supplement information (available online at stacks.iop.org/ERL/16/084058/mmedia)). From a methodological point of view, our estimates can be considered to be the best estimates possible since they use the official fuel statistical data and country-specific EF values (Tier 2 emission estimates in the IPCC definition). Thus, our estimates serve as the truth in this study when other estimates are examined, as well as a reference to allow errors and uncertainties to be calculated in terms of deviations from our estimates.

While the usage of the UAD in the recent studies, such as Le Quéré et al (2020a) and Liu et al (2020a, 2020b), are not exactly the same, the basic assumption in those studies is that changes in the activity levels are proportional to the emissions. The estimation can be done as follows:

$$\begin{equation}{\text{Emission}}\left( t \right) = {\text{ UAD}}\left( t \right) \times {E_{{\text{ref}}}}.\end{equation} \tag{ 3 }$$

The emissions in the recent studies are estimated by scaling the reference (or baseline) emission (E_ref) using the relative change in UAD at time t. We obtained emission values by scaling our January fuel-based emission estimate using monthly relative changes indicated by UAD, as shown in the equation (3). We collected two UAD that have been used in the previous publications (e.g. Forster et al 2020, Le Quéré et al 2020a), such as Apple's Mobility Trends Reports (Apple Inc. 2021) and Google's COVID-19 Community Mobility Reports (Google LLC 2021), as well as actual traffic count data.

Apple's Mobility Trends Report (hereafter, Apple data) is based on data sent from users' devices to the Maps app service. Apple data is published on a daily basis and reports daily changes in requests for directions on the Maps app by three transportation types (driving, transit, and walking) for several spatial levels, such as countries/regions, sub-regions and cities (Apple Inc. 2021). The values were normalized by the value on the day 13 of January 2020. We used values reported for driving in Japan. Day in the Apple data is defined as midnight-to-midnight, Pacific time. However, we used the values as reported without any adjustment. The daily values before the day 13 of January (baseline) were assumed to be the same as the baseline (value = 100, which means no changes from the baseline). The values for the 11th and 12th of May, which were missing, were set as an average of the values for the 10th and 13th of May.

Google's COVID-19 Community Mobility Reports (hereafter, Google data) are similar to the Apple data, but intend to show how people's movements change compared to a baseline (Google LLC 2021). The baseline was defined as the median values for the corresponding day of the week, during 3 January to 6 February 2020 (5 week period). The mobility trends are reported for six categories, such as grocery and pharmacy, parks, transit stations, retail and recreation, residential, and workplaces. Following Forster et al (2020), we used values reported for the transit stations category. We are aware that the Google data has been updated over the past year. Thus, the values used in this study might not be exactly the same as ones used in Forster et al (2020).

The traffic count data we used in this study were collected from a nation-wide automated system. The raw traffic measurement (count) data were being collected at approximately 39k locations (an average of our study period) at a 5 min interval and compiled by Japan's National Police Agency. The data are provided through the Japan Road Traffic Information Center (JARTIC) (JARTIC 2021). We calculated the national monthly total traffic counts for the first six months of 2020 and the entire year of 2019 (total 18 months). We then used the relative changes from January to scale our January fuel-based estimate. We also used traffic count data for two additional purposes in this study: (a) to evaluate the performance of Apple and Google data as a proxy for traffic count and (b) examine the performance of traffic data as an estimator of CO₂ emissions (see section 2.3).

We also collected the recent near-real time estimates made by Le Quéré et al (2020a) and Liu et al (2020a) and included them in the emission comparison/evaluation in this study. The recent studies established a relationship between Apple Mobility data and/or TomTom data and traffic emissions, and estimated daily emissions in 2020. In this study, the original daily values were aggregated to monthly levels, and scaled using our fuel-based January value in order to focus on emission relative changes. Le Quéré et al (2020a) included the international shipping emissions in their 'surface transport' sector. However, since the international shipping emissions are only a few percent of Japan's national total emissions, the relative changes should be largely driven by emissions from traffic. A summary of emission estimates compared in this study is shown in table 1.

The percent uncertainty U associated with the emission estimate from equation (1) can be calculated as

$$\begin{equation}U = { }\sqrt {U_{{\text{AD}}}^2 + U_{{\text{EF}}}^2} \end{equation} \tag{ 4 }$$

where U_AD is the percent uncertainty for AD and U_EF is the percent uncertainty for the EF (IPCC 2006). Using the reported uncertainty estimates for the fuel data and EF (5%, 2 sigma for both), the uncertainty for our emission estimates is calculated as 7% (2σ).

Similarly, the uncertainty of the UAD estimates could be calculated in the way as seen in equation (4) as a combination of the percent uncertainties by replacing the U_EF with the uncertainty estimates of the reference emissions U_Ref

$$\begin{equation}U = { }\sqrt {U_{{\text{UAD}}}^2 + U_{{\text{Ref}}}^2} .\end{equation} \tag{ 5 }$$

U_Ref could be assessed using the uncertainty estimates provided for the original estimates. If U_Ref is obtained by disaggregating original estimates in time or space, one might need to add the associated disaggregation uncertainty and/or error (Oda et al 2015, 2019). As described in section 2.2, our UAD-based estimates share E_Ref, which is our January fuel-based estimate, and thus, we focus on the assessment of U_UAD. Another reason that we focus on the assessment of U_UAD is because E_Ref in equation (3) is often subject to systematic errors due to revisions to the underlying statistical data (Andres et al 2014, Marland et al 2009) and/or errors in them (Guan et al 2012). E_Ref will be updated when the new statistical data become available, while UAD is likely to remain the same. Such systematic errors are not often explicitly included in the common uncertainty. An example of an exception is the assessment done by Andres et al (2014) for the global total emission estimates. The U_UAD is often not directly measurable and any alternative data to serve as truth does not exist; the nature of proxy makes it more difficult to evaluate (Oda et al 2019). However, in this study, we could attempt to evaluate U_UAD by comparing UAD (Apple and Google data) to traffic data as the UAD was used as a proxy for traffic under the assumption that traffic volume is proportional to emission changes.

We also identify uncertainties that are not captured in equation (5), namely uncertainties associated with the emission calculation (or model). Such uncertainties include (a) conceptualization uncertainty and (b) model uncertainty (IPCC 2006). Following the suggestions from the IPCC guidelines, we will examine those two uncertainties, which are often poorly characterized or may not be characterized at all, as seen in the recent near-real time estimates. We should be able to approach the uncertainties using our fuel-based estimates as a reference in combination with the traffic data. For example, the conceptualization uncertainty could be assessed by comparing the traffic-based estimates to the fuel-based estimates and the uncertainty should show up as the differences. The traffic-based estimates can be viewed as the case where UAD is the perfect estimator of traffic data. The model uncertainty due to the incomplete model representation could be examined by comparing UAD to traffic data, as described earlier. Errors and uncertainties associated with the mismatch of the system boundary in the calculation and spatial and temporal representativeness of UAD are difficult to clearly define, but should be captured in this assessment.

The sources of errors and uncertainties discussed here are challenging to disentangle and assess and provide statistically meaningful error and uncertainty estimates individually. This study attempts to assess them where possible. We also acknowledge there is no perfect single metric to show these degrees of errors and uncertainties, and thus we calculate and provide multiple metrics. The set of assessments we deliver in this study essentially corresponds to the QA/QC and Verification activities suggested by the IPCC guidelines as a good practice (IPCC 2006). The IPCC guidelines suggest that these assessment activities could happen not only after obtaining the emissions, but also during the emission development process. By doing so, one could obtain robust estimates by capturing error/uncertainty sources as much as possible and potentially mitigate them where possible.

Our monthly emission estimates are shown in figure 1 (2020 as blue solid line with dots, and 2019 dashed grey line, the calculated emission values are shown in table S2 in supplement information). A summary of the emission comparisons is shown in table 2. Our fuel-based calculation shows that the total traffic CO₂ emission for the first six months of 2020 was 80.6 MtCO₂, which was 11.4% lower than the total emissions from the same six month period in 2019 (90.9 MtCO₂). While the 2020 emissions started at the same level as the 2019 January emission (the difference was only 1.5%), the 2020 emissions started deviating from the 2019 reference values in March, and showed significant decline in April and May when Japan's state of emergency was in place (7 April–6 May, and then extended to 27 May). Japan's state of emergency did not impose a physical lockdown for identified severe areas (seven prefectures), and the core of Japan's approach has been to prevent the spread of the pandemic by avoiding the 'three Cs: closed spaces, crowded places, and close-contact settings' (Government of Japan 2020) with a target of reducing the contact by 70%–80% (Prime Minister of Japan and His Cabinet 2020c). Japan's government asked their citizens and businesses to reduce the activity level, while maintaining the necessary businesses thorough citizens' efforts rather than forcing them by penalty or fines (Prime Minister of Japan and His Cabinet 2020c). Essentially, the measures to decrease the spread of the virus were voluntary. Thus, Japan's approach to the COVID-19 pandemic has been considered to be a soft approach compared to ones taken in many other countries. Its performance has been analyzed and discussed (e.g. Feder 2020, Gordon 2020, Nishimura 2020, Normile 2020, Wingfiel-Hayes 2020). In light of the 'soft' approach, the emission reduction confirms that the citizens and businesses reduced the level of their economic activity in response to the request under the state of emergency. The mean emission reduction during the two months was 23.8% relative to the 2019 level.

Zoom In Zoom Out Reset image size

So how well was the emission reduction captured by the UAD-based approaches? Figure 1 also compares several values derived using UAD, such as traffic count (green), Apple data (pink), and Google data (dark green) and evaluates the performance of UAD for estimating emissions. In the figure, the errors due to the use of UAD are manifested as a deviation from our fuel-based approach. Table 2 shows several calculated metrics, such as R², bias, and mean absolute error (MAE). Here, the traffic-based emissions also can help us to examine the performance of Apple and Google data as a proxy for traffic volume or estimator of CO₂ emissions. Apple and Google data are used in the recent publications and thus this comparison should allow us to characterize the recent estimates in terms of the use of UAD, while our calculations do not fully replicate their daily values precisely. As described in the section 2, all the monthly values were obtained by scaling the same January fuel-based emission estimate. Thus, we can focus solely on the relative changes estimated by different UADs.

We confirmed that all the UAD indicated the decrease in the activity, thus the resulting emissions decreased towards the period of the state of emergency and started recovering in June. However, our comparison shows the emission seasonalities derived from different UADs can vary by a large degree (R² = 0.54–0.93; Bias = −11.2%–11.0%; MAE = 10.6%–12.7%, see table 2). For example, the traffic count data and Apple data seem to be systematically overestimating the emissions in comparison to our fuel-based estimates. Both cases show an increase in February, especially in Apple data. The traffic volume in January is typically lower than other months, due to the significantly lower traffic volume during the New Year time in Japan (23% lower than the day 13 reference level in 2020). We speculate that the emission increase from January to February could be partially explained by the low traffic volume in January, while the traffic volume should have returned to normal level by the reference day for the Apple (13 January). Also, as noted by Apple Inc. (2021), the relative volume increase since 13 January is consistent with their normal, seasonal usage of the Apple Maps app in many countries/regions, sub-regions, and cities. While we are unable to identify the reason, these could show up as a significant overestimation in February and March, which is a good characteristic as a proxy for traffic (R² = 0.75, see table S2 in supplemental information), but not for CO₂ (R² = 0.54).

The emission monthly changes from the Google data are closer to our fuel estimates (R² = 0.93) than that from the Apple data (R² = 0.54), and did not have the overestimation seen in the traffic-based estimates and Apple data-based estimates in February. The Google-based emission monthly changes also correctly indicated the start of the emission decline in March, but significantly overestimated the emission reduction (by 9.9%). Just by looking at the values reported for the other five categories (see figure S2 in supplemental information), the 'workplaces' or 'retail and recreation,' or the average of them could be an excellent estimator of CO₂. On the other hand, 'grocery and pharmacy' and 'parks' seem to be in better agreement with the monthly traffic changes. Adequate information to understand and explain this is not available for evaluation due to the nature of the data provided (privacy policy), this shows a challenge of using UAD as a proxy, and the need for performance evaluation. While the choice of 'transit stations' for the CO₂ estimation does make sense, the 'workplaces' might not be the best estimator of traffic. However, it was not a big issue and the performance as an estimator for CO₂ is more concerning.

Figure 1 also compares our fuel-based estimates to the recent near-real-time estimates, such as Le Quéré et al (2020a) (median values as solid, high values as dotted, and low values as dashed) and Liu et al (2020a) or the Carbon Monitor (Liu et al 2020b). We found that Carbon Monitor underestimated the emission reduction by 9.1% and Le Quéré et al (2020a) study overestimated the emission reduction by 7.9% (median case) during the period of Japan's state of emergency. The Le Quéré et al (2020b) estimates (also see daily estimates shown in figure S3 in supplemental information) show very different monthly changes from the Apple data case we created. We speculate that the Confinement Index (CI) function used in Le Quéré et al (2020a) is a step function, and the activity level changes from the Apple data were only used to determine the magnitude of the emission changes during the confinement period. The CI was defined based on the policy implemented, rather than quantitative information, and only indicates the timing, duration, and level of severeness of the confinements (Le Quéré et al 2020a). Interestingly, the Carbon Monitor emission change is in significantly good agreement with the one based on traffic (R² = 0.95), rather than with our fuel-based estimates. The emission model was constructed based on the traffic counts data collected in Paris, France (Liu et al 2020b). The model seems to be a good estimator of traffic change in Japan at the national level, but not so much as we had hoped for CO₂ emissions.

We further examined errors and uncertainties in the traffic-based estimates and, more fundamentally, the basic assumption by looking at the values in 2019. Figure 2 shows our fuel-based CO₂ emission estimates for 2019. The 2019 comparison further demonstrates the challenge in the use of traffic data for estimating CO₂ emissions. As also shown in figure 1, the traffic-based estimates are also systematically higher than the fuel-based estimates in 2019, while the seasonal patterns do have similarities in noticeable peaks, such as ones corresponding to the end of Fiscal year (March, i.e. peak season for moving), summer vacations (around July–August), and snow season (December). However, the correlation of the two are not so high. This could be attributable to the lack of the consideration of car/fuel types in the traffic-based approach, sampling bias in traffic data, and the lack of the regional specificities/differences. We see this as an error associated with the methodology. Because of the systematic bias, the emission reduction estimations solely based on the traffic-based approach, which fortunately none of the published studies attempted, could be further biased by 7.2% (18.6% emission reduction).

Zoom In Zoom Out Reset image size

We also looked at the Carbon Monitor estimates, which showed a very good correlation with the traffic-based approach in 2020. In fact, the Carbon Monitor emissions in 2019 looked very different from traffic-based estimates this time. Unlike the 2020 estimates, the 2019 emission calculations in Carbon Monitor begins with annual sectoral total emissions (Liu et al 2020a, 2020b). In fact, their traffic emissions are scaled using a portion of the total 2019 emissions. Thus, these emissions are essentially constrained by the total and thus these emissions should be discussed separately from the 2020 estimates. Since the emissions are disaggregated from a constrained sectoral total, the better agreement with our fuel-based estimates, compared to the case in 2020, was not surprising. Emissions differences among different estimates for established countries, often including Japan, are considered to be small and should agree very well (e.g. Andres et al 2012).

We also revealed that the Carbon Monitor estimates do not seem to capture the emission seasonality shown by our fuel-based estimates, even without the impact of COVID-19. This could be overlooked if monthly emissions are presented as monthly total emissions (see figure S4 in supplement information). In that presentation, the month-to-month emission variations are largely explained by the different numbers of the days in a month and thus it would yield a higher correlation. The overlooked emission variations were small compared to the magnitude of monthly emissions due to the use of the 2019 sectoral total emission as a constraint. However, the systematic biases at the monthly temporal scale were likely to be aliased to the daily estimates via temporal downscaling, while errors in downscaled emissions by themselves can be expected to be much larger at higher temporal frequencies.