Can citizen scientists provide a reliable geo-hydrological hazard inventory? An analysis of biases, sensitivity and precision for the Rwenzori Mountains, Uganda

Rwenzori Mountains are located in tropical Africa (Peel et al 2007) across the southwestern border of Uganda and eastern D.R Congo. This mountainous region suffers from frequent landslide (Jacobs et al 2016a) and floods (Jacobs et al 2016b). The Ugandan side of the mountains is characterized mainly by protected zones (national parks, forest, and wildlife reserves) and inhabited zones (figure 1). This study focuses on landslide and flood events on the inhabited Ugandan side of the Rwenzori Mountains, specifically those occurring in the two districts of Kasese and Bundibugyo.

Zoom In Zoom Out Reset image size

2.2.1. Citizen science-based inventory

2.2.2. Satellite imagery-based inventory

2.2.3. Expert-based field assessment and validation

From mid-September to mid-December 2020, a field assessment and validation was conducted by the first author to systematically map landslides and flood occurrences (figure 2) in the 30 selected parishes and verify the events reported in the CS database and those events identified through visual interpretation of satellite imagery. For each parish, a local guide was hired to ensure all the possible areas of the administrative unit were reached. This facilitated mapping as many landslide and flood events that occurred in the period of May 2019 and May 2020 as possible. The CSs reports, and the RS-based inventory were also verified by systematically visiting these locations.

Zoom In Zoom Out Reset image size

Each event that would be found to exist in the field and correctly identified by the CSs and/or mapped from the satellite imagery, was labelled True Positive (TP). Those erroneously mapped as landslide or flood events or found not to exist in the field, were labelled False Positives (FP). Events that were found to exist in the field but had been missed by either the CSs or the RS inventory process were labelled False Negatives (FN). The process is illustrated using a hypothetical field space in figure 3. These classifications were then used in subsequent analyses outlined below and summarized in the schematic figure 4.

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

2.3.1. Sensitivity, precision, F1

Sensitivity (also called true positive rate or recall rate) is used to refer to the proportion of existing positives that are correctly identified or detected as positives (Powers 2008, Tharwat 2018, Chicco and Jurman 2020). In previous studies, sensitivity has been used to measure the performance of diagnostic tests (Genders et al 2012) and classification performance in computational linguistics (Powers 2008). In this study, sensitivity is used to refer to the proportion of landslides/floods that occurred and were correctly reported by CSs and/or mapped from satellite imagery. It is defined as given in equation (1).

$$\begin{equation}{\text{Sensitivity}} = { }\frac{{{\text{TP}}}}{{\left( {{\text{TP}} + {\text{FN}}} \right)}}.\end{equation} \tag{ 1 }$$

Precision (confidence) has been applied in information retrieval and data mining studies to measure the predictive performance of models. It refers to the proportion of predicted positives that are truly positives (Powers 2008, Boughorbel et al 2017). In this study it denotes the proportion of detected landslides or floods that were real landslides or floods in the field. It is calculated as given in equation (2).

$$\begin{equation}{\text{Precision}} = \frac{{{\text{TP}}}}{{\left( {{\text{TP}} + {\text{FP}}} \right)}}.\end{equation} \tag{ 2 }$$

The F₁ score (F measure) is an integration of both sensitivity and precision. It gives the harmonic mean of sensitivity and precision and ranges between 0 and 1 (Tharwat 2018). Although the metric has received some criticism (Flach and Kull 2015), it is still a widely used measure of classification or predictive performance studies (Genders et al 2012, Chicco and Jurman 2020, Chicco et al 2021). Here it was applied to measure the overall performance of CSs and satellite in detecting landslide and flood occurrences. The higher the F₁ score, the better the detective performance of the hazard inventory approach. It is calculated as in equation (3).

$$\begin{equation}{{\text{F}}_{\text{1}}}\,{\text{score}} = {{{\,}}}\frac{{2{\text{TP}}}}{{2{\text{TP}} + {\text{FP}} + {\text{FN}}}}.\end{equation} \tag{ 3 }$$

The comparison of the CS-based- and satellite-based inventories with the field data and subsequent calculation of sensitivity, precision and F₁ statistics allowed a direct contrasting of the data quality of the two former inventories (figure 4).

2.3.2. Spatial bias analysis

CS-based inventories can be affected by spatial biases (Zhang and Zhu 2018). In our study, the landslide inventory in particular could be affected by biases: landslide events are spatially more dispersed than floods and are more frequent but smaller in size, which could render them less noticeable. CSs might have more knowledge or be more likely to visit areas closer to their homestead (Pernat et al 2021). In addition, landslide inventories by CSs but also by experts (Dickinson et al 2010) can be affected by a bias towards those areas closer to the road network (Wang et al 2020). This creates incompleteness issues in such citizen-based databases. Here we examine the potential sources of spatial biases by analysing factors that determine whether a citizen scientist detects and records a hazard event. To do this, a binary logistic regression model was applied. Logistic regression is a widely used modelling approach in determining relationships between a binary dummy dependent variable and explanatory variables (Coughlin et al 1992, Genders et al 2012, Nishadi 2019). In our case, the binary dependent variable is whether or not a landslide that was confirmed in the field was reported by the citizen scientist (1) or not (0). If landslide reporting ability is represented as Ls_report and X₁, X₂, X₃, ..., X_n represent a set of potential explanatory variables, the probability of a landslide being reported by CSs is given as shown in equation (4).

$$\begin{align}&P\left( {{\text{L}}{{\text{s}}_{{\text{report}}}} = 1} \right)\nonumber\\ &\quad = \frac{1}{{1 + {e^{ - \left( {\alpha + {\beta _1}{ }{X_1} + { }{\beta _2}{X_2} + {\beta _3}{X_3} + \ldots \ldots \ldots \ldots .{\beta _n}{X_n}} \right)}}{ }}}.\end{align} \tag{ 4 }$$

where $P\left( {{\text{L}}{{\text{s}}_{{\text{report}}}}} \right)$ is the probability of a landslide being detected and ${\beta _1}$ to ${\beta _n}$ are regression coefficients. The first set of explanatory variables used in equation (4) model were landslide related factors. These included landslide size, slope, altitude, landslide-citizen scientist home distance, landslide-road access distance and impact (weighted sum of effects on humans, infrastructures and crops, appendix B). Although even a low impact landslide event can be detected and reported by a citizen scientist, the more disastrous an event is, the higher is the chance that such an event will be discussed on different platforms and the easier it might be for a citizen scientist to learn about it and detect it. The regression analysis was repeated while accounting for a fixed effect of CSs (citizen scientist specific unique attributes). To achieve this, unique identity (citizen scientist ID) numbers were introduced in the model. To gain an insight on how much variance is explained by each of the factors included in the regression analysis, a univariate logistic regression analysis was performed for each of the variables. The Tjur's coefficient of discrimination, D, was then used as an indicator of the individual contribution for a specific variable in explaining variance in landslide detectability by the CSs (Tjur 2009, Allison 2014). We then investigated which factors could explain variations in the observed individual odds ratios of CSs. This explorative analysis was done through a Spearman rank correlation matrix involving individual odds ratios of CSs, age, years spent reporting, size of the area monitored and number of reports per citizen scientist.

Out of the 255 and 59 landslides and floods retained in the CS-based inventory after manual review and automated screening of incoming reports, 221 landslides and 46 floods were verified in the field. Of the 459 landslides and 59 floods identified on satellite images, 373 and 57 respectively could be verified in the field (table 1). The remaining instances could not be verified due to accessibility constraints in the field and were not considered in further analysis. Results show that CSs identify and report landslide and flood hazards in their localities with 99% and 100% precision respectively (table 1). Table 1 however indicates also a relatively low sensitivity for CS-based landslide and flood detection (51% and 70% respectively). RS-based landslide and flood detection is less precise particularly for landslides (39%) in addition to having even lower sensitivity. RS-based flood detection has a higher sensitivity compared with CS. Overall, floods are more easily detected by both CSs and RS as implied by F₁ scores of 0.8 and 0.8 respectively.

Table 2 shows a direct comparison between CS and RS inventories in the form of a confusion matrix. Only 16% of the 425 landslides mapped in the field were detected by both satellite and CS and 26% were detected by neither inventories. Remarkably, 35% of the landslides was detected by the CS but not by the satellite interpretation and 23% of satellite detected landslides were not identified by CSs. The high sum of this diagonal demonstrates that both inventories are to a large extent complementary and typically do not detect the same events. While the average size of the landslides detected and not detected by the CSs is not statistically significant (Prob > z = 0.088), satellite imagery detects larger events and tends to miss out on the small ones (Prob > z = 0.000). Through logistic regression (appendix C; table 8), it was found out that landslide detection by RS is significantly influenced by landslide size, slope and altitude (Tjur's D = 0.226). A marginal effect post estimation showed that the chances of missing out some landslides on a satellite image increase as the landslide area reduces.

Results of the logistic regression show that the variation in landslide detectability by CSs is significantly influenced by the landslide's impact, the distance between citizen scientist's residence and the landslide, the distance from the access roads to landslide and altitude (table 3). The odds ratios confirm the hypothesis that larger impact and shorter distance increase the likelihood of landslide events being detected. However, based on the Tjur's D, these factors only explain a small proportion (Tjur's D = 0.138) of the variations in landslide detectability. Through univariate analysis, a Tjur's D for each factor was generated to gain an understanding of the maximum contribution of each individual factor to the variance in landslide detectability by CSs (table 3 column 3). Based on the univariate analysis results, the landslide distance from the citizen scientist's home plays the most important role, followed by impact and slope in influencing landslide detectability.

To check potentially controlling factors related to the individual CSs included a fixed effect for each individual in the logistic regression. This increased the explanatory power of the model (Tjur's D = 0.435 from 0.138). In a final step, we explored which personal characteristics could influence the resulting odds ratios by simple correlation with the variables as presented in table 4.

CSs can correctly identify and report georeferenced hazard information comparable to research experts. Although no control trial was done to quantify the added value of the training workshops, this high precision rate after training workshops and in-field practice concurs with what Lewandowsli and Spencht (2015) report on the importance of adjusting protocols and volunteers training in improving data quality (Lewandowski and Specht 2015). According to Paul et al (2014), CS-based data collection programs result in datasets with quality attributes similar to those established systematically by experts (Paul et al 2014). However, their inventories (CSs) suffer incompleteness challenges especially those that involve mapping small, frequent but spatially spread events like landslides (Rohan et al 2021).

Despite the relatively low sensitivity, the CS database outperformed the RS-based inventory in terms of sensitivity, precision and F₁ scores when considering landslides. The low sensitivity and precision of RS-based inventory compared to CSs and other studies are due to several factors. We used satellite images at a 3 m resolution while other studies used Google Earth images that offer a 0.3–0.6 m resolution (Hao et al 2020, Ubaidulloev et al 2021). The images available in Google Earth are usually acquired with a lower temporal resolution compared with the PlanetScope images, and therefore may capture landslide scars that are already (partially) concealed by the regeneration of the vegetation (Dewitte et al 2022). This reduction of the spectral signature of the landslides identified via Google Earth is however counterbalanced by the higher spatial resolution of the images. In addition, CSs visit the field after a few days, allowing them to identify very small landslides that cannot be seen on a satellite image. Furthermore, the inventory was limited to areas monitored by CSs. These areas are characterized by cultivated landscapes made of parcels that present shape that could be mistaken for landslides on the satellite image. The confusion matrix (table 2) demonstrates that the CS inventory and the RS inventory are largely complementary. However, the CS inventory in turn is affected by biases predominantly relating to a fixed individual citizen scientist effect and to lesser extent relating to the landslide characteristics, impact, and location. The further the event from the residence of the citizen scientist or the community access roads, the higher are the chances of such an event being missed. The magnitude of impact and nature of the element impacted are also important. This supports our hypothesis that more destructive events are more readily communicated or discussed on different platforms and so easier to detect by CSs. People are more motivated to report events that affect their houses or cause injuries/death than those that affect crops and pastures. This could partially be due to the perception that CSs reports reach humanitarian agencies who eventually distribute relief items mainly to those whose houses get damaged. Most landslide events that affects houses in the study area are usually small resulting from (a) steep slope cuts while establishing platforms for house construction and (b) small scale cropping farming on the steep slopes. This partially explains why CSs report more of small events. The observed improvement in the explanatory power of the model after introducing the citizen scientist ID implies that CSs specific characteristics (here bundled into one fixed effect) play an important role in influencing the likelihood of landslides being detected. Thus, while CS-based inventories might depend on socio-cultural, infrastructural, motivational factors of the CSs and nature of the hazard, satellite imagery largely depends on nature of the target object in the feature space (size of the object) and the prevailing atmospheric conditions (Agapiou et al 2011, Selva 2021). In general, it should be noted that the ability of CSs to detect landslides that have occurred largely depend on whether they get informed about their occurrence. The positive correlation between number of reports and age of CSs (table 4) can be explained by the fact that the older the CSs are, the higher the chance of them having a stable residence in their village and the wider the social network built from whom news about landslide occurrences can be received. In addition, older CSs tend to stay longer in the reporting network as indicated by a strong positive correlation (r = 0.55) between years of operation and CSs age (table 4).

CS-based inventories and RS-based inventories not only differ in which landslides are detected, but both inventory methods also fundamentally differ in the type of information that can be collected: CSs can collect data on the impact associated with the hazards that is not visible through remotely sensed information. RS-based mapping on the other hand provides an opportunity to record geo-hydrological hazards even in inaccessible areas (Ji et al 2020). On the other hand, while revisit periods of very high-resolution satellites are shortening, image interpretation is oftentimes still hampered by cloud coverage, while CSs tend to visit, and map affected sites within few days after occurrence (Jacobs et al 2019). The implication here is that CS-based landslide detection and very high-resolution (⩽3 m) satellite imagery are complementary in spatial extents, types of landslides and types of information collected, but not substitutes. Because of this complementarity, CS data represent an added value to hazard assessment and risk analysis for example in determining thresholds for landslide triggering factors (Monsieurs et al 2019) and supplementing incomplete landslide inventories (Samodra et al 2018). CSs generate reliable georeferenced hazard information including information on timing and damage inflicted, particularly in data-poor countries like Uganda where extensive disaster risk is high. Such events do not easily make it to social or formal media platforms in addition to not meeting the impact thresholds of the existing national (DesInventar) data repositories (UNDRR 2019b) and global (EM-DAT) databases (IFRC). To reduce incompleteness challenges, it is relevant for projects and programs aimed at spatio-temporal explicit hazard assessment to complement high spatial resolution satellite image-based hazard inventories with CS hazard sensing. By visiting the affected scenes within a few days and documenting the impact caused, CS-based inventory can be useful in temporal landslide assessment and vulnerability analysis which are prerequisites for risk analysis.

Although we find the quality of the CS-based inventory of good quality, maintaining a big network of CSs to cover a large geographical area over time can be challenging and require investment to equip, train and continuously monitor the activities of CSs. Although CSs report geo-hydrological hazards with high precision, the inventory should be carefully used for regional susceptibility and risk assessments as it is not free from spatial biases. In addition, the satellite imagery digitization was entirely made by the main author of this manuscript. Although efforts were made to do it systematically, it does not completely erase possibility of omitting or including certain features that would respectively be or not be included by other users.