A meta-analysis of the relationship between climate change experience and climate change perception

We first took account of all relevant literature from previous reviews [11, 18, 19, 25], and a new search for more recent ones was conducted. We searched article abstracts, keywords and titles for terms related to extreme weather and climate change in January 2022. A comprehensive search strategy was designed before pre-registration, and we selected Web of Science (WoS), PsycINFO and ScienceDirect as our searching database (ref to supporting materials to see full search process). Searching keywords used include three clusters, namely climate change related keywords (i.e., 'Climate chang*', 'Global warming*', 'Climate risk*', 'Climate variabilit*', 'Climate extrem*' and 'Climate uncertaint*'), perception-related keywords (i.e., 'perception*', 'awareness', 'opinion*', 'attitude', 'belief*' and 'experience*'), and extreme weather related keywords (i.e., 'extreme', 'weather*', 'cyclone', 'drought', '*fire', 'flood*', 'hail storm', 'heat wave', 'sea level rise', 'storm', 'tornado', 'thunder', 'tropical storm', 'typhoon', 'wildfire', 'wild fire' and 'cold wave'). Additionally, we manually checked the following high-related journals, which are: Environment and Behavior, Global Environmental Change, Journal of Environmental Psychology, Climatic Change, Environmental Politics. The PRISMA flowchart of this paper is reported in supporting materials.

Once the metadata had been retrieved, Zotero was used to import and de-duplicate the literature information, with the ability to download accessible articles in bulk. We used Rayyan [26], a web and mobile app for systematic reviews, to initially screen the metadata. Two pre-trained raters performed independent screening with the assistance of Rayyan according to the coding manual, and the results were checked by another rater. Disputes that arose during this process were addressed by the corresponding author. We then retrieved the full text of the literature after completing the abstract screening process. For articles where full text is not available, we tried to reach out by searching for author information in the metadata, and if the contact fails, the article would not be included in the analysis (figure S2). In this paper, climate change perception was defined as the attitudes, beliefs, attribution and intentions towards climate change, and we only included studies that report the correlation coefficient (or other related statistics) between extreme weather and climate change perception [11, 18, 27]. We found a variety of acceptable extreme weather events, including cold wave, cyclones, droughts, floods, heatwaves, temperature anomalies, wildfires, and precipitation anomalies, etc. An overview of this criterion is provided in our research protocol.

We adopted Ringquist's suggestion that for r-based effects [28], we used the following rules for their calculation: (1) If an article reports correlation coefficients, we directly converted the correlation coefficients to Fisher's z. (2) If an article reports regression coefficients or t- or z- type statistics, we first converted the statistics to r and subsequently converted them to Fisher's z. (3) If an article uses an experimental design, we first calculated Cohen's d, and then converted it to Fisher's z. The calculation of effect sizes is shown in detail in the pre-registration file. In addition to the calculation of effect sizes, other possible moderators were also extracted from the article, such as the types of extreme weather, the way perception was measured, the country or region, the sample size, the duration of the disaster, etc. These moderating variables were used in the subsequent meta-regression analysis.

Since most of the included studies have multiple outcome variables, to harmonize the dependency of effect sizes within studies, we estimated a multi-level meta-analysis model [29, 30], using the R package metafor [31]. The three-level model is consisted of three different levels of variability regarding effect sizes : the sampling variance of the individual extracted effect sizes (level 1), the variance between effect sizes extracted from the same study (level 2), and the variance between effect sizes extracted from different clusters (level 3) [30]. Variability of level 2 and 3 are estimated, while level-1 variability is assumed to be known and computed as the observed sampling variance of the extracted effect sizes. Before modelling, we checked the anomalous studies. Using infl function in metafor package, we eliminated possible outliers by calculating metrics such as cook's d [32]. Also, since meta-analysis is sensitive to estimators, we used multiple estimators (REML, ML, HS, DL) for regression in order to ensure the validity of the results [33]. Besides, the assessment of publication bias is a crucial part of refining meta-analytical analyses. We examined publication bias using funnel plots which can reflect the bias through asymmetry of the plot. As demonstrated in figure S3, there exist an asymmetry in the effect sizes, which indicates a potential publication bias.

We conducted a systematic search for studies related to climate change experiences and climate change perceptions, in which the samples exhibited a high degree of diversity and a visible increment. As many as 302 effect sizes of different climate change experiences were reported in the 70 articles eventually included in the study. Over twenty types of events related to climate change such as floods [23, 34], hurricanes [35, 36], temperature anomalies [37], droughts [12], precipitation anomalies [38, 39], and other climate-related natural hazards [40] are included. These samples covered 351,378 participants in 12 countries, among which 11,831 individuals (3.4%) were directly exposed to climate change events. We grouped these events into 12 categories, i.e., general temperature anomalies, cyclone, general hazard experience, flood, drought, general weather anomalies, precipitation, snow anomalies, heat wave, cold wave, wildfire and others (all the criterions for classification are presented in method section). Table 1 shows descriptive statistics for the dataset classified by different climate events, where general temperature anomalies accounts for 33% of the samples. Drought ranks second accounting for 8%. In the included articles, effect sizes were used to measure the magnitude of the correlation between climate change perception and personal experiences with climate events because they are independent of the unit of measurement and are not influenced by sample size (e.g., the correlation between reported typhoon weather experiences and reported climate change perceptions can be uniformly translated into effect sizes of the same scale). A higher value means a positive relationship of the climate-related event on climate change perception (more intensive climate event is associate with belief that climate change is human-induced or support for climate change policies, etc.). The effects of all climatic hazards featured conspicuous variability, with minimum values less than or equal to zero. Flood and general temperature anomalies have the two largest values of maximum effect size.

Meanwhile, measurements of climate change events (e.g., self-reported or objective data) and types of climate change perceptions were extracted and used to explain the heterogeneity in effect sizes. A full table of these data is displayed in supplementary information table S1.

By implementing multilevel meta-analysis, we first synthesized the effect sizes for all studies. After controlling for individual study errors, sampling error, and effect size dependency, we found positive but very weak effects (r = 0.0981, 95% CI: 0.0614,0.1348; outliers were dropped, see Method section for detail). Considering the results of the meta-analysis are very sensitive to different models and estimators, we re-calculated the pooling effect using a different method [33]. In the most basic random effects model, the DerSimonian-Laird estimator was used, and the corresponding r was estimated to be 0.0544 (Hunter-Schmidt: 0.0542, Maximum-likelihood: 0.0629, Restricted maximum-likelihood: 0.0629; figure 1(A)), which still shows a negligible correlation. Despite substantial heterogeneity (Q = 8513), evidence shows that in general across all types of events, climate change experience is not correlated with climate change perception, and that the causal relationship between them is, furthermore, likely to be negligible [41].

Zoom In Zoom Out Reset image size

We further successively explored the temporal evolutionary trends of effect sizes. In figure 1(B), the x-axis denotes the effect size (correlation coefficient between climate change awareness and climate change experience) and the y-axis represents the heterogeneity in the samples. The trajectory from gray to black reveals the evolutionary trend over time. This illustrates the process of changes in effect sizes as the number of studies increases, starting with the first paper in 2006 (denoted in light grey) and ending with the last study in 2022 (denoted in black). It is evident that the heterogeneity is low at the beginning (because the lack of literature), then increases as more studies of different designs are involved, and finally decreases again. The effect sizes fluctuate drastically in early years, then gradually stabilize and aggregate as the results accumulate. It is worth mentioning that during the entire time span covered by involved studies, the effect sizes fall into the weak effect interval less than 0.3 at most moments.

Subgroup analysis was conducted to examine the heterogeneity associated with different types of climate-related events. Does exposure to a specific climate change event results in different levels of climate change perception? Based on the classification above, we performed a multilevel meta-analysis for each group, and as before, we controlled for correlation in the same article, heterogeneity across experiments, and statistical error. As shown in figure 2, the point estimates for all climate events are greater than zero, but the 95% confidence intervals for most estimates contain zero, which implies insignificant correlations. Only the effect size for floods, droughts, general temperature anomalies and general hazard events are significantly higher than zero, indicating a valid but weak correlation.

Zoom In Zoom Out Reset image size

General hazard experience exhibits the highest correlation with climate change perception (z = 0.154 corresponding r = 0.153, d = 0.312). General hazard experience includes studies that did not clearly indicate a specific classification, such as studies using the description 'experience of a natural disaster' or 'personal experience with climate change' in the questionnaire. Also, a few studies using synthetic indices, such as the Climate Extreme Index, are also included in this category [42, 43].

General temperature anomalies likewise exhibit a result that is significant and close to the general extreme experience. In this subgroup, most studies use local temperatures or deviations of local temperatures from normal to measure climate change experiences. In a previous meta-analysis, the researchers included 31 climate change experiences specified as local temperatures to examine the relationship between the remaining climate change belief [21]. In their study, a low effect size was summarized (d = 0.17, corresponding r = 0.085). By expanding the sample size, we included 97 effect sizes for temperature or temperature anomalies (Heat wave was separated from Cold wave when high and low temperatures could be identified). We obtained a slightly larger effect size (r = 0.10, corresponding d = 0.201), and such effect size indicates a limited influence of temperature on the perception of climate change.

Droughts and floods both show weak but significant effects. Droughts include directly identifiable dry weather events, while precipitation anomalies (unspecified restricted or excess precipitation) are not included in this classification. Overall, compared to floods (z = 0.0362, corresponding r = 0.0362, d = 0.072), droughts (z = 0.0437, corresponding r = 0.0437, d = 0.087) reflect a higher effect size, but both of their effects are small.

Also, it is noted that the error of some types of climate-related events is substantial, such as the two categories: wildfires and others (see note in the figure 2). They both have small sample size (only two for wildfires), and the category of others contains a variety of events or measurement such as wind speed, sea level rise, and diseases due to climate change, which leads to high heterogeneity and errors.

Overall, our subgroup analyses suggest that the relationship between climate change experience and climate-related hazards is likely not significantly influenced by different climate change events. Most of the specific climate change experiences do not have a correlation with perceptions. However, general hazard experiences and general temperature experiences are weakly correlated with climate change perceptions, while negligible correlations existed for floods and droughts. Next, we will take a closer look at the other variables that predict the effect size in the moderation analysis.

Meta-regression is used to examine the effect of experimental design on effect sizes. Key characteristics of the studies were selected as moderating variables, including the method of measurement (self-reported or objective), directly or indirectly exposure to climate event (victims or not), region of the research, whether it is a temperature-based study, whether behavior is studied, whether the sample size is large (namely, N > 500, follow the categorization of Nisa's study [44]), and the time of publication. These variables were treated as dummy variables if they are discrete, and we used multilevel meta-regression with REML estimators to compute the model.

As shown in table 2, we find that three moderating variables show significant impact on the value of effect size: method of measurement, whether the climate event was directly experienced, and the sample size of the study. Among these, climate change experiences measured by self-report significantly increased the effect size (p-value = 0.0009, standard error = 0.0370), implying that higher correlations may be observed in studies that use solely questionnaires rather than external indicators of experience. This result also indicates that, compared to actual climate change experiences, perceived climate change experiences are more correlated with climate change perception, confirming the hypothesis brought by previous systematic review [11]. Peoples' recognition of experiences of climate change events may be determined by their knowledge about the physical characteristics of climate change and their prior beliefs about the phenomenon. Thus, people who report such experiences may tend to have a better understanding of climate change and be more likely to believe that human-caused climate change is happening and poses a threat. Interestingly, direct exposure to climate change events (i.e., studies that used external indicators of direct exposure to climate-related events as a sample) reports a lower correlation compared to indirect exposure (p-value = 0.0199, standard error = 0.0623). A possible reason for this is that some people's feeling of negative impacts will elicit a state of aversive arousal [45] and people's refusal on attributing their extreme weather experience to the climate change [14]. Finally, studies with a broader spatial scale are more likely to report lower effect sizes, compared to studies focusing on small areas. The regression result for sample size is marginally significant(p-value = 0.0610, standard error = 0.0559), suggesting that small-scale studies may tend to report an overestimation of correlations.