Consensus revisited: quantifying scientific agreement on climate change and climate expertise among Earth scientists 10 years later

Syntheses of existing scientific studies on climate change, such as the Intergovernmental Panel on Climate Change (IPCC) assessment reports, have published increasingly strong attribution statements on anthropogenic global warming (AGW). Over time, these have strengthened from qualified statements such as 'the balance of evidence suggests that there is a discernible human influence on the global climate' (Houghton et al 1996) to more definitive conclusions such as 'it is extremely likely that human influence has been the dominant cause of the observed warming since the mid-20th century' (Qin et al 2014). The 2021 IPCC Technical Summary states 'Human influence on the climate system is now an established fact: The Fourth Assessment Report (AR4) stated in 2007 that "warming of the climate system is unequivocal", and the AR5 stated in 2013 that "human influence on the climate system is clear". Combined evidence from across the climate system strengthens this finding. It is unequivocal that the increase of CO₂, methane (CH₄) and nitrous oxide (N₂O) in the atmosphere over the industrial era is the result of human activities and that human influence is the principal driver of many changes observed across the atmosphere, ocean, cryosphere and biosphere.' (Arias et al 2021).

While reports such as the IPCC's act as definitive summaries of climate science, psychological research documents the important role of simple, quantitative estimates of expert agreement on specific questions such as attribution of global warming. Specifically, the gateway belief model identifies the influential role of perceived consensus about AGW on a range of climate perceptions and attitudes (van der Linden et al 2015). Increased awareness of the scientific consensus on climate change is associated with greater acceptance that human-caused global warming is happening, and results in higher support for climate mitigation policies (Ding et al 2011). Similarly, messages communicating the scientific consensus on climate change increase not only perceived consensus, but also perceptions that climate change is real and human-caused, as well as support for climate action (Lewandowsky et al 2013). The gateway belief model underscores the importance of research into the consensus in order to communicate the degree of expert agreement on AGW.

Hartnett (2011) reviews the concept of consensus and distinguishes it from its close cousin: unanimity. They write that 'consensus is defined by Webster's dictionary as "agreement of the majority in sentiment or belief" and by the Oxford dictionary as "general agreement."' (p 2). Hartnett's analysis of consensus is in the arena of facilitation of group decision processes; and in this context they define another useful term: a decision rule—the criterion by which a group finalizes a decision. There are various levels of agreement that can be utilized as the decision rule: unanimity, majority, or a supermajority (they write: 'a supermajority threshold can theoretically be anything between simple majority and unanimity. Common figures include 60%, 65%, two-thirds and 75%' (p 30); they also write that some groups opt for person-in-charge verdicts or a decision by a governing committee as decision rules).

Literature assessing the level of agreement among climate scientists has typically applied the term 'consensus' to communicate the high degree of agreement among climate scientists regarding human-caused climate change. While there may be ambiguity around what constitutes consensus, in this manuscript we apply the consensus definition of 'general agreement' (in agreement with the definitions found in the Cambridge, Merriam Webster, and Oxford dictionaries) and assess the level of agreement that exists (or does not) among climate scientists.

There is difficulty in defining who should 'count' as a climate science expert when assessing the level of consensus among climate scientists. In this manuscript, we explore several approaches to assessing expertise including self-reported areas of expertise, number of scientific publications, percent of climate relevant publications, as well as an objective abstract review analysis of self-reported climate experts and assess whether there is general agreement at each level of expertise.

The first study to quantify the level of scientific agreement on climate change was a content analysis of abstracts from 928 papers on 'global climate change', finding zero papers rejecting AGW (Oreskes 2004). The next quantification of consensus was a survey of 3146 Earth scientists taken from a directory of geoscience faculty at reporting academic and research institutions (Doran and Kendall Zimmerman 2009). This survey asked participants their views on whether global warming was happening and human-caused, as well as questions exploring their degree of expertise in climate science. Overall, 80% of participants agreed that human activity was a significant contributing factor in changing mean global temperatures. Among participants with the greatest expertise in climate research—listing climate science as their area of expertise and more than 50% of their recent peer-reviewed papers relevant to climate change—the consensus on human-caused global warming was 97.4%.

Shortly following the Doran and Kendall Zimmerman (2009) study, a content analysis of public statements about climate change found 97.5% agreement among the most published climate scientists (Anderegg et al 2010). The third study to find 97% scientific consensus on AGW was a content analysis of the abstracts from 11 908 peer-reviewed studies on 'global climate change' or 'global warming' (Cook et al 2013). Among abstracts stating a position on AGW, 97.1% endorsed the consensus position.

Verheggen et al (2014) conducted a survey of scientists, whose contact information was acquired by searching the terms 'global warming' and/or 'global climate change', for publications between 1991 and 2011 via Web of Science. From this pool, 90% of respondents with more than ten climate-related publications, agreed with anthropogenic greenhouse gases being the dominant driver of recent global warming. Similarly, Carlton et al (2015) surveyed scientists across a range of biophysical disciplines, finding 96.7% consensus among those whose majority of research concerned climate change.

A 2016 synthesis of existing consensus studies found estimates ranged between 90% and 100% consensus, with multiple studies converging on 97% agreement (Cook et al 2016). A consistent pattern found across studies was that higher consensus was associated with greater expertise in climate research.

There is a dearth of research into how consensus on climate change has evolved over time. Shwed and Bearman (2010) analyzed citation networks, finding a strengthening consensus over the 1990s although their quantification of consensus differs from the consensus studies discussed here and does not bear direct comparison. Cook et al (2013) found a slightly increasing trend in scientific agreement from 1991 to 2011, albeit starting in the high 90s at the start of the period. In this study, we replicate in large part the methodology of Doran and Kendall Zimmerman (2009), surveying the Earth science community to measure acceptance of AGW, a type of direct methodological replication that has yet to be conducted. We address the following exploratory research questions (RQ):

RQ1: What is the current extent of agreement among climate experts on the role humans play in global warming?

RQ2: Does the strength of the relationship between expertise and acceptance that global warming is happening and human-caused vary depending on the measure of expertise used?

Names and contact information of active Earth scientists were mined from the Directory of Geoscience Departments, 53rd Edition 2018, by the American Geosciences Institute (Wilson 2018), which lists all geosciences faculty at reporting academic and research institutions (mostly in Canada and the U.S., but also international institutions). In addition to university faculty, this directory includes researchers at state geologic surveys associated with local universities and researchers at U.S. federal research facilities (e.g. U.S. Geological Survey, NASA—National Aeronautics and Space Administration, NOAA—National Oceanic and Atmospheric Adminstration, and U.S. Department of Energy national laboratories). Following the approach of Kendall Zimmerman (2008), an email list was created by cutting the spline off the hardcopy directory and feeding individual pages through a scanner equipped with Optical Character Recognition software to compile a digital list of individuals' emails, last name, first name, university affiliation, and academic area of expertise. The scanned text was then imported into a spreadsheet and manually cleaned to build the mailing list.

Email addresses were uploaded to online email verification software Kickbox®. The program flagged 1009 emails as undeliverable which were then manually checked by conducting an internet search of their name and/or affiliation to identify errors in the email address (typically off by 1–2 characters). Email addresses were corrected by hand if they were found online. Unresolved emails were deleted from the list, as were duplicate addresses. Near duplicates (one letter off) were also checked and deleted. Out of the original 11 350 email addresses, 421 emails were deleted during this step.

The final email list of 10 929 Earth scientists was imported into the survey platform Qualtrics® (see supplementary materials section S1 for survey design (available online at stacks.iop.org/ERL/16/104030/mmedia)). All names on the list were emailed an invitation to participate in the online survey on 10 September 2019. The recipients were sent up to three reminders if they did not initially complete the survey, and the survey was closed on 16 October 2019. Out of the 10 929 invitations, we received 2780 responses, corresponding to a response rate of 25%. The previous survey of geoscientists conducted in 2009 yielded a response rate of 31% (Doran and Kendall Zimmerman 2009), although other more recent surveys of scientists have yielded lower response rates ranging from 9% to 16% (Besley et al 2018).

The survey was developed to assess the current state of consensus on AGW among geoscientists. The survey also included questions about other impacts, responses, and conceptions about AGW, however we do not focus on the other questions in this paper. Additionally, participants provided multiple self-reported indicators of their expertise on climate change, such as the highest level of education completed, number of scientific papers published, and number of years worked in the field of climate change.

2.1. Measurement of human influence on global warming

2.2. Quantifying expertise on climate change

In order to address RQ2, we used seven different approaches to quantify the level of expertise on climate change.

2.2.1. Self-reported expertise in climate science and/or atmospheric science

We first asked participants their top three areas of expertise with the following question: 'Which are your areas of expertise? ⁴ ' Only participants who selected climate science and/or atmospheric science within their top three areas of research expertise were retained for analysis in the approaches described below.

2.2.2. Self-reported number of scientific publications

A single question asked participants to report the number of peer-reviewed articles they published in any scientific area between 2015 and 2019 ('To the best of your knowledge, about how many peer-reviewed publications in any scientific area have you published over the last five years?'). Response options were 0 (6.4%), 1–3 (17.3%), 4–6 (13.6%), 7–9 (9.4%), 10–15 (15.6%), and 16 or more (37.6%).

2.2.3. Self-reported number of climate relevant publications

An additional question asked participants to indicate the percentage of their peer-reviewed publications that were in the area of climate change between 2015 and 2019 using a sliding scale ('To the best of your knowledge, what percentage of your peer-reviewed publications in the last five years have been in the area of climate change?', sliding scale from 0% to 100%, mean = 39.39%, S.D. = 34.56, n = 507). We then multiplied the self-reported number of peer-reviewed publications in any scientific area between 2015 and 2019 by the self-reported percentage of peer-reviewed publications that were in the area of climate change during the same time period. Because response options for total number of publications were a range, we used the midpoint of each option (for example, if the respondent chose 1–3 papers, we assigned a value of 2 papers. For the highest category of 16+ we assigned the value of 16). We then rounded the product of the percentage and number of papers to arrive at the number of climate-relevant publications (we also excluded anyone who failed to respond to either the percentage of climate papers or total number of papers). We then defined four categories to represent the range of climate-relevant publications: 0–5 (68.6%), 6–10 (14.9%), 11–15 (13.5%), and 16+ papers (3%).

2.2.4. Self-reported number of years worked in climate change

Participants were asked to report the number of years they had worked in the area of climate change ('How many years, if any, have you worked in the area of climate change? [If you have not worked in climate change, please enter 0]'). Participants could enter any integer (automatically verified) in an open-ended response box. We grouped the responses into five categories: 0–5 years (23.5%), 6–10 (14.3%), 11–15 (12.0%), 16–20 (17.7%), and 21+ years (32.5%).

2.2.5. Self-reported 50% or more of scientific publications in the area of climate change

Following Doran and Kendall Zimmerman (2009), we considered an individual a self-identified climate expert if they reported 50% or more of their scientific publications were in the area of climate change.

In order to independently assess how many peer-reviewed papers each author published, and hence, what percentage of their research is climate relevant, we conducted an abstract review analysis. We first exported all scientific papers co-authored by the self-identified climate experts from Web of Science for five years (2015–2019). This group contained 192 participants, however three of those participants did not answer the question about attribution of global warming (this is why table 1 lists n = 189 for 'self-identified climate experts'). Another 15 participants were not included in the abstract review analysis because we could not find any published papers on Web of Science between the years 2015 and 2019. This generated a database of the titles and abstracts of 3006 peer-reviewed publications from the 174 self-identified climate experts.

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

Table 1. Summary of results from geoscientist survey response to question about attribution of global warming. Each row represents a different method to quantify respondents' level of expertise, which is shown from lowest (all survey respondents) to highest (confirmed climate experts) level of expertise on climate change. For the 'odds ratio' column, one asterisk corresponds to p < 0.05, and two asterisks correspond to p < 0.01.

		Main reason for observed warming on Earth
Group	Survey response/Description	Human activity	Natural processes	No warming	Odds ratio
All survey respondents (n = 2548)	All respondents that answered the question	91.1% n = 2320	7.9% n = 202	1% n = 26
All climate scientists and atmospheric scientists (n = 511)	Respondents that listed climate science and/or atmospheric science as top three in area of expertise, see figure 1 for breakdown	94.3% n = 482	4.9% n = 25	0.8% n = 4	1.8**
Self-identified climate experts (n = 189)	Respondents that listed climate science and/or atmospheric science in their top three areas of expertise and indicated >/=50% publications in the area of climate change. See figure 2(a) for results from abstract review analysis of self-identified climate experts	96.3% n = 182	3.2% n = 6	0.5% n = 1	2.1*
Confirmed climate experts (n = 153)	Respondents that listed climate science and/or atmospheric science in their top three areas of expertise, and that actually published >/=50% publications in the area of climate change, fact checked through abstract review analysis, figure 2(b)	98.7% n = 151	1.3% n = 2	0% n = 0	5.6**

Each of the 174 authors was assigned an author identification number, and all personally identifiable information was removed from the compiled list of abstracts. The abstracts were given unique abstract identification numbers, and that list was randomized. The remaining information was abstract identification number, paper title, and the text of the abstract.

The abstract database was content analyzed by a group of five people (coders) in order to determine whether the publications were relevant to climate science. First, coding instructions were developed for the abstract review analysis to delineate which abstracts were considered climate-relevant (details about coder recruitment and the coding instructions are available in the supplementary material section S2). Abstracts were coded as relevant if they dealt with methods of climate science, paleoclimate, impacts of climate change, or mitigation and adaptation (a categorization framework adapted from Cook et al 2013). Abstracts were coded as not relevant if they were commentary about climate science, dealt exclusively with energy topics (with the exception of the impact of energy use on the climate), social science research around climate change, or any other non-climate subject. Further details on content analysis procedure and interrater reliability results are outlined in the supplementary materials section S3.

Abstract review analysis results were compiled for each self-identified climate expert, which included the total number of papers published between 2015 and 2019, and the number of those papers that were coded as climate relevant during the process outlined above. We then were able to assess how each self-identified climate expert responded to our survey questions in the context of how many confirmed climate relevant publications they co-authored.

2.2.6. Confirmed 50% or more of scientific publications in the area of climate change

Since some respondents might intentionally or unintentionally provide inaccurate responses to self-reported measures of expertise (e.g. the percentage of climate relevant papers published in the last five years), we sought to verify their responses of their publication record. We used the results from the abstract review analysis to confirm claims of publishing 50% or more climate relevant papers. This resulted in a group of confirmed climate experts (n = 153) who actually published 50% or more of their research in the field of climate change.

2.2.7. Regression analysis

We quantified the relationship between various measures of expertise and AGW by performing logistic regressions, predicting the likelihood of attributing global warming to human activity vs. any of our measures for expertise, controlling for age and gender. For the three dichotomous measures of expertise, we report the odds ratio (OR) of an expert indicating warming on Earth is due to human activity in comparison to a non-expert. For the three ordinal measures of expertise, we report the OR for the difference in likelihood of reporting AGW between two ordinally-stepped categories (i.e. between those who had 0 scientific publications and those who had 1–3 scientific publications).

Out of all survey respondents who answered our primary question about attribution of global warming (n = 2548), 91.1% responded 'Human Activity', 7.9% responded 'Natural Patterns', and 1.0% responded 'No Warming'. No respondents selected 'Do Not Know'. This is roughly 11 percentage points higher than the 80% agreement found by Doran and Kendall Zimmerman (2009) when asked an equivalent question ⁵ . This suggests that the number of Earth scientists that do not agree with AGW has shrunk by more than half over the last ten years indicating an increase in overall consensus on AGW amongst Earth scientists. Of these participants who responded to the question about global warming, 2495 also reported their highest level of education completed. 90.9% held PhDs, 8.1% held master's degrees, and 1% held bachelor's degrees.

Additionally, each respondent could choose their top three areas of expertise (subdisciplines of Geosciences). The most common area of expertise chosen was Geochemistry (n = 620), followed by General Geology (n = 448), and Sedimentology/Stratigraphy (n = 447). The subdiscipline with the highest level of agreement on our primary question about attribution of global warming was Biogeochemistry/Ecology, where 96.3% of respondents who chose this as one of their areas of expertise replied 'Human Activity' (n = 296), 3% replied 'Natural Patterns' (n = 9), and 0.7% replied 'No Warming' (n = 2). The subdiscipline with the lowest level of agreement on our primary question about attribution of global warming was 'Economic Geology', where 84.1% of respondents replied 'Human Activity' (n = 127), 12.6% answered 'Natural Patterns' (n = 19), and 3.3% answered 'No Warming' (n = 5). The breakdown of the survey results by respondents' sub-discipline and location is shown in the supplementary materials section S4.

We used various metrics of expertise to calculate consensus based on responses to the question about attribution of global warming, with results summarized in table 1 and figure 1.

3.1. Self-reported expertise in climate science and/or atmospheric science

3.2. Self-reported number of scientific publications (n = 503)

3.3. Self-reported number of climate relevant publications (n = 497)

3.4. Self-reported number of years worked in climate change (n = 502)

3.5. Self-reported 50% or more of scientific publications in the area of climate change

3.6. Confirmed 50% or more of scientific publications in the area of climate change

3.7. Regression analysis

We used survey data to measure the current extent of agreement on AGW among Earth scientists. Across all definitions of expertise, our study indicates there is strong and robust consensus among geoscientists that the Earth's temperature is getting warmer mostly because of human activity, such as burning fossil fuels. The percentage of scientists who accept AGW ranged from 91.1% (all respondents) to 100% (most actively publishing climate experts, 20+ climate papers, figure 2). Respondents who chose Economic Geology as one of their areas of expertise had the lowest level of agreement on AGW (84.1%) (see supplementary material S4 for breakdown by subdiscipline). Doran and Kendall Zimmerman (2009) found consistent results when asking a similar question about AGW, where 47% of Economic Geologists surveyed responded that human activity is a significant factor in changing global mean temperatures. This suggests an increase in agreement on AGW amongst Economic Geologists, however the subdiscipline still has a 7% lower agreement compared to all survey respondents. We found a similar large increase in level of agreement on AGW among those self-identifying as Meteorologists—from 64% in Doran and Kendall Zimmerman (2009) to 91% in our study.

Our study suggests that expertise predicts consensus, where the higher level of expertise results in a higher level of agreement on AGW. One exception was the scientists with the most years worked in the area of climate change (20+), had lower consensus than groups with less time in the field. However, this finding may be a function of age rather than expertise (where older scientists may be less likely to accept AGW; in the general public, older individuals are more politically conservative and less likely to accept AGW, Ballew et al 2019) but this study does not distinguish between these cross-correlated factors. The strength of the relationship between expertise and acceptance of AGW was higher for more stringent definitions of expertise such as objectively confirmed climate experts based on the number of climate-relevant publications.

Our findings suggest that there is value in utilizing multiple definitions and measures of scientific expertise since each measure of expertise has some limitations. For example, some early career scientists may be very knowledgeable and list climate change as an area of expertise but may also have few years of experience in the field and few publications. Other early career scientists might list relatively few years of experience, but prolifically publish a large number of papers. Alternatively, some later career scientists might indicate many years of experience, but may not have recently published many papers, especially if they have transitioned into a more policy-focused position with little time for research. Moreover, we demonstrate the value of supplementing self-reported measures of publication records, which are prone to biased recall, with content analysis to objectively verify individual publication records. Future research should continue to develop new and innovative techniques to define and measure expertise that leverage multiple methods. We acknowledge that there is a need for further conceptual work on who is labeled as a climate 'expert' in order to consistently report the level of agreement on AGW.

Our results also indicate that the scientific consensus on human-caused global warming has increased since the survey of geoscientists conducted in Doran and Kendall Zimmerman (2009). The finding of a strengthening consensus is consistent with the increasing trends found in Cook et al (2013) and Shwed and Bearman (2010). However, one limitation of this study is that while both the samples in Doran and Kendall Zimmerman (2009) and this study were taken from equivalent sources (two editions of the AGI directory), the survey samples from 2009 to 2020 are not identical.

Given the persistent gap between expert consensus on AGW and public understanding, it is imperative to strengthen efforts to engage and educate people about the scientific consensus on climate change. Such efforts are essential to helping our society make more informed decisions about how to stabilize our climate.

We thank Lindsay Drury for manually scanning all of the pages of Wilson (2018). We thank Sarah Anderson for assistance with compiling publication data from Web of Science for the group that self-reported more than 50% publications in climate change. We would also like to thank the five volunteer coders from George Mason University who helped with the abstract review analysis. This study was unfunded.

The data that support the findings of this study are available upon reasonable request from the authors.

The authors submitted the proposed research to the George Mason University Institutional Review Board (IRBNet number: 1457552). All authors took social and behavioral research trainings at their home institutions prior to the study and received certificates of course completion. The project was also submitted to the Open Science Framework (OSF) public registry and can be found at https://osf.io/d97fy/. Consent was obtained from the participants at the start of the survey in Qualtrics (see supplementary materials section S1 for survey design).