Evaluating the impact of Big Science/research infrastructures

Since the evolution of the modern Big Science laboratory in the twentieth century (Heilbron et al 1981, Hermann et al 1987, Seidel 1992), they have consumed a disproportionate amount of public funds relative to their numbers (Webre 1988, Lefevre 2009). Modern day Big Science often requires significant investments from one or more national governments. It is therefore not surprising that such stakeholders will ask what the return is on these investments so that future Big Science projects can be justified to taxpayers. From the perspective of scientists, the return on investment is in terms of new understandings of fundamental phenomena; yet the traditional answer to such stakeholder queries is usually articulated in terms of economic impact (Autio et al 1996, 2004, Scarrà and Piccaluga 2020). This chapter seeks to outline the breadth of Big Science impacts beyond these traditional economic measurements. It begins by looking at the traditional methods for capturing the economic return on Big Science before looking at how the academic literature articulates how to measure the scientific impact and productivity of Big Science before looking at its broader societal impacts. However, it is broadly acknowledged that direct measurement of the effects of Big Science is not feasible, and most metrics capture the by-products of knowledge production (Cronin and Sugimoto 2014). As will become apparent in this chapter, there has been a transition from more traditional measures associated with trying to capture the by-products of knowledge production towards much broader metrics that seek to quantify the outcomes that accrue to society.

As noted above, one of the most frequently used justifications for Big Science is to emphasise the economic benefits. These include direct economic benefits as a result of concentrating spending within the geographic region. These benefits include direct spending on construction as well as the clustering effect from the demand for high technology components (Simmonds et al 2013). Much of the literature examining this topic focusses on the benefits of publicly funded basic research in a generalised sense (Salter and Martin 2001). However, there are still some pockets of literature specifically examining the economic benefits of Big Science which are examined below.

The most traditional method for quantifying the economic return on Big Science investment has generally been to look at patents. Using the somewhat simplistic linear model framework for understanding the return on investment from Big Science, investments in such basic research generate new knowledge that are pulled by the free market through to new products (Godin 2006). During this process the private sector patents this knowledge, so it was traditionally used by governments as a way of understanding how investments were converted to economic growth. There is some evidence that the demand created by Big Science projects can lead to new innovations, particularly in terms of new process innovations. However, while product innovations do occur, they are considered to be relatively less numerous compared to process innovations (Florio et al 2018). Certain authors have sought to capture the economic impact of procurement for Big Science projects by performing patent analysis (Castelnovo et al 2018), which unfortunately was unable to capture these process innovations. Therefore, solely using patents which primarily capture product innovations, may be inadequate for understanding the relationship between Big Science projects and economic growth.

Patent analysis is not the only way to understand the economic impact of Big Science. The laboratory itself can be a major source of demand for high technology products through its procurement policy. One of the hallmarks of Big Science is that many of the experiments can be considered instances of 'super high' technology whereby the technology is discovered in the context of application (Shenhar 1993, Shenhar and Dvir 1996). Such technologies can cause substantial changes across the entire economy. The World Wide Web (WWW), originally developed at CERN as a way of synthesising several pre-existing technologies to identify and access information held on different computers (Berners-Lee 1989), is perhaps the most famous example of such a spillover technology. Such spillovers can occur at any stage in the project and have been categorised into three components: market spillover, where enterprises are pulled into the direct supply chain for the project; network spillover, where the technology diffuses into other economic sectors; and knowledge spillover, where human capital diffuses into the wider economy (Choi et al 2017).

Market spillover is perhaps the most obvious economic impact that can be generated by Big Science projects. In the short-term, Big Science projects naturally create local demand for components for the apparatus. For example, in the cases of ITER and the LHC, these major projects incentivised industry to create new production lines for high tech components in the medium term. It also provided some companies with the platform to stand out in the market and create a long-term comparative advantage over competitors as their greater capacity allowed them to benefit from economies of scale. Big Science laboratories appear to be more willing to give external companies more time to experiment and refine new technologies or to even work with them compared to industrial customers (Autio et al 2004). Given that many Big Science projects could be using a high or 'super high' level of technological uncertainty, having a known customer base for emerging technologies may help to pull them across the gap between the laboratory bench into the market.

In the case of network spillover, the wider economy can benefit from the demand temporarily created by the construction phase of Big Science projects. Often by design, Big Science projects require significant volumes of high technology components with low tolerances for error that can incentivise industry to create new production lines to satisfy this demand (Autio 2014). This leads to two developments. First, as economies of scale mean that the unit cost of such components decreases, other industries may find lower barriers to entry for developing new products using these components (Castelnovo et al 2018). Second, the laboratory helps to create the specifications for the new components that become the standard for future industry (Autio 2014). This has been demonstrated to some degree through the temporary demand and specifications created for niobium-titanium wire for the Tevatron in the 1980s; this helped to make MRI machines technically and financially viable for healthcare (Hesla 2017).

Lastly, in the case of knowledge spillover, this is largely characteristic of the highly educated workforce that characterises Big Science laboratories. This has traditionally been described as the spillover of ideas created in the basic sciences whereby tools and techniques developed in the laboratory can diffuse into the wider economy through industrial participation in Big Science projects and research programmes (Choi et al 2017). However, this may only capture part of the reality as there will also likely be human capital exchanges as a community of practice develops and some workers move from the laboratory into industry and vice versa.

These economic impacts may be measurable at all stages of a Big Science project's lifecycle rather than being restricted to after first physics (Autio et al 2004, Bianco et al 2017). However, most of the research outlined in this chapter does not capture intangible benefits that may accrue to enterprises linked to Big Science projects. This represents something of an opportunity for future research. Comparatively modest enterprises could benefit from the reputational halo effect of being a contractor in a Big Science project. While such benefits may not materialise in either the short or medium term, they could be considered as preparatory work for the future (Shenhar et al 2001).

This section briefly examines the scientific impacts of Big Science projects. Naturally the scientist may view the production of new knowledge or understandings of nature as the principal benefit of Big Science. Yet the scientific impacts of Big Science generally cannot capture the intangible benefit of this and tend to focus on publications. The traditional measurements for scientific productivity and impact depend on the unit of analysis, if the individual scientist is the unit of analysis metrics such as the h-index (Kumar 2009, da Silva et al 2018), journal impact factor (IF) (Archambault and Larivière 2009), and author-level eigenfactor can be used (West et al 2013); these all seek to quantify an individual scientist's productivity and centrality to the field ¹ . Classically one might judge the impact of research to be best judged through the simple measure of citations on the basis that the more citations to an article, the more important the contribution to the field (Kumar 2009). But this type of metric quantifies the scientific impact of the experiments conducted with the apparatus rather than of the apparatus directly. Equally these metrics can only judge the value of the apparatus after project completion, this reality means that scientists can use the sunk cost fallacy to their advantage. Equally these metrics tend to privilege new apparatus that can quickly make new discoveries over mature apparatus that may still be producing valuable results (Bianco et al 2017). As an example, the Tevatron was closed relatively soon after the LHC achieved first physics, despite its experiments continuing to produce valuable data (Oddone 2011a).

While the scientific impacts of Big Science may be obvious to practitioners who work on or with the apparatus, external stakeholders such as funders or member state governments may no longer perceive those benefits quite so readily as the scientific community. It is likely the scientific community will use practical or technical parameters to judge the impact of a completed project (Hallonsten 2016). In the case of high energy physics these technical parameters will be the obvious dimensions of beam energy or luminosity or slightly less obvious technical parameters such as machine reliability or the ability to access the scarce resource of beamtime (Hallonsten 2016). There may be a tradeoff to be made between using sufficiently advanced technology to achieve new experimental possibilities and machine reliability (Hoddeson and Kolb 2003, Hoddeson et al 2008). This may not be the tradeoff that is expected with some scientists believing that a reliable machine may be overengineered and therefore too expensive (Hoddeson and Kolb 2003, Hoddeson et al 2008). Scientists may also articulate the ideal of collaborating to make new discoveries, there can also be an element of 'collaborative competition' between laboratories to be able to lay claim to the 'energy frontier' (Lederman 1983). Equally there is the intangible impact generated by making new discoveries which are extremely important to advancing the scientific field (Abe et al 1995, Aad et al 2012). However, these are relatively difficult concepts to communicate to a government and the general public. Even if they could be communicated in such ways, funders may not necessarily appreciate these impacts in quite the same way as the scientific community.

Qiao et al (2016) classified the scientific effects of Big Science into four categories, which are how the apparatus advances the field which has already been addressed above, how the laboratory develops the organisation and trains future scientists, how the laboratory can act as a networking hub for building relationships with other scientists, and for linking with other industries to create a knowledge ecosystem.

First, in terms of the effect of Big Science projects on the development of human capital, namely future scientists, these effects have been well documented in the literature and are intrinsically linked to the exceptional intellectual environment that characterises Big Science laboratories (GSF 2014). While the traditional and usually false impression of science as the product of a single individual's toil is relatively rare, with many collaborations composed of hundreds or even thousands of scientists, it does offer the opportunity for greater networking (Blankenship 1974, D'Ippolito and Rüling 2019). The evolution of modern Big Science has had some impact on the nature of training for future scientists. Historically, experiments and accelerators were sufficiently small and on a short enough timescale that a scientist could obtain significant experience of all experimental phases from initiation to completion. The modern nature of Big Science, with very long experimental lifetimes, now means that the type of training is much more specialised but is tied to these lifecycles (GSF 2014). For example, a scientist trained during an experiment's construction phase will receive a very different training compared to a scientist trained during data analysis. Second, Big Science laboratories act as central nodes within a wide network of associated universities, private research institutions, and other private sector organisations (GSF 2014, Qiao et al 2016). This can create a clustering effect in the local geography as wider agglomerations develop (GSF 2014, Qiao et al 2016). This allows the laboratory to act as a focal point for the community as part of a local 'triple helix' system of universities, industry, and government which can support the circulation of talent within the local knowledge ecosystem (Etzkowitz and Leydesdorff 2000). Third and finally, the laboratory also acts as a networking hub for building relationships between stakeholders (Qiao et al 2016). While this is linked to the clustering identified above, the Big Science laboratory acts as a non-human artefact for focussing human minds from a variety of different stakeholder groups. This is particularly true during the construction of a new piece of apparatus (Qiao et al 2016).

The societal impact of Big Science is a somewhat contested concept in the literature; while some authors have linked Big Science to the development of human capital and new innovations that can benefit society (Magazinik et al 2019), others have said that the evidence is less clear cut, in particular when the scope of analysis is limited to the local level (Gastrow and Oppelt 2018). Some authors have examined the role of using science to address societal needs (Ciarli and Ràfols 2019). Given its fundamentally basic nature, it is not surprising that there is relatively little literature focussing on the use of Big Science laboratories for addressing societal needs. Although societal impact may not be the primary mission of the laboratory, there is evidence of the impact that Big Science laboratories have on society, this is addressed in this section.

Giudice (2012) examined the potential benefits to society of Big Science, and classified the societal benefits according to (i) the benefits created by clustering significant financial and intellectual resources together, (ii) the spinouts from technological developments during the R&D phase of the project, (iii) the close relationship between the laboratory and the private sector for component manufacturing necessitating new manufacturing techniques, (iv) the opportunity for countries to participate in scientific experiments where they lack the resources to do it alone and for the Big Science project to be used as a vehicle for promoting international links—often referred to as 'science diplomacy' (Ruffini 2017, Robinson 2019), (v) the training opportunities for early career researchers, and (vi) the intrinsic benefits of making new discoveries and uncovering new knowledge. Many of these benefits that Giudice (2012) regarded as societal have already been examined above, indicating that the societal benefits of Big Science can be conflated with economic or scientific benefits. Nonetheless, this can be considered a framework for future work.

One of the most obvious societal benefits from investment in Big Science surrounds the development of human capital. As noted above, the laboratory lies at one focus of a scientific ecosystem, with a wide variety of users at all career stages often based at academic institutions who visit to conduct experiments (Leon and Elias 2010, Qiao et al 2016). Each one of those users are technically competent individuals in their area of expertise trained at their home institution to be able to work on the machine and to contribute to the scientific method. As such it can be argued that the Big Science project acts as a hub for the development of social capital across the community and for the mass transfer of tacit knowledge between early career researchers and more experienced individuals (Hislop 2013, Qiao et al 2016).

One of the most obvious benefits that has accrued to society as a result of high energy physics has been the discovery of new radiation-based medical techniques. Such medical applications of Big Science have been explored since its beginning in the 1930s. The link between what were then the nascent fields of nuclear physics and nuclear medicine can be best exemplified by the early links between the Berkeley Radiation Laboratory and the Donner Laboratory. While Ernest Lawrence's Radiation Laboratory sought to understand the inner workings of the atom, John Lawrence's ² Donner Laboratory aimed to direct these high energy particle beams to understand human physiology and to treat clinical conditions (Williams 1999). The first use of a radioactive isotope to treat human disease took place in 1936 where radioactive phosphorous-32 created in the Berkeley cyclotron was used to treat a patient with leukaemia (Williams 1999). This synergy between high energy physics and nuclear medicine continued such that in the 1970s, a collaboration between Fermi National Accelerator Laboratory and a private medical provider allowed patients to receive treatments using the laboratory's linear accelerator's neutron beam capabilities (Awschalom et al 1979). Even hadron and anti-hadron therapies were first developed using adapted linear accelerators from high energy physics (Dosanjh and Bernier 2018).

The societal impact of Big Science also includes its contribution to human development. While most Big Science laboratories tend to be constructed in more developed countries, the case of the Square Kilometre Array (SKA) offers an interesting example of using a Big Science project as a means to develop both regional infrastructure and human capital. The central mission of the SKA was to construct a radio telescope with a collecting area equivalent to one square kilometre. However, one of the challenges with radiotelescopes is that the cost to build a single very large radio telescope is much greater than an array of many small telescopes (Dewdney et al 2009). Two sites have been selected for construction of these radiotelescopes—the first being in Western Australia and the second being in an area not traditionally known for radio astronomy, namely the Karoo Desert in South Africa (Walker 2019). While the benefits that have been created as a result of significant investments in that region of South Africa are relatively traditional and could be covered in the economic benefits section above, a new phenomenon was identified which unusually led to a broadening of SKA's mission. This was the massive development of human capital in South Africa in both direct and indirect forms. Essentially it was the creation of a network spillover effect and its direction to develop an entire region throughout the process and entire economic sectors, creating a new dynamic in Big Science as a platform for development. Some of those who worked on SKA described it as if someone had taken a high-pressure bottle and removed the cap—all of the human potential was there, and the project served to focus that potential and provide it with an outlet (Dewdney 2019). This was also investigated by Gastrow and Oppelt (2018), who confirmed that there are benefits to the national innovation system and human development at the national level. However, the benefits at the local level in the Karoo could not be identified as there was no absorptive capacity for the local economy to engage with the advanced technologies deployed on-site. In fact, Gastrow and Oppelt (2018) argued that there was evidence that the construction of SKA in the area had a short-term detrimental impact as Internet infrastructure was constrained. The local effect of SKA was subsequently investigated by Gastrow and Oppelt (2019) who noted that there was a gap between how local communities hoped SKA would benefit them and what benefits SKA could actually deliver while staying within its project scope.

However, quantifying the scale of the benefits that accrue to society may require very different metrics drawing on interdisciplinary forms of expertise. For example, how one measures the societal benefit as a result of these new medical technologies could be captured in terms of the additional years of life as a result of treatment adjusted for the quality of that extension (Vergel and Sculpher 2008, Devleesschauwer et al 2014), but obviously one does not necessarily need to quantify the benefits to the individual from a significantly longer or improved lifespan which may not be quantifiable.

The notion that the impact of Big Science cannot be captured by any single one of these dimensions or indicators is not an innovation but has existed since the 1980s (Martin and Irvine 1983, Leydesdorff 2005). Instead, the academic literature has suggested that these be used in combination as partial indicators, in line with other domains that emerged during this time period (Martin and Irvine 1983), but this itself may be inadequate for capturing the real return on investment that can be easily articulated. As will be seen below, there is a trend in progress away from indicators towards using scientific investments to achieve certain outcomes.

Over the past ten years, a new dynamic in research evaluation has emerged that is still being rolled out across government portfolios. So far, the evaluation methods examined for this chapter have tended to focus on specific outputs from Big Science projects, whether in terms of scientific or economic outputs. However, governments have shifted in focus from using such outputs as the metric for judging the return on Big Science investments towards using investments to achieve certain outcomes. This shift from an output-oriented approach towards a goal-oriented approach can be seen above in the case of SKA in South Africa, where it has been used to build up human capital and develop a region (Bhogal 2018). This raises the challenging question regarding how to measure such returns on investment. However, the tools for the creation of these goals and judging whether the goals have been met are still relatively new. Some experimental proposals have been to combine traditional metrics with more narrative descriptions of the benefits from scientific funding. These proposals involve combining narrative descriptions of the beneficial outcomes with more traditional metrics that examine the collaboration's construction by mapping the diversity within the collaboration in terms of their disciplinary interrelatedness, their geographic distances, the relative similarities of their respective working environments, the social capital between collaborators, and the level to which hierarchical structure help or hinder collaboration.

A particularly tricky aspect of Big Science for policymakers is evaluating when a piece of apparatus has reached maturity whereby its productivity is at a stage where the increasing maintenance costs no longer justify the operating expenses. The decision to continue investing in a mature apparatus can be very challenging when new apparatus have been commissioned with new capabilities (Cho 2010). Compounding these challenges is the significant risk of community outcry if there is the perception that a useful facility is being closed to satisfy budgets (Oddone 2011b, Reich 2010, Samuel Reich 2011). Traditional methods for evaluation tend to also create further complications as the most eye-catching metrics are displayed early during the operating life of a Big Science project or at specific endpoints. One suggestion to try to rectify this situation is to capture data between these endpoints at designated waypoints that can capture both scientific and economic impacts (Bianco et al 2017). These are described in terms of knowledge, commercialisation, or universal waypoints and broadly capture knowledge production by synthesising some of the metrics discussed above (Bianco et al 2017). These capture publications or patents, commercialisation grants or licensing agreements, or even whether the facility has secured additional funding to continue research (Bianco et al 2017).

As this chapter has demonstrated, the popular justification used by scientists for Big Science investments has tended to focus primarily on the economic impacts that they generate. This is not surprising as the major funders for Big Science projects tend to be national governments who must justify their investments to the citizenry and these types of impact are often the most obvious returns on investment. However, there can be very long lead times before these benefits become apparent in the real economy. But it is perfectly reasonable and often accepted by the public that Big Science investments can be substantially justified on scientific grounds alone. This has historically been the case during the early days of Big Science in the early post-war period. Since the 1970s Oil Crises in the West there has been a tendency to play down the significance of scientific impact as a justification for Big Science investments (Hoddeson et al 2008). Amongst the new trends that have developed over the past 20 years has been an increased focus on the societal benefits that arise from Big Science projects, whether in terms of human capital or as using the project as a way to develop a region. This was observed in the case of the Square Kilometre Array in South Africa, which is unusual amongst Big Science projects in that it is being constructed in a less economically developed country and the project is being used as a way to develop human capital to construct and service the final apparatus (Bhogal 2018). The topic of evaluation of Big Science projects will continue to develop and amongst recent developments are new efforts to combine traditional metrics with more narrative descriptions that can capture benefits from increasingly interdisciplinary research (Dave et al 2016).