Loss of sustainability in scientific work

When asked by a student, 'Dr Einstein, are not these the same questions as last year's (physics) final exam?', he replied: 'yes; but this year the answers are different'. To Einstein, it was very clear that his field was evolving at a rapid pace and that new scientific discoveries outdated historical knowledge. This observation might be even more relevant today. Since Einstein's days, the global scientific output, measured in the number of publications per year, has grown exponentially. The number of articles published every year in all of physics increased a thousandfold from 200 papers in 1900 to about 200 000 in 2010 [1, 2]. The total scientific output is about ten times higher than in physics [3].

Scientific work usually builds on pre-existing knowledge, which is acknowledged through citations in publications. Only a small fraction of this knowledge is actively accessed and actually cited [4]. We find that less than half of all papers published in American Physics Society (APS) journals received more than one citation during the last ten years. Most works become outdated quickly, are never looked at, are forgotten and never mentioned again. This is especially true for the increasing number of non-reproducible papers that do not contribute knowledge sustainably [5, 6]. On the other hand, there are scientific milestone papers—typically exceptional works—that continue to attract citations, even after decades [7]. What emerges from the process of knowledge creation, selection, and forgetting is a 'living' or active body of knowledge that is actively used—ideally—for pushing the scientific Frontier forward. The active knowledge is a tiny subset of all the existing knowledge stored in papers. Its extent can be estimated using citation data.

Large-scale publication and citation data has become increasingly available in recent years. It opens up the possibility to reconstruct the history of science (production and reception) on the granular basis of individual publications. In particular, it becomes possible to trace citations in so-called citation networks.

There are different types of citation networks. In its simplest form, a citation network, M_ij , captures which paper j cites paper i. If paper j cites paper i then M_ij = 1, otherwise M_ij = 0. These networks should not be confused with other popular citation based networks such as the co-citation network or the bibliographic coupling network. The co-citation network, C_ij = MM^T, contains the information of how often papers i and j appear together in the reference list of another paper [8]. C_ij = 2 means that there exist two papers that cite both, i and j. The bibliographic coupling network, B_ij = M^T M, on the other hand, states how many references papers i and j have in common [9]. Every paper, i, is uniquely associated with a publication date, t_i , which makes it possible to bring in the time component of citations.

There are immediate lessons that have been learned from citation networks that are important in our context. Early contributions date back to the 1960s [10] where citation networks were pioneered by investigating links between scientific publications. Along with other insights, most strikingly it was noted, that the percentage of papers receiving n citations in any year approximately scales as a power law in citations n. It follows that most papers receive a small number of citations, while a few papers in the 'fat tail' of the distribution receive many. This observation has been confirmed [11, 12]. In follow-up work, an explanation was offered in terms of 'cumulative advantage' [13], stating that the probability for a paper to get cited is proportional to the number of citations it accumulated up to that point in time. This process is also called preferential attachment [14]. It has been studied in depth and confirmed by a growing number of authors, both in a theoretical context and in the context of citation networks [15–21].

While preferential attachment is a plausible mechanism for understanding the structure of citation networks, recent studies suggest that preferential attachment alone does not result in accurate predictions of several features of citation networks [22, 23]. Other factors need to be taken into account. Timing of publication has been found to be a contributing factor and a first-mover advantage was proposed along with preferential attachment [24]. If a work is among the first in an emerging field, then there is not much competition, and it may receive more citations. It was noted that papers that receive citations unusually quickly tend to continue to receive more citations than usual, even though they might not be the most cited papers at the time [25].

Time not only plays a role in the success of a paper, but also in its 'decline'. In this context, the notion of 'half-life of literature' has been introduced some time ago [26]. It is defined as the time during which the second half of all papers that are still being cited were published. This time is rather short, because most papers cite the recent past [26]. This fact was also observed in [7, 10]. In contrast to the half-life of nuclei, the 'half-life of literature' is not a constant, but depends on the overall growth of the citation network. For that reason, citation networks must always be considered against the exponential growth of the number of publications. In physics, this growth is characterized by a doubling time of about 11.8 years [2]. In [2, 27] it is noted that the number of publications per author (productivity) has practically not increased during the last century while the average number of authors per paper has increased [2]. The observed growth in the number of publications can therefore be largely attributed to the growth in the overall number of scientists and research funding. From the existence of a half-life of literature and the growth of the network follows, immediately, that the distribution of the age of citations (time between the publication of the citing and the cited paper) is highly skewed towards more recent papers. This general trend also persists if one accounts for the exponential growth of the number of articles published each year [7]. For this reason it is apparent, that there is a strong preference of papers to cite more recent works, an effect that is often referred to as aging [28, 29].

Preferential attachment in combination with aging are the two mechanisms that form the basis for a number of mathematical models that reasonably explain citation networks and their dynamics. In [30], an evolving network model with preferential attachment and a power law aging of nodes is considered. It demonstrates a strong first mover advantage. Another model with preferential attachment and power-law aging [31] shows that if aging is included in the model, the associated degree distribution of the resulting network deviates from a strict power law. They confirm their findings on a Hollywood actor network. In [22] an author-paper network is simulated, where authors can read, write, and cite papers, and search for references in a recursive manner. Aging is included by fitting a Weibull distribution to the time-dependent attachment probability of papers. This model explains the deviations from strict power laws in the degree distribution and is supported by empirical data from articles of the Proceedings of the National Academy of Sciences (PNAS). A stochastic model of citation dynamics that builds on [22] combines preferential attachment and aging with a recursive search and copy mechanism [32]. The main result is a non-linearity in the network growth that allows for individual papers to achieve diverging citation trajectories. The rate at which both, patents and scientific articles, acquire citations is studied in [33, 34]. There, this rate is modeled with power law preferential attachment and exponential aging. Strikingly, in this model, the aging component is completely independent of the preferential attachment. Yet another model for patent citation networks assumes the probability to cite a patent to follow linear preferential attachment, as well as power law forgetting with an exponential cutoff [35]. It presents a method to determine this probability from empirical data and shows that the scaling behavior can be explained by the combination of preferential attachment and aging. In all the mentioned models, preferential attachment and aging have an impact on the fate of a paper. Some models allow for short-term performance predictions (citations) [25, 32, 34]; some also do predictions on longer horizons [28] of individual papers.

What has been studied in much less detail is the dynamics of scientific work being forgotten. For how long is scientific work relevant? How long will it take before today's work is forgotten? Which works have the potential to become milestones, and under what conditions does this happen? Can these conditions be identified by seeing papers not in isolation, but as a part of the citation network? What has been missing in this context so far is a model that focuses on the forgetting aspect in the context of a growing citation network. We want to add that missing piece to understand the dynamics of forgetting in order to better estimate the future state of the system as a whole. From that, we can then understand how certain papers stay relevant for decades, while most papers are forgotten.

To understand the forgetting process more quantitatively, we introduce a variable to represent the total number of citations originating from papers published in year i that cite papers published in year j,

$$\begin{equation}{c}_{{t}_{i}{t}_{j}}\equiv \sum\limits _{k,l}\begin{cases}{M}_{kl},\quad \hfill & \text{if}\;\text{publication}\;\text{year}\;\text{of}\;\,k={t}_{j}\hfill \\ \hfill & \text{and}\;\text{publication}\;\text{year}\;\text{of}\;\,l={t}_{i},\hfill \\ 0\quad \hfill & \text{else}\hfill \end{cases}.\end{equation} \tag{ 1 }$$

This is shown in figure 1, where each dot represents a paper published in a particular year and each arrow represents a citation from one paper to another.

Zoom In Zoom Out Reset image size

In figure 2(a) the values of ${c}_{{t}_{i}=2015,{t}_{j}}$ (citations per year from papers published in 2015) are shown (blue) for all American Physical Society journals that include about 600 000 papers published between 1893 and 2015 and around 5000 000 citations. Next, we look at a hypothetical situation, where all papers have the same probability of being cited, independent of their age. We call the corresponding variable ${\bar{c}}_{{t}_{i}{t}_{j}}$ (red dashed line). It is given by ${\bar{c}}_{{t}_{i}{t}_{j}}=\frac{{N}_{{t}_{j}}}{N}{\sum }_{{t}_{j}}\,{c}_{{t}_{i}{t}_{j}}$, where ${N}_{{t}_{j}}$ is the number of papers published in year t_j , N is the total number of papers published in all years $N={\sum }_{{t}_{j}}\,{N}_{{t}_{j}}$ and the sum represents the total number of references of papers published in year t_i . We can see that ${\bar{c}}_{{t}_{i}{t}_{j}}$ is directly proportional to the number of papers published in every year and can be used as a normalization factor, to account for the growing number of publications. By dividing ${c}_{{t}_{i}=2015,{t}_{j}}/{\bar{c}}_{{t}_{i}=2015,{t}_{j}}$, we get a measure that is independent of the number of papers published each year. We call this resulting curve the 'forgetting curve' of the APS, shown in figure 2(b). Note a general trend of a strong preference towards citing papers that are not older than ten years. In a randomized scenario, where new publications would cite other publications with equal probability, the forgetting curve would fluctuate around a constant value of 1. For simulation results on ${\bar{c}}_{{t}_{i}{t}_{j}}$, see SI 1 (https://stacks.iop.org/NJP/24/053041/mmedia). Note that spikes in the forgetting curve coincide with the publishing years of milestone papers. These heavily cited papers stand out from the bulk of papers and cause the corresponding year to be over-represented. We find that in years, where a milestone paper was published, these papers, while making up only 0.03% of the papers published, attract on average 14% of all citations to papers published in that year. For more details, see SI 2.

Zoom In Zoom Out Reset image size

The forgetting curve can be used to infer the age structure of the literature new knowledge is built on. We explain the curve with the help of a simple model of citation dynamics that takes into account in different ways the bulk of publications and the exceptional works.

To model the dynamical process of citing literature, we devise a simple generative model. We consider the case where a new paper i is published at the time t_i and ask how likely will the paper i cite the paper j? To answer it, we measure the function that modulates the probability of attracting citations for individual publications. This function is based on the age of the referenced paper j, t = t_i − t_j , and the number of citations the paper j received in the past, up to time t_i . We call this function the attachment kernel, f(k, t). Assuming that the underlying mechanisms of how we choose our references do not change over time, the dependence of t_i and t_j of the attachment kernel can be condensed into one variable t. To measure the attachment kernel, we adapt a method used in [19] and apply it to the 120 years of empirical citation data of the APS; see SI 3. The resulting kernel can be approximated well by a linear function in k and a power law decay in time,

$$\begin{equation}f(k,t)=(k+1){(t+1)}^{-\alpha }.\end{equation} \tag{ 2 }$$

The exponent α cannot be completely decoupled from the number of citations k, as it shows a slight dependency α = α(k). However, we show that for the bulk of papers, α can be well approximated as a constant from the empirical data by its mean value over all observations. For the mean, we find a value of $\bar{\alpha }=1.42$. We confirm the linear preferential attachment in k found in other works [21]. Here, the offset by 1 represents the finite chance of a publication currently without citations to get cited for the first time. The temporal term represents forgetting.

To this point, the model covers the bulk of average papers. However, as noted before, there are milestone works that are clearly exceptional. These works are causing visible spikes in the forgetting curve figure 2(b) and thus follow a different attachment kernel. If we assume, that the underlying preferential attachment mechanism is the same also for milestones, we can determine their attachment kernel by measuring the value of the exponent α_m of any individual milestone paper m. For our analysis, we define milestones as the top 30 most cited papers in the APS, excluding review papers. For each of these papers, we measure the exponent α_m based on the waiting times between events, where these papers receive citations. For details, see SI 4. The attachment kernel for milestones has the form:

$$\begin{equation}f(k,t,{\alpha }_{m})=(k+1){(t+1)}^{-{\alpha }_{m}},\end{equation} \tag{ 3 }$$

where α_m can be different for every individual milestone, m. With these ingredients, we can now define the generative model as follows. An exponentially increasing number of papers is published over time. Each of these papers, i, cites multiple other papers j that were published before. The probability of j to be cited by i is proportional to the attachment kernel f(k, t), where k is the number of citations that paper j has at that point in time and t is the time difference in years between the publishing times of paper i and paper j. To normalize the probability, f(k, t) is calculated for all average papers and f(k, t, α) for all milestone papers at each point in time. Here, we define milestones as the top 30 most cited papers in the APS, excluding review papers. Following this model, one can generate a simulated citation network.

3.1. Null model validation

To test the model, we generate a randomized citation network that serves as a null model. To generate it, we take the publishing dates and out-degrees of papers from the APS, but omit the information about which paper cites which. The process of citing is replaced by a random process, where probabilities for individual papers to receive a citation are proportional to their respective attachment kernel, f(k, t). We include the publishing dates and individual attachment kernels f(k, t, α_m ) for milestones in this model. We then assign the references paper by paper, ordered in time, until we have reconstructed the full citation network. For details, see SI 5. On the reconstructed network, we measure the forgetting curve. In figure 3 the resulting forgetting curve for the null model (red dashed line) is compared to the empirical forgetting curve (blue solid line) that was introduced in figure 2(b).

Zoom In Zoom Out Reset image size

The general features of the null-model are in good agreement with the empirical values, especially for the more recent decades. To assess how far the null model deviates from the empirical data, we calculate the mean absolute percentage error (MAPE) on the forgetting curves. This value is calculated by taking the absolute value of the relative deviation between null model and empirical data for each data point. The resulting values are then averaged to arrive at the MAPE. When comparing the null model to empirical data, the MAPE is below 6% for the last 30 years. It increases for the earlier years, up to values of 35% if the whole 120 years of the APS, including early years with low numbers of publications and high volatility, are taken into account. The low MAPE for recent years implies that the null model accurately reproduces the forgetting curve of the real APS. Note that also some spikes caused by milestone papers are present in the null model. The magnitude of these spikes is captured well for years after the 1990s. For earlier years, low publication numbers are resulting in higher volatility. Nevertheless, the impact of the largest milestones is reflected moderately well, with exception of the year 1935. In this year, a major milestone of physics was published by Einstein, Podolsky and Rosen, which remained almost unnoticed for decades, serving as a prime example of a 'sleeping beauty' [36]. This unusual citation pattern leads to an underestimation of citations in the null model. For more details, see SI 6. Overall, the similarity of the null model and the empirical data implies that the attachment kernel was measured accurately and captures the essential underlying citation dynamics.

3.2. Out-of-sample prediction

3.3. Predicting milestones

We next test the predictive power of the model for milestone papers by asking how predictive the exponent α_m is for a paper becoming a milestone in the future. We first define a set of potential milestones as all those papers that had at least 200 citations by the end of 2005, a condition that is matched for 814 papers in the data set. Next, true milestones are defined as those papers that receive at least 50 citations in the year, ten years after they received their first 200 citations. At this citation rate they would be ranked among the top cited papers within a few decades. For each potential milestone, m, we measure the exponent, α_m , based on the first 200 citations. If this exponent is small, the forgetting can be over-compensated by the preferential attachment mechanism. Thus, if the measured exponent is below a threshold, α_m ⩽ α_c, we predict that the paper will receive more than 50 citations in one year after ten years.

The predictive power of this binary classifier is seen in the receiver operating characteristic (ROC) curve in figure 4(a), where we find an area under the curve (AUC) of up to 0.89. The high value for the AUC suggests that α_m is a good predictor of the future success of a paper, with high specificity and sensitivity. While our definition of true milestones was somewhat arbitrary, we confirm that a wide range of thresholds results in reasonably high AUC values and the choice of threshold thus does not significantly impact our results. Our definition of potential milestones is based on the requirement for a high number of data points in order to get a low-variance measurement of the exponent α_m . For details, see SI 4. Note that on the left side of the ROC curve in figure 4(a) there is a convex section that indicates, that there are some papers that initially have a very small α_m exponent but fail to stay popular in the long run and thus are erroneously classified as milestones in our model. We hypothesize that the convex section is caused by papers with a sudden but brief success—'flash in the pan' papers.

Zoom In Zoom Out Reset image size

For the prediction, we estimate an individual constant α_m based on the first 200 citations of each potential milestone. If instead of taking the first 200 citations, these exponents α_m are derived from several hundreds of citations, a closer look reveals that many of the exponents come with a positive slope as a function of the number of citations k. This indicates that for milestone papers, the attachment kernel cannot be factorized as f(k, t, α_m ) = g(k)h(t, α_m ). For that reason, if one is interested in predicting citation trajectories of individual papers with more than a few 100 citations, α_m should not be taken as a constant but rather as a function of k. We find, that this function is best approximated by a linear function ${\alpha }_{m}(k)={\alpha }_{{m}_{0}}+\beta k$, which can be estimated from the data in an analogous way. For details, see SI 4.

3.4. Predicting future citation landscapes

3.5. Empirical citation trends

In this paper, we focused on understanding how scientific work is being utilized or forgotten. In empirical citation networks, we observe two main mechanisms: a linear preferential attachment and a power-law temporal forgetting. We capture these empirical findings in a generative model in which newly published papers cite older papers with a probability proportional to an attachment kernel f(k, t). The kernel reflects two main factors: the cited paper's academic age and the number of citations it already received. This result implies that an article's probability of receiving a citation grows linearly with the citations already received and decays with its publication age as a power-law, characterised by an exponent α.

While there is a strong preference towards citing recent papers, some milestone works –potentially of exceptional impact—continue to attract citations even after decades. These singular milestones, m, are characterized by an exponent α_m < 0.85. In comparison, the exponent for the bulk of non-exceptionally successful papers is about α ∼ 1.4. Based on α_m , one can predict whether a paper will continue to be successful beyond the age of ten years. The corresponding ROC analysis yields an AUC of 0.83, confirming that the exponent is a good predictor indeed.

The model reproduces the highly skewed distributions of the number of citations and the age of references observed in other works. In particular, the forgetting curve that results from the model shows a strong tendency to cite recent papers [10, 26], while older papers fade progressively in oblivion [7, 10, 26]. The spikes observed in the forgetting curve highlight the impact of milestone papers that are remembered for a long time [7, 32].

The linear preferential attachment mechanism in our model confirms the linear dependence on the number of citations, originally found in [21]. The second mechanism we find is power-law-like forgetting. While forgetting, or aging, is a mechanism found in most models of citation dynamics, its detailed functional description remained controversial [22, 29–32, 34, 35]. In [22] an author-paper network yields an attachment kernel with an aging component that follows a Weibull distribution. [32, 34] both investigate the citation rate of individual papers and find exponential aging, while [35] studies a patent citation network and finds power-law aging with an exponential cutoff in intrinsic time, where each new patent represents one timestep [30, 31], both find power-law aging in intrinsic time. [30] applies it to the context of citation networks while [31] applies it to a Hollywood-actor network. In [37], the attachment kernel of online media content is described by a model based on preferential attachment, modulated by an exponential forgetting, for which exponents are experimentally determined from data. Our model finds a power-law-like forgetting mechanism with an exponent larger than one for the bulk of average papers. For milestone scientific papers, we find values below one. Power-law-like forgetting implies that papers are forgotten relatively quickly and only have a limited time to attract attention. However, power laws also indicate that old papers keep a chance to get cited. This result enables a few notable publications that attracted much attention early on to continue attracting citations for a long time.

There are several limitations to our model. In particular, it relies on the assumption that the way scientific literature evolves will not change fundamentally. This includes the assumption that the scientific output continues to grow exponentially, an assumption that might not hold forever [38]. This observation limits the prediction of the number of citations a paper will receive in the future, simply because the total number of citations is directly proportional to the number of articles published. However, our results only partially depend on this assumption, as the general shape of the forgetting curve is also recovered well for sub-exponential growth and even for linear growth. On the other hand, our model also assumes no fundamental changes in how citations are chosen. This assumption might not hold if, for instance, disruptive new technologies are introduced and adopted globally, as had been the case with the internet.

To forecast citation trajectories of individual highly cited papers, our model provides a simple indicator based only on the citation network. For higher accuracy, additional parameters, such as the popularity of the authors, the size of the field, or the novelty of the publication, should be taken into account [2, 24, 25, 28, 39–43]. However, even taking these factors into account, there remains uncertainty due to unpredictable events such as a highly relevant discovery that went unnoticed for a while, termed 'sleeping beauty' [44] or the rediscovery of a paper outside its original domain, called exaptation [45].

Restricting our model to only age and number of citations makes it easily transferable to other domains. The possibility that the life-cycle of scientific literature might be different in physics than for other fields like biology, medicine, or philosophy, and even in non-scientific domains such as the music and entertainment industry, is an interesting research direction.

Our model allows for projecting the current citation ecosystem into the future and investigating its sustainability. Through the empirical forgetting kernel, and under the assumption that the number of papers will keep growing exponentially, it is possible to simulate possible future citation networks. For instance, one learns that 95% of today's (and yesterday's) work will be outdated before the end of the next three decades. It is this finding, and the fact that papers are receiving fewer and fewer citations on average and have decreasing chances of becoming a milestone, that has substantial implications both for individuals and for scientific policies. Partially, the observed dynamics might be attributed to an increasing fragmentation in the system, driven by the emergence of sub-fields. The fragmentation becomes evident, if one looks at the increase in the number of journals in the APS over the last decades. In this fragmented system, papers might become isolated inside of sub-fields and reach smaller scientific audiences on average. On the other hand, a much more worrisome interpretation of our findings hints towards an increasing quantity-over-quality dynamic, where increasingly short lived publications receive little attention and have diminishing chances of becoming works of great impact.

The data generated and/or analysed during the current study are not publicly available for legal/ethical reasons but are available from the corresponding author on reasonable request.

The authors declare no conflict of interest.

NR, ST, VS, VL conceptionalized the work and designed the model. NR, MF and WS did the data preparation. NR and WS did the implementation. NR did the analysis. NR and ST produced a first draft, all contributed to the final paper.

This work was supported by the Austrian Research Promotion Agency FFG (Grant Nos. 882184, 873927).