Coevolutionary dynamics of aspiration and strategy in spatial repeated public goods games

The evolutionary dynamics remain largely unknown for spatial populations where individuals are more likely to interact repeatedly. Under this settings, individuals can make their decisions to cooperate or not based on the decisions previously adopted by others in their neighborhoods. Using repeated public goods game, we construct a spatial model and use a statistical physics approach to study the coevolutionary dynamics of aspiration and strategy. Individuals each have an aspiration towards the groups they are involved. According to the outcome of each group, individuals have assessment of whether their aspirations are satisfied. If satisfied, they cooperate next round. Otherwise, they switch to defecting. Results show threshold phenomenon for harsh collective dilemma: cooperators sticking to high levels of aspiration can prevail over defectors, while cooperators with other levels are invariably wiped out. When the collective dilemma is relaxed, cooperation is greatly facilitated by inducing a high level of diversity of aspiration. Snapshots further show the spatial patterns of how this coevolutionary process leads to the emergence of an optimal solution associated with aspiration level, whose corresponding strategy are most prevalent. This optimal solution lies in one and the highest aspiration level allowed, and depends on the intensity of the social dilemma. By removing the memory effect, our results also confirm that repeated interactions can promote cooperation, but to a limited degree.


Introduction
The essence of many problems concerning human cooperative behaviors can be captured by a social dilemma of generating benefit to others at a cost or refrain from doing so while still harvesting the benefit produced due to other's efforts [1][2][3][4]. Rational reasoning dictates that we should take the second action-free-riding. In reality, however, cooperation prevail in the real world [1,3]. Understanding the emergence and persistence of cooperative behavior is a major scientific puzzle which can be analyzed by the framework of evolutionary game theory [5,6]. In this framework, successful strategies with a higher fitness is more likely to spread. Selfish individuals tend to stick to the strategy optimizing their respective fitness, which shall lead the population to the fitness trough as a whole [7][8][9][10][11]. Various mechanisms helping populations escape from the tragedy of the commons have been proposed [3,6,12,13]. One intensively studied is direct reciprocity [7-11, 14, 15].
Several studies have devoted to exploring the evolution of contingent cooperation in group interactions [4,23,[26][27][28][29][30][31]. In [29], it is assumed that conditional cooperators cooperate only if there are a certain number of other players willing to cooperate. Results show that the most cautious cooperators are the ultimate winner of the evolutionary process [29]. Moreover, the authors [30] have explored the effects of tolerance towards defectors on the evolution of cooperation. When only individuals with payoffs above a threshold are able to organize the public goods game, choosing a proper threshold could efficiently enhance cooperation level [31]. Individuals, be they animals or humans, have the instinct of pursuing advantages and circumventing adverse environments [29][30][31][32]. Meanwhile, they have a sense of fairness in mind when making decision [4,33,34]. When interactions repeatedly happen, individuals can base their decisions on previous experiences and timely adjust their strategy according to the assessment of outcome of interactions. Instead of reciprocating towards some specific individuals, one focal individual assesses the group as a whole. Aspiration serves well as a measure of this assessment. When the number of cooperators in the group meets their aspiration, she cooperates next round. Otherwise, she defects. The authors [4] have revealed that fairness, equivalent to a moderate aspiration level, emerges in the well-mixed population of finite size. Occurrence of repeated interactions is more likely in structured populations [35][36][37]. Though increasing effort has been spent on investigating the evolution of cooperation in structured populations [35-39, 32, 40-43], repeated interactions have been rarely dealt with. We shall construct a spatial model and set out to study the coevolutionary dynamics of cooperation and aspiration in the settings of repeated interactions.

Model
Let us consider a spatial population structured by a lattice with periodic boundary conditions. Each node represents an individual. Each individual has eight neighbors (the Moore neighborhood) [8]. Each interaction involves nine participants, the focal individual and their eight neighbors. The interaction is characterized by the repeated public goods game [44,45]. Each individual decides whether or not to cooperate. A cooperator pays a cost (say 1) to bring all participants each a benefit δ·1, where δ=F/9 is the normalized enhancement factor and thus F the enhancement factor. A defector pays nothing and generates no benefit. The interaction continues to L rounds. In the first round, players has no information of history to refer to. From the second round on, players, especially those who have cooperated in previous round, can adjust their strategies based on the outcome of previous round. We refer to figure S1, available online at stacks.iop.org/NJP/20/063007/mmedia, for a typical repeated public goods game. In terms of the possible outcomes of group contribution, these contingent cooperators can be divided into 9 classes, say c M (Mä{1, 2, K, 9}), as the possible number of cooperators in the group ranges from 1 to 9 (including the focal cooperator). Here M represents aspiration level. These cooperators are somewhat analogous to tit-for-tat for pairwise interactions. They cooperate in the first round, and would continue to cooperate in second round if their aspirations are met. Otherwise they would not contribute. It follows the same logic for their strategy adoption in future rounds. Along with these 9 different classes of contingent cooperators, we incorporate the strategy AD (acronym of always defect) to encode unconditional defectors.
At the end of these L rounds of interactions, each individual accumulates their payoff. All individuals at this time synchronously update their strategy and aspiration. Each individual picks up one of their neighbors at random and adopts the neighbor's strategy and aspiration level with the probability given by the Fermi function s s 1 e , where s A and s B are the focal individual and the selected neighbor's strategies, P A and P B the focal individual' and the neighbor's payoffs, respectively [46][47][48]. The parameter β is the selection intensity, defining the degree of contribution of payoff to fitness [49,50].
The simulations are carried out on a population of size 100×100. The population is structured by the lattice with periodic boundary conditions. At start of the evolution, half of the population are randomly assigned to be defectors and the rest cooperators. We first study how iteration without memory affects the evolution of cooperation. They are unconditional cooperators as they always cooperate in each of L-round public goods games. Next, we probe the population dynamics when only one class of conditional cooperators compete with defectors to survive. Cooperators are endowed with a fixed aspiration. And finally, we study the full population dynamics where aspiration and strategy coevolve [51]. At start of the evolution, cooperators and defectors hold the same share of the population. These cooperators each are endowed randomly with one of the nine levels of aspiration. We control that the numbers of cooperators with each level are equal. We would like to emphasize that difference between these three sections lies in how many classes of conditional cooperators are present in the population at the starting point of the evolutionary process.

Effects of unconditional cooperators on the evolution of cooperation
Before analyzing the full population dynamics, we first study how repeated interactions without memory affects the evolution of cooperation. When no memory is involved, all cooperators are undifferentiated. Competition advances between defectors and cooperators. At start of the evolution, half of the population are randomly assigned to be cooperators and the rest defectors. Each individual takes part in nine public goods games, one centered on themselves and the others on their neighbors. Each public goods game repeats as many rounds as L.
If not stated otherwise, interactions take place following this way throughout this study. Figure 1 shows that the fraction of cooperators exhibits a monotonous increasing dependence upon δ, one result widely reported in spatial public goods game [35][36][37][38][39]32]. This result still holds for repeated interactions such as L=3, 5, and 10. One can easily observe that δ required for arriving at the same non-zero cooperation level declines as L increases, and that fractions of cooperators corresponding to L=3, 5, and 10 almost coincide with each other, as a function of δ. We thus claim that repeated interactions itself does promote cooperation in structured populations, but to a limited degree. In fact, as individuals do not change strategy from round to round, the repeated interactions without memory can be transformed into a single interaction with multiplying selection intensity L·β. On this front, our results confirm that strong selection favors the evolution of cooperation in structured populations. When the selection intensity exceeds a threshold, further growth in it barely affects the equilibrium dynamics.
3.2. The evolution of cooperation when cooperators are endowed with a uniform, non-evolving aspiration Next we assign a uniform aspiration level to cooperators. Defectors have no aspiration as they always defect independent of the group composition. In the evolutionary process, when conditional cooperators imitate defectors, they turn into defectors. When defectors imitate conditional cooperators, they turn into cooperators and their aspiration level is set to be the same as these imitated (i.e., conditional cooperators). Such cooperators take over half of the population at start of the evolution. The rest are defectors. Similarly, each individual takes part in nine public goods games, one centered on themselves and the others on their neighbors. Each public goods game repeats as many rounds as L. Cooperators shall terminate cooperating in future once their aspirations are not met. Figure 2 presents the fraction of cooperators as a function of δ for different aspiration levels. Compared with repeated interactions with no memory, introduction of aspiration favors the evolution of cooperation. The higher the aspiration level, the more remarkable the promoting effect. Intriguingly, though the threshold of δ required for the onset of cooperation 4 markedly declines with increasing aspiration level, the The population is structured by a lattice with periodic boundary conditions. Each node represents an individual and is linked to eight neighboring nodes. As the repeated interactions without memory is considered, feasible strategies are just to defect (defectors) and to cooperate (cooperators). At the start of the evolution, half of the population are randomly assigned to be cooperators and the rest defectors. Each individual takes part in nine public goods games, one centered on themselves and the others each centered on each of the neighbors. Each public goods game repeats as many rounds as L. We average the fraction of cooperators over a window of 5000 generations after the transient generations 2×10 5 as the population's equilibrium cooperation level. Each data point is averaged over 100 independent runs in terms of initial strategy distribution. The population size is 100×100. The selection intensity is β=1.00.
threshold required for the full pervasion of cooperators remains almost identical for M=1, 2, 3, and 4. Only when M arrives at 5 and beyond, the corresponding threshold significantly falls. A natural question arises why the interval of δ responsible for the coexistence state widens for the moderate aspiration levels, say M=4, 5, 6.
The robust coexistence of defectors and cooperators covering so wide an interval of δ can be explained if we distinguish cooperators according to whether they continue to contribute up to L rounds, hence multiplying the public goods produced, or just contribute in the first round and cease contributing from then on, in the public goods game centered on themselves. Depending on this, we designate cooperators correspondingly to be either 'active' or 'inactive'. An active cooperator means that at least M players have cooperated in the group centered on themselves. Such cooperators shall always cooperate in future L − 1 rounds. On contrary, an inactive cooperator means that fewer than M players have cooperated in the group centered on themselves. As such cooperators' aspirations are not satisfied, they shall switch to defecting in future L−1 rounds. For a low level of aspiration, most of cooperators are active and they are continuously exploited. Interspersed cooperators quickly get extinguished, as they are always vulnerable in the sea of defectors. Small cooperator clusters surrounding by defectors cannot withstand the engulfing of defectors. Only by aggregating very closely can cooperators resist the invasion of defectors, or expand their territories (see top row in figure 3).
For moderate levels of aspiration, different scenarios appear. At this time, cooperators differentiate their strategies towards groups of varying composition. They cooperate when a proportion of other players are also willing to cooperate. But when the proportion is too low, they just cooperate in the first round. This diversity in strategic behavior would seem to be able to enhance cooperation effectively, but it is not always so. The δ required for cooperators to have chance to survive really decreases, while the full pervasion of cooperators still requires the same δ compared with low levels of aspiration. This observation can be attributed to different status of cooperators locating on different places in the population. Specifically, there are places where the local densities of cooperators are high and thus meet their aspirations. These cooperators are frequently active. These cooperators can efficiently assimilate defectors. However, there are also some places where cooperators are inactive as their local density is low and fails to reach their aspiration level. These inactive cooperators are often sparsely connected with some active cooperators. Defectors come in between them. Inactive cooperators can exploit active cooperators, yet defectors can prevail over inactive cooperators. The population dynamics exhibit the rock-paper-scissor like cycle (see the second and third row in figure 3), a phenomena also frequently found in other systems [52,53]. This cyclical dominance is so anchored that contingent cooperators are hard to absorb defectors, and thus explaining the widened interval of δ accommodating the coexistence state.
When cooperators have a very high aspiration towards groups, the cyclical dominance is broken. As well established, sparsely located cooperators are almost certainly wiped out by defectors. But due to both initial random strategy distribution and the stochasticity in strategy replacement, occasionally some cooperators come together and form very close clusters. These clusters are destined to reverse the direction of the evolutionary Figure 2. Fraction of cooperators as a function of the normalized enhancement factor δ for different levels of aspiration in repeated interactions. The population structure is the same as adopted in figure 1. The evolutionary competition proceeds between defectors and cooperators endowed with a preassigned aspiration level. At the start of the evolution, half of the population are randomly assigned to be cooperators with the preassigned aspiration level and the rest defectors. Each individual participates in nine public goods games, one centered on themselves and the others each centered on each of their neighbors. Each public goods game repeats as many rounds as L. The total contributions in the first round affects players' future strategies. From round 2 to round L, players' strategies remain unchanged. We average the fraction of cooperators over a window of 5000 generations after the transient generations 2×10 5 as the population's equilibrium cooperation level. Each data point is averaged over 100 independent runs in terms of initial strategy distribution. The population size is 100×100. The selection intensity is β=1.00.
race. The population dynamics mainly unfold in the borders of cooperators and defectors. Defectors can at best free ride on the contributions of the contingent cooperators residing on the borders one time. From the second round on, these cooperators immediately switch to defection and thus inactive. But these cooperators, when grouping with inside cooperators, can accumulate a large amount of payoff. Inside cooperators are more often active and thus garner larger payoffs. Therefore, both inactive cooperators and active cooperators predominate defectors. In this situation, no matter how small these cooperator clusters are in size in the initial stage of their formation, they can expand their territories irreversibly and eventually take over the whole population (see bottom row in figure 3). These are the reasons why even very low values of δ can both ignite the onset and induce the full dominance of cooperation.

The full population dynamics when aspiration coevolves with strategy
Let us now probe the full population dynamics, where individuals synchronously update both their strategies and aspirations according to the probability as given by the aforementioned Fermi function. We would like to emphasize that at start of the simulation, half of the population are assigned as defectors and the rest cooperators. These cooperators are split into nine classes in terms of aspiration level. Each cooperator is randomly endowed with one level but we ensure that each class comprise · cooperators. Conditional cooperators' strategy may vary with round in the repeated public goods game. Once their aspirations are not met, conditional cooperators cease cooperating in the remaining rounds. Results are presented in figure 4(a). For extremely low δ (δ<0.18), the social dilemma is so entrenched that even variation in cooperators' aspiration levels is unable to prevent the demise of cooperators. Cooperators begin to have chance to survive at δ = 0.18, much smaller than the ones needed for the onset of cooperation in repeated interactions without . Strategy distribution in typical evolutionary processes for different values of δ when the evolutionary competition proceeds between defectors and cooperators endowed with a preassigned aspiration level. Each row is related to a typical evolutionary process starting with random initial strategy distribution. Time point at which the snapshot is taken is given in each panel. The aspiration level is preassigned as M=1 (top row), 3 (the second row), 6 (the third row), and 9 (bottom row). Values of δ are 0.80, 0.65, 0.50, and 0.20 from the top to second to third to the bottom row, respectively. Blue are defectors, while gray and black are inactive and active cooperators, respectively. The population size is 100×100. The selection intensity is β=1.00. memory, and still smaller than in preassigned level scenarios as M=1, 2, 3, 4, 5, 6. Starting from δ=0.18 up to 0.28, cooperators and defectors coexist. The larger the value of δ, the higher fraction of cooperators in the coexistence state. Further growth in δ enables cooperators to homogenize the whole population. The dependence of fraction of cooperators on δ shows no difference with single-round spatial public goods game, while cooperators of varying aspiration levels play crucially different roles in promoting cooperation.
We next scrutinize their respective contributions of contingent cooperators C M (Mä{1, 2, 3, K, 9}) to the overall cooperation level. Results are somewhat novel. Depending on δ, shares of cooperators of varying aspiration levels differ. When δ is small, the dilemma in the public goods game is still very harsh. Though some cooperators just deliver their contributions to groups comprising high portions of cooperators, they still are unable to ward off defector's invasion. For cooperators who almost always cooperate, they fare worse and are destined to be wiped out. The population is quickly overwhelmed by defectors.
As δ exalts and locates in the interval (0.18, 0.28), defectors can still quickly invade neighborhoods of cooperators with low aspiration. For neighborhoods of cooperators with high aspiration, there are some exceptions. The relaxation in the social dilemma, together with the repeated interactions, offers such cooperators some chance to resist defector's invasion and even assimilate defectors. Once inside these cooperator neighborhoods reside no defectors, these neighborhoods strengthen themselves against defector's invasion. For δ=0.20, detailed record on evolutionary processes demonstrates that the competition mainly progresses between defectors and cooperators with the highest aspiration level (see figure 4(b)). At this time, such cooperators, albeit still in a disadvantageous position, have a non-zero likelihood to eventually win the evolutionary race (similar to the bottom row in figure 3). As δ rises to 0.25, dilemma is further weakened and cooperators is upheld to a competitive position. Most of the evolutionary processes end up with C 9 being the ultimate winner. Defectors can but very rarely win. Interestingly, a third possibility can also appear that defectors and cooperators with sub-highest aspiration level (=8) coexist, suggesting that the territory of such cooperators expands at a rate comparable to that defectors engulf such cooperators' periphery. It should be noted that cooperators with low and moderate aspiration levels fail to survive the evolutionary race (see figure 4(c)). . Fraction of cooperators as a function of the normalized enhancement factor δ when aspiration coevolves with strategy. Each interaction is likely to repeat as many rounds as L, depending upon the specific composition of each group. The population structure is the same as adopted in figure 1. At the start of evolution, half of the population are randomly assigned to be defectors and the rest cooperators. Of these 100×100/2 cooperators, cooperators with each of these nine levels, {1, 2, 3, 4, 5, 6, 7, 8, 9}, account for a ninth, respectively. They are randomly endowed with these levels. Each individual participates in nine public goods games, one centered on themselves and the others each centered on each of their neighbors. The total contributions in each round affects players' strategies in next round. We average the fraction of cooperators over a window of 5000 generations after the transient generations 2 × 10 5 as the population's equilibrium cooperation level. Each data point is averaged over 100 independent runs in terms of initial strategy distribution. In panel (b) δ=0.20; (c) δ=0.25; (d) δ=0.30; (e) δ=0.50; (f) δ=0.70. The population size is 100×100. The selection intensity is β=1.00.
As δ continues to grow, dilemma is greatly loosened. When it arrives at 0.30, the chance that defectors win over, and even coexist with cooperators are negligible, only 0.7%. One can also find that the fraction of cooperators with the highest aspiration level tilts down, while cooperators with the sub-highest aspiration level accounts for the highest fraction (see figure 4(d)). Meanwhile, cooperators with moderate levels of aspiration such as M=4, 5, 6 begin to have chance to survive, though still holding low shares. This inherent properties of the model become more remarkable with the growth of δ. First of all, there exists an optimal solution R M * in term of aspiration level. This optimal level lies in between one and nine (the highest possible aspiration level) and is subject to change of δ. For this class of cooperators sticking to such optimal aspiration level, it attains the highest fraction among all nine classes of cooperators (see figure 4(e)). Next, unanimous criterion for contributing does not form [54], given that several classes of cooperators coexist in the population. Finally, as the dilemma attenuates, the diversity of the aspiration level of cooperators is enhanced. At this time, cooperators with the optimal level of aspiration are still the most prevalent, but its fraction falls. Fractions of cooperators, whose aspiration levels away from this optimal level, ramp up (figure 4(f)). Figure 5 presents several typical evolutionary processes, revealing the underlying reasons for the emergence of such an optimal level of aspiration and the highest fraction of cooperators associated with this level. For large values of δ, cooperators with moderate aspiration levels account for the highest fraction. This result is in line with the one reported in [4]. In this situation, cooperators with high aspiration levels undertake much part of the work to compete with defectors, depressing defector's advantage over cooperators with not so high aspiration levels. Cooperators with moderate levels can make full use of this opportunity to spread. This has two reasons. On the one hand, they can boost payoffs greatly by grouping with cooperators with high aspiration levels. On the other hand, they themselves also have a certain degree of competence against defectors. This second advantage for cooperators with low levels of aspiration are greatly discounted. As a result, cooperators with moderate levels of aspiration spread most quickly. This determines their eventual highest fraction, as the population dynamics shall drift neutrally once defectors die out.
Our model philosophy is similar to the one refined in [4] but carried out in spatial populations. For large values of δ, our model recreates the main results reported there: the population dynamics stabilize at a high level of diversity of aspiration, there is an optimal aspiration whose corresponding strategists are the most prevalent. There results do not qualitatively change for larger population sizes, more rounds of repeated interactions, stronger selection intensity. This general feature still holds for lower values of δ in their model, but disappears in our model. When the collective dilemma allows for coexistence of defectors and cooperators, it is the strategy C 9 that becomes the ultimatum victor. Other strategies invariably lose out before they can get a foothold. These differences may result from two factors. In [4], mutation is uniform. AD is just a strategy, and there are five types of contingent cooperators. Putting selection force aside, uniform mutation undoubtedly favors evolution towards contingent cooperators. It is because the population flows into contingent cooperators at a rate five times as fast as that the population flows away from them. Seeing contingent cooperators as a whole, the mutation is biased towards cooperation. In present study, strategy mutation is not involved and thus this bias is absent. We have also initialized just half of the population as cooperators at start of the evolution. The impartiality in proportions enables us to fully explore how diversity in aspiration coevolves with strategies.
In fact, our main results still hold for asynchronous updating. In the asynchronous updating, one individual randomly selected imitates the strategy of one of their neighbors each generation. Payoff of each individual comprises two parts, one derived from participating in the repeated public goods game centered on themselves, the other from participating in the repeated public goods games centered on their neighbors. We have to point out that for harsh dilemma, it gets a bit harder for cooperators to survive faced with competition with defectors, as asynchronous updating somewhat decelerates the aggregation of cooperators in the evolutionary process compared with synchronous updating (see figures S1, S2 and S3). As a result, the normalized enhancement factor required for the onset of cooperation in asynchronous update slightly increases (see figures S1 and S2). So does δ required for arriving at the same cooperation level also increases in asynchronous update. When δ increases to and exceeds a threshold where cooperators are able to permeate the whole population, this difference disappears.

Summary
We have constructed a spatial model to study the coevolutionary dynamics of aspiration and strategy. Individuals each participate in repeated public goods games organized by individuals in their neighborhood. Multiple rounds of interactions allows cooperators to adjust their strategies according to the performance of groups they are involved. Cooperators are thus divided into nine classes based on their aspiration levels. Our results show that repeated interactions without memory intensifies spatial reciprocity, benefiting the survival of cooperators, but to a limited degree. Whenever strategy and aspiration coevolve, we have found that cooperators with high levels of aspiration play a decisive role in competition with defectors for harsh dilemma, where cooperators with low and moderate levels of aspiration are doomed. These results differ starkly from the ones reported in [4] (see figure 2(a) there). There are two main explanations for this difference. In the first place, the evolutionary dynamics start from a more adverse environment for cooperators since all classes of cooperators combined account for half of the population. And secondly, harsh dilemma and strong selection leave shorter periods of time for cooperators to cluster. Only cooperators with higher levels of aspiration can quickly form clusters before being wiped out. Cooperators with other levels of aspiration can neither cluster between themselves nor cluster with cooperators of higher aspirations.
Growth of enhancement factor relaxes the dilemma, improving the competitiveness of cooperators. For sufficiently high values of δ, the stationary distribution exhibits general feature that there exists an optimal level of aspiration, such that cooperators adhering to this level are mostly selected and thus account for the highest fraction of all classes of contingent cooperators. Cooperators with moderate levels of aspiration can escape continuous exploitation. They are also willing to contribute in groups which are only partially cooperative. Balance between the exploitation prevention and the generosity puts such cooperators in a competitive position when confronted with defectors. At the same time, when such cooperators meet cooperators with higher levels of aspiration, they can quickly cluster and spread out. These two cooperation-enhancing effects are weakened or even disappear for cooperators with low levels of cooperators. The evolutionary force thus induces a compromise between too low aspiration levels, which get eliminated before getting a foothold, and too high aspiration levels, associated with harsh coordination barriers hard to surmount. Further growth in δ barely affects the population dynamics, while lower values of δ break up the balance, necessitating very high levels of aspiration for cooperators to survive.
What is more, for small δ where each class of contingent cooperators alone have no chance to survive, in the coevolutionary settings they survive and even are most prevalent. In this sense, aspiration and contingent cooperation coevolve, meaning that a high level of diversity of aspiration and contingent cooperators can mutually reinforce each other. A necessary condition for such coevolutionary dynamics is weak dilemma.
Further efforts along this research line worth questioning include the effects of mutation on both strategy and aspiration, longer memory instead of one-step memory, larger strategy space such as the ones considered in [55,56]. Our framework can also be extended to any population structure [57][58][59][60] only if we make a modification in the definition of aspiration level: the level is associated with the fraction of cooperators in each group rather than the number of cooperators in the group.