Introduction: bringing together Darwinian evolution and games

Compared to either of its parent fields, mathematical population genetics and rational-choice game theory, evolutionary game theory is still a relatively young field. At present, a growing list of example models has been worked out (reviewed in [4, 7, 8]); some classification has been carried out based on symmetry, particularly from a dynamical-systems perspective; and suites of analytic methods now exist, drawing from the established fixed-point analysis of strategic games [25], from non-linear dynamics [4, 7, 12], and to a limited extent from stochastic process theory [3, 6].

A large amount of analysis has been carried out for games in the strategic form ² (also called 'normal' form) and some work has been done using the extensive form [26]. Since the extensive form is a refinement of the strategic form in which the structure of play is made explicit [13], the relatively limited treatment it has received represents only a very early stage of exploration of a potentially rich and important topic.

The majority of the existing literature on evolutionary games grows out of the study of either fixed points or dynamical systems and has been based on the classical replicator equation [4], which is a mean-field equation ³ . The concept of evolutionary stability of equilibria, introduced by Maynard Smith and Price [27], is defined in terms of infinitesimal perturbations about the solution given by the replicator equation. Some research has been carried out on non-infinitesimal population fluctuations (inevitable in finite-sized populations and, as we will show, sometimes important even in infinite-population limits). Work on fluctuations divides into studies concerned with refining equilibrium selection to exclude ambiguity and studies that treat ongoing dynamics as an empirical consequence of ambiguity that models should not seek to exclude.

In general, games will admit multiple Nash equilibria or evolutionary stable states [28–31]. A tradition in economics has been to seek equilibrium refinements [14], which reduce this multiplicity by placing further restrictions to rule out sub-sets of equilibria in different contexts. In the presence of finite fluctuations, the long-run probabilities for a population to be found within basins of attraction of different equilibria will generally differ and the ratios of these probabilities can generally be made to diverge with large population size or small fluctuation strengths ⁴ . Therefore, by a process analogous to annealing, stochasticity may be used to reduce the number of equilibria that are populated with non-zero measure in the long run [32], providing an evolutionary argument for refinement. Adopting an alternative emphasis, a few studies have been performed in which stochastic dynamics in the presence of multiple equilibria was the primary focus [33–37]. As in the study of the extensive form in evolutionary dynamics, the topic of stochastic aggregate dynamics is an exceedingly broad and important area into which only introductory forays have been made. It will be our main area of emphasis.

We will study evolutionary game theory in its stochastic form. Stochasticity arises from the population-level events which are already recognized in population genetics: sampling of individuals who will interact (here, by playing a game together), culling (death) and replacement (replication). In some of the examples we will also consider stochasticity within the course of play of a single game. In games for which strategies consist of moves that could be shuffled as part of the reproductive process, the place at which crossover occurs is an additional source of randomness. Beyond the mean-field analysis of the replicator equation, new phenomena arise that are not possible in deterministic systems. We are also led to ask which outcomes predicted by the replicator dynamic are robust in the presence of fluctuations, a seemingly innocent question that in statistical mechanics and field theory has led to a radical reconceptualization of the nature of objects and interactions [38–40].

In suitable weak-fluctuation limits, we will recover standard results from the replicator analysis concerning bifurcations to multiple equilibria or limit cycles (other attractors could be included but are not pursued here). These phenomena are an important source of multilevel dynamics in ontogeny ⁵ and evolution. They cause individual dynamics, through mutually reinforcing cooperative effects, to become entrained by population states which then take on dynamics of their own. We classify bifurcations according to symmetry, as is done in the dynamical-systems approach [4]. However, making a slightly different emphasis than the typical one from dynamical systems, we view bifurcations not as fundamental changes of symmetry groups, but as changes in the representation of symmetries by dynamical states. The presence of underlying symmetries that are merely hidden, in the stochastic domain, is the basis for proofs that multilevel dynamics is a robust property against all orders of fluctuation corrections, even if we cannot compute or efficiently simulate them. The existence of hidden symmetries causes multiple equilibria to escape the filters of equilibrium refinement inherently, allowing us to use stochastic approaches such as annealing [32], not with the goal of singling out a unique static equilibrium, but to identify sources of long-run dynamics that are not sensitive to fine modeling assumptions. We will recover important symmetry-derived theorems of condensed matter physics and field theory, such as Goldstone's theorem, and show the forms that they take in evolutionary dynamics, particularly as these result from new roles of time in irreversible stochastic processes.

We then consider effects that cannot be produced at all in the deterministic approximation, including fluctuation-controlled dynamical regimes that persist in infinite-population limits, creep and forms of symmetry breaking that resemble glass phases, where the number of ordered macrostates and the complexity can be 'open-ended'. The potential for open-ended complexity is an often cited property of evolving systems [41] that simple bifurcations do not possess, so it is important to have examples in evolutionary game theory where at least the rudiments of an unlimited diversity of macrostates can be exhibited.

We will be interested in common mathematical elements of evolutionary game theory as a system for studying evolutionary dynamics, with an eye toward empirical applications. Therefore, we will bypass the appeals to scientific narratives that often play a large role in the motivation of game models for particular cases. We focus instead on the consequences of stochasticity that must affect the choice and interpretation of all game models from empirical observations. Any attempt to use evolutionary games to understand natural phenomena will inevitably include sampling fluctuations from observations and hence uncertainty in model identification ⁷ , as well as stochasticity in model dynamics that will affect analysis and prediction.

De-emphasizing narrative forces us to ask what can be justified statistically in the choice and interpretation of a game model, in the presence of uncertainty, noise and error. By defining games statistically, we obtain a clearer abstraction of the role of games as a general framework in evolutionary dynamics: we will argue that the proper abstraction for the role of games is as models of development, which complement the models of information dynamics formalized in population genetics. We will introduce the problem of identifying a game model as a problem of non-linear regression following the empirically motivated methods of Fisher's theorem [44] and the more general Price equation [45–47]. Regression estimates begin with the lowest-order (generally linear) models of fitness and recursively construct the dynamics of the game through the addition of higher-order interaction terms as required. We will thus embed evolutionary game theory within the larger suite of formal evolutionary methods, so that approaches such as quantitative genetics coincide (tautologically) with the lowest-order estimators for games. Since any finite sample supports the inference of only a limited number and precision of model coefficients, we will be forced to address the problem of justifying models that formally involve infinite hierarchies of coefficients (even when these are all set to zero, they nonetheless exist as modeling choices ⁸ ), and the related problem of determining which predictions from a statistically estimated model are robust.

An immediate consequence of incorporating stochasticity in all elements of model selection, analysis and interpretation is that regression coefficients generically come to depend on scale. Relevant scales may be the population size, or the time interval or number of interaction events over which samples are drawn. The scale dependence of regression coefficients that is readily demonstrated in models reminds us that scale is also an inherent property of empirical observations. Therefore the concept of calibrating the coefficients in a game model to describe a natural system is one that inherently invokes the scale at which the model is to be estimated and analyzed. We will go further to propose that the meaning of the fundamental abstractions of evolutionary game theory, such as individuals, genes, or strategies, should come to be understood as scale-relative concepts, for the same reason that elementary particles and forces are now understood to be scale-relative concepts in physics.

In a modeling framework where the detailed model description becomes scale-dependent, symmetries take on elevated importance because they are the invariant properties that identify systems across all scales. The changes of symmetry representation by population states, as population size or interaction strength are changed, then define the robust dynamical regimes or phases in which variation, interaction, replication and selection act. This view is very compatible with the modern understanding of multilevel selection [47, 53, 54] and the hierarchical role played by evolutionary dynamics in the evolution of developmental complexity [22, 23]. We believe it leads to a conceptualization of the fundamental abstractions of evolutionary game theory in keeping with the best understanding in modern evolutionary theory. In addition to demonstrating some qualitative categories of scale-dependent description, and showing why they matter, we derive methods to quantitatively compare model descriptions that differ at multiple scales because they incorporate different degrees of correlation and we show that these methods match well against simulation results, even in some cases of large fluctuations.

We construct evolutionary game theory as a general framework to classify and interpret fitness models—their quantitative dependence on population state and, if desired, explicit representations of the interaction sequences that determine fitness—within the axiomatic structure of population genetics. We begin in chapter 2 with the Price equation, an accounting identity for any process satisfying the assumptions of population genetics, in which fitness universally appears as a summary statistic [47]. ⁹

Fitness is defined in terms of the number of offspring relative to the number of their parents, grouped by the parents' type. It is a descriptive statistic, which can be computed for any given realization of an evolutionary process. If the purpose of an evolutionary account is not only description or historical reconstruction of a particular instance, but also estimation of a process model for change, then fitness (as well as other parameters) must be given a model in terms of properties of individuals and populations. The model estimation problem is to determine what structure and what coefficients can be justified from empirical observation. Once a population process model is chosen, one must often also define closures for it ¹⁰ , which are approximations that permit calculations from finite orders of terms or parameters. The empirical calibration of fitness models leads to the approach of Fisher [44] and Price [45], who replace fitness (the summary statistic) with models that are meant to match regression coefficients of fitness on individual and population states, obtained from statistical samples. Since regressions can be performed on interaction terms of arbitrarily high order, in principle this approach provides a full basis for the identification of those aspects of a process that affect fitness; one can then ask in a principled way how much detail is supported by empirical evidence and attempt to systematically construct least-committal models [55, 56] for undetermined parameters.

The mechanism of heredity in any population process determines which consequences of events affecting parents persist as features of the population state of the offspring. Since the filter of natural selection—the part of the population process represented explicitly in population-genetic models—acts to narrow the distribution of properties of offspring, population genetics comprises the information incorporating aspects of evolutionary dynamics ¹¹ . The formal equivalence between the replicator equation and Bayes's theorem for updating probability distributions [58] provides a way to quantify this concept of information and also to show that selection is a statistically optimal method for incorporating information within a population about its environment.

Not all properties of organisms are directly preserved by mechanisms of heredity and the difference between what is preserved and what is generated and acted on by selection is the difference between genotype and phenotype. The complement to the information transmitted via a distribution of genotypes is the collection of all other aspects of phenotype, which are constructed through non-heritable interactions within generations. We will refer to these as development, broadly construed. From the perspective of classical population genetics, development consists simply of a genotype/environment → phenotype/fitness map, but we are concerned with the actual generating processes responsible for that map.

The complementarity between the information incorporating function of selection and heredity, and the constructive role of development, for us defines the respective roles of population genetics and games within evolutionary game theory.

Starting from the Price equation and a need for closures, a general polynomial expansion of frequency-dependent regression models for fitness is equivalent, at order k, to treating development as a k-player normal-form game with uniform matching of individuals. This equivalence is a tautology, meaning both that the normal-form game interpretation is always available and at the same time that it is highly ambiguous about the mechanism that constitutes the 'game' and about its interpretation. If, beyond the aggregate statistics of fitness, we are given more information about the frequencies with which individuals are sampled to interact in the population, we may resolve the normal form into contributions from assortative matching and a set of payoffs which differ from the mere coefficients in the fitness function. If we know more about the internal structure of interactions—which may be temporal sequence, signaling or imitation, or even just linkage—then we may refine the normal form to a particular extensive-form game [26]. Further elaborations, to include constraints on joint actions by multiple individuals, could be developed to make contact with the coalitional-form representation from cooperative game theory, but we do not pursue those systematically in this monograph. The sequence of one-to-many mappings, from the normal-form to the extensive-form and the coalitional-form solution concepts, constitutes a well-understood approach to refining the definition and interpretation of games in classical game theory [13] and we think it provides a useful level of discipline also for the interpretation of evolutionary games.

In this way games emerge as a highly general, if not all encompassing, framework to model development.

A central theme in our approach to the topic of evolutionary games is that interactions among individuals in single events may produce population behaviors that, in aggregate, are describable with games of a similar form but with coefficients that may differ from those that the individuals directly experience. Most obviously, individual traits may polarize population states or lead to distributions of fluctuations that feed back so that higher-order correlations become part of the best average estimates for individual fitness.

In the domain of toy models, we often have the option to regard such differences as artifacts of the adoption of coarse-grained descriptions, but we think that if games are to become a serious tool for the analysis and interpretation of empirical phenomena, it is better to start to think of such scale dependence of parameters as an essential feature of the definition of such concepts as individuality, agency and interaction. Such a revision in the notion of what constitutes an elementary particle have been fundamental to a radical reformulation of the conceptualization of objects and interactions in physics [38–40]. Modern writing on evolution makes a serious effort to understand the way structured interactions, replication and selection interact at many scales, to produce multiple novel levels of individuality both in development [22] and as levels of selection [47, 53, 54]. We believe that a statistical notion of individuality and interaction is already inherent in modern biology and we make that integral to the way we present game models.

When the use of evolutionary games is altered from toy modeling of a hypothesized 'fundamental' interaction and its scale-dependent approximations, to an attempt to represent data in which all estimation and prediction problems involve uncertainties that depend on scale, a different approach is required to specifying what constitutes a model of a particular actual phenomenon, at all scales of interest. We therefore introduce fitness models and their classification in terms of symmetry groups and the representation of symmetries by population states. Symmetries are scale-invariant properties of systems and changes in their representations (known as symmetry breaking [59–61]) imply robust predictions for multiscale dynamics.

Stochasticity and correlation are the causes of parameter change in models of the same system at different scales and the stochastic effects that are robust within symmetry classes (and that lead to symmetry breaking) are collective fluctuations [62]. We introduce in chapter 2, and develop in detail in chapters 5–8, a set of examples of major classes of symmetry groups and categories of symmetry breaking and show how each implies a distinctive form of scale dependence in fitness or dynamics. These include the emergence of new units of selection or of coalitional behavior from interactions that are non-cooperative at the scale of individual interactions.

The preceding four points provide a set of abstractions of evolutionary games that frees the fundamentals of the theory from arbitrary narratives invoked to justify particular cases and also acts as a guard against over-interpretation. However, these points are not useful in practice without ways to identify the relevant classes of collective fluctuations or their consequences for the expression of symmetry and for parameter changes across scales.

Evolutionary population processes are extended-time, irreversible Markov processes. In general, their distributions over collections of events could be too complicated to permit any robust characterizations. However, a feature of even moderately large populations or times that can make such processes tractable and can make games a stable and useful class of models is the tendency for probability distributions to converge toward a small number of exponential families. Within these families, the combinatorics of large numbers of agents or events may produce leading-log probabilities of fluctuations with a scaling relation known as the large-deviations property, in which the dependence on system scale separates from the dependence on the structure of the fluctuation [63].

Chapter 4 is devoted to a derivation of the large-deviations theory of discrete population processes tailored to the structure of evolutionary game models introduced in chapter 2. We show how the large-deviations limit singles out classes of collective fluctuations and how their properties can be computed to connect game descriptions across scales.

Good treatments of the consequences of large numbers and aggregation exist for evolutionary dynamics [3, 6] and for population processes more generally [64, 65]. Many of these draw from the probability literature and are concerned with convergence and laws of large numbers. Our approach will be one more familiar to physicists and will emphasize the extraction of terms most directly responsible for multiscale dynamics and multilevel selection.

These first five points have addressed general conceptual foundations of evolutionary game theory. The next four points concern particular applications, which are nonetheless of wide interest within either population biology or game theory.

For many applications of evolutionary games it is not necessary (or not empirically warranted) to go beyond the normal form and the assumption of random matching. For others, though, the sub-structure of play when a collection of agents is brought together in a single interaction is central to the question being asked. The standard way to represent event sequences and information conditions in game theory [13, 25] is to refine the normal form to the extensive-form representation, most familiar as a 'game tree'.

Therefore, after introducing the general role of games in population processes in chapter 2, we provide a non-exhaustive but systematic introduction to extensive forms in chapter 3. A dedicated treatment of the extensive form is given in [26] and many more applications still could be developed. Here we will mostly be concerned with the two concepts of neutrality and repetition.

Neutrality [66 – 71] arises in evolutionary dynamics when distinct genotypes have the same fitness in populations that they themselves produce. It is an important form of symmetry under interchange of agent types, and a property that complex developmental programs can be expected to produce very frequently. A summary of some combinatorial counts of game trees enables us to provide explicit examples.

Repetition is a property of game interactions studied extensively in the economics literature [72–75] to model the relation between long-term and short-term incentives. We consider it because it is a source of some very well-known models of neutral evolution, for which we can demonstrate new consequences of collective fluctuations. A widely used framework to study repetition in rational-choice game theory is the repeated game, a particular kind of extensive form built up by recursive attachment of a normal-form stage game to build up the game tree. Repeating the play of a stage game is the simplest and most generic way to produce complexity and with it the problems of understanding error and limitations on strategic capacity.

The next three topics consider other, more specific, uses of the extensive form and of repetition.

Mendelian heredity—the property that the sources of traits are replicated as discrete units which are shuffled like cards rather than being mixed like paint—was one pillar of the modern evolutionary synthesis [54, 76] and is incorporated as an essential assumption within standard population genetics. The convention of formalizing evolution in terms of invariant hereditary 'particles' has presented one of two important challenges to the use of evolution to describe social, behavioral, or institutional dynamics [36, 77]. ¹²

Given the importance of particulate heredity to current evolutionary thinking, we briefly consider its origin in biological systems. We argue that the standard invariant hereditary unit in biology—the gene understood as a non-recombining region of DNA—results not from the properties of DNA as a system for storing information, but from modularity in development which is either reinforced by selection or recapitulated in transmission mechanisms. The extensive form of a game inherently produces a kind of modularity in the sequence of play and offers a natural set of elementary units (the moves) to vary through mutation or to shuffle via crossover. We show, first in chapter 7 and in a different way in chapter 8, how the 'gene' description that arises out of the modularity of game trees leads in the presence of crossover to a multilevel evolutionary dynamic that must select both genes and their covariance within genotypes. In this way we connect the use of the extensive-form game as a general framework to the statistical and developmental origins of modularity that justify the gene concept in biology.

The repeated games, with payoffs accumulated from the moves in each stage of play, have become a standard framework, in both evolutionary [8, 11, 79] and rational-choice game theory [25, 72, 74], to study the relations between short-term and long-term rewards for nominally equivalent move profiles. Repeated games are most often invoked as a framework to study the 'paradox of cooperation'.

In the play of a repeated game, correlations between moves in different stages can re-direct the force of selection (in the evolutionary context), or the non-cooperative equilibrium condition (in the rational-choice framing) to favor outcomes very different from those that would be favored by the stage game in isolation. The approaches to repeated games in evolutionary and rational-choice game theory have diverged sharply in the study of how repetition produces these differences. The divergence of approaches is at first surprising because for finite normal-form games, a result known as the fundamental theorem of evolutionary game theory [7] ¹³ asserts that the evolutionary stable strategies are the same fixed points as Nash equilibria. The forward looking self-consistency of rational choice and the recursive filtering of natural selection might thus be expected to yield the same solutions.

Rational-choice repeated game theory, however, concerns cases with a very large or even infinite degeneracy of Nash equilibria, or with ambiguities in the way equilibrium should be formalized, typically arising from an infinite or indefinite number of stage repetitions and associated normal forms that are not finite. Ambiguities in equilibrium selection are treated with a poorly formalized notion of 'prior agreement', which requires only conditions of feasibility and individual rationality of outcomes ¹⁴ . A set of results known as (the) folk theorems [75] of repeated game theory show that any outcome in the large set of feasible and individually rational move profiles can be supported as an equilibrium.

In evolutionary models with repeated games, the notion of prior agreement is replaced by direct specification of the set of strategies that may exist within evolving populations and the ways they are generated. Depending on how strategies are restricted, or on the dynamics of type changes, limited sets of marginally stable or even unique stable equilibria may be selected.

In chapter 9 we construct the mapping between the evolutionary and rational-choice approaches to strategy selection in repeated games. We show how different forms of coordination in the repeated game, which appear as features of development in the evolutionary setting, may also be cast as signaling about strategy types, or even as kinds of public information that have been introduced in rational-choice game theory to expand the notion of Nash (aka non-cooperative) equilibrium to include correlated equilibria. Our interest is to use the explicitly dynamical and constructive approach to strategy generation in the evolutionary approach to define systematic approximations to the concepts of indefinite repetition and prior agreement in the rational-choice setting and to understand how these approximations imply bounds on strategic complexity.

The classical replicator equation [4] often used to study evolutionary games is very restrictive: by omitting explicit fluctuations, it is effectively a single-scale description. It also uses a particular set of closure assumptions, known as mean-field assumptions, which can sometimes be invalid even in infinite-population limits. For us, the stochastic population process is fundamental at all levels because stochasticity is the basis for criteria of robustness.

It is now well understood that the large-deviations scaling limit is the essential property behind the concept of entropy and thermodynamic limits [62, 63, 80]. A game model at the scale of individual interactions is the mechanical description, from which a (possibly different) game model of the aggregate population behavior is derived as the corresponding thermodynamic limit. The difference between individual and aggregate models results from entropy corrections, equivalent to those in any thermodynamic theory, but derived for extended-time irreversible Markov processes. A low-dimensional example in which the control over dynamics is shifted from deterministic parameters (mean fitness and mutation) to fluctuation entropies is solved in detail in chapter 7. We expect that the most important and common use of evolutionary entropy will come not from low-dimensional systems with imposed symmetries, but from the high-dimensional neutrality produced by complex developmental trajectories, sketched in chapter 3.