Stochastic gain in finite populations

Flexible learning rates can lead to increased payoffs under the influence of noise. In a previous paper [Traulsen et al., Phys. Rev. Lett. 93, 028701 (2004)], we have demonstrated this effect based on a replicator dynamics model which is subject to external noise. Here, we utilize recent advances on finite population dynamics and their connection to the replicator equation to extend our findings and demonstrate the stochastic gain effect in finite population systems. Finite population dynamics is inherently stochastic, depending on the population size and the intensity of selection, which measures the balance between the deterministic and the stochastic parts of the dynamics. This internal noise can be exploited by a population using an appropriate microscopic update process, even if learning rates are constant.

Figure 1

(Color online) Average payoff difference of a population with an adaptive learning rate ${\eta }_x=1∕2-(1∕2)\tanh \left({\alpha }_x\Delta \pi \right)$ compared to a population with a constant learning rate of ${\eta }_y=0.5$ in the matching pennies game using the local update rule (see text). (a) Payoff differences for a constant population size of $N=100$ and for different $\alpha$. The average payoff difference is small if the intensity of selection $w$ is small otherwise the population with an adaptive learning rate has a higher average payoff. The stochastic gain effect becomes more pronounced with increasing $\alpha$ (averages over $2\times {10}^4$ random initial conditions in the interior until the maximum time $T={10}^5$ or until the absorbing boundaries of the system are reached). (b) Payoff differences for three different population sizes for fixed $\alpha =1.0$. The total payoff decreases with higher population size $N$ because the noise intensity decreases. In the limit of $N\to \infty$ we obtain the equation for the stochastic replicator dynamic without external noise. Thus the payoff difference converges to zero. Averages over 7000 random initial conditions until the absorbing boundaries are reached or until the maximum time $T=N\times {10}^3$.

Figure 2

(Color online) Average payoff difference of a population with an adaptive learning rate ${\eta }_x=1∕2-(1∕2)\tanh \left({\alpha }_x\Delta \pi \right)$ against a population with a constant learning rate of $\eta =0.5$ in the matching pennies game using the frequency-dependent Moran process in both populations. The parameter values in (a) and (b) are identical to those of Fig. 1, but in the Moran process shown here the maximum intensity of selection is given by $w=0.5$. Qualitatively, the stochastic gain effect does not depend on the details of the update mechanism in finite populations: With $\alpha$ increasing from $\alpha =0.0$, the payoff advantage of the adaptive population increases. However, there is an optimal $\alpha$ for which the stochastic gain effect is most pronounced; see text. With increasing $N$, the system approaches a deterministic replicator system and the intrinsic noise vanishes. Thus, increasing $N$ leads to smaller payoff differences. Moreover, the finite-size effect of a negative payoff difference for low intensity of selection vanishes.

Figure 3

(Color online) Payoff distribution and stationary distribution for the Moran process in the strategy space spanned by the state space for a constant population size of $N=100$, encoded by a color scale where bright colors indicate high values. (a) Average payoffs $⟨{\pi }^X⟩=[{\pi }_0^{X}i+{\pi }_1^X(N-i)]∕N$ of the adaptive populations $\left(X\right)$ are shown. In the bottom left and top right areas (green, +) the payoffs of the adaptive population $\left(X\right)$ are positive, whereas in the bottom right and top left areas (red, −) the payoffs are negative. The adaptive population can obtain a higher stationary probability density in the bottom left and top right areas, leading to the stochastic gain effect. (b) Stationary distribution for $\alpha =0.0$, invariant under rotation by $\pi ∕2$ and approximately rotationally invariant close to the fixed point of the replicator dynamics at $(x,y)=(0.5,0.5)$. The population dynamics drives the system around this point. (c) With increasing $\alpha =1.0$ the system is driven to the interior. Now the areas where the average payoff for the adaptive population is higher shows a larger stationary probability density. (d) Same as in (c) but the stationary distribution for $\alpha =10.0$ is shown. Increasing $\alpha$ further leads to a smaller probability density in the areas where the payoff difference is high and thus, the payoff difference decreases again for large $\alpha$ (for all panels, parameter values are $N=100$, $R=5\times {10}^5$ independent realizations, $w=0.35$, maximum number of time steps $T=N\times {10}^3$).

Figure 4

(Color online) Average payoff difference of a population using the Moran process against a population using the local update rule for two different sizes of population $N$. Both populations do not change their learning rate $({\eta }_0=0.5)$. Independent of the intensity of selection $w$ the population using the Moran process obtains a higher payoff. Thus, the stochastic gain effect can also be observed in the absence of variable learning rates. The total payoff decreases with higher population size $N$. In the limit of $N\to \infty$ both dynamics result in the equation for the deterministic replicator dynamic, without external noise thus the payoff difference tends to zero. Parameter values are $N=500,1000$, $T=N\times {10}^3$ maximum number of time steps, and $R=15000$ realizations.