Maximizing cooperation in the prisoner's dilemma evolutionary game via optimal control

The prisoner's dilemma (PD) game offers a simple paradigm of competition between two players who can either cooperate or defect. Since defection is a strict Nash equilibrium, it is an asymptotically stable state of the replicator dynamical system that uses the PD payoff matrix to define the fitness landscape of two interacting evolving populations. The dilemma arises from the fact that the average payoff of this asymptotically stable state is suboptimal. Coaxing the players to cooperate would result in a higher payoff for both. Here we develop an optimal control theory for the prisoner's dilemma evolutionary game in order to maximize cooperation (minimize the defector population) over a given cycle time $T$, subject to constraints. Our two time-dependent controllers are applied to the off-diagonal elements of the payoff matrix in a bang-bang sequence that dynamically changes the game being played by dynamically adjusting the payoffs, with optimal timing that depends on the initial population distributions. Over multiple cycles $nT$ $(n>1)$, the method is adaptive as it uses the defector population at the end of the $n\mathrm{th}$ cycle to calculate the optimal schedule over the $n+1\mathrm{st}$ cycle. The control method, based on Pontryagin's maximum principle, can be viewed as determining the optimal way to dynamically alter incentives and penalties in order to maximize the probability of cooperation in settings that track dynamic changes in the frequency of strategists, with potential applications in evolutionary biology, economics, theoretical ecology, social sciences, reinforcement learning, and other fields where the replicator system is used.

Figure 1

Twelve regions in the $(a_{12},a_{21})$ plane define which of the games is being played. There is no loss of generality in choosing $a_{22}$ at the origin. Our control problem starts ($t=0$) in the Prisoner's dilemma corner and asks what is the best path to travel, subject to a constraint, in order to minimize the defector population at the end of time $t=T$.

Figure 2

Optimal trajectories and optimal control sequences for small, medium, and large initial defector populations, with ${\vec{D}}_T=(0.5,0.5)$. (a) Defector frequency for initial state $x_2\left(0\right)=0.1$; (b) optimal control sequence for initial state $x_2\left(0\right)=0.1$; (c) defector frequency for initial state $x_2\left(0\right)=0.5$; (d) optimal control sequence for initial state $x_2\left(0\right)=0.5$; (e) defector frequency for initial state $x_2\left(0\right)=0.9$; (f) optimal control sequence for initial state $x_2\left(0\right)=0.9$; (g) defector frequency for initial state $x_2\left(0\right)=0.95$; and (h) optimal control sequence for initial state $x_2\left(0\right)=0.95$.

Figure 3

(a) Relative reduction as a function of initial defector proportion. Maximum reduction (shown as dashed vertical line) occurs for initial defector proportion of around $22\%$. In the limits $x_2\left(0\right)\to 0,1$, the exact schedule $\vec{u}\left(t\right)$ does not matter, only the total dose $\vec{D}\left(T\right)$. The controls schedule is much more efficient at reducing the proportion of defectors for small populations than for large; (b) turn-on time for $u_1\left(t\right)$ and $u_2\left(t\right)$ as a function of initial defector population $x_2\left(0\right)$. In the narrow (approximate) range $0.89\le x_2\left(0\right)\le 0.91$, the times change abruptly and the control sequence has five segments as shown in Figs. 2 and 2.

Figure 4

The sequence of four games that produce the optimal trajectory if the switching times are chosen as shown in Fig. 2. (a) The first game is a prisoner's dilemma game, where $x_2=1$ is an asymptotically stable fixed point; (b) the second game is a Leader game with interior fixed point $x_2=2/5$ being an asymptotically stable fixed point; (c) the third game is a Deadlock game with $x_2=0$ being the asymptotically stable fixed point; (d) the fourth game is game No. 10 in Fig. 1 with $x_2=0,1$ being asymptotically stable fixed points.

Figure 5

Defector frequency reduction using optimal control over five consecutive cycles. (a) Defector optimal frequency with $x_2\left(0\right)=0.9$ over five cycles as compared with frequency associated with sequential control and constant control; (b) same as (a) plotted on log-linear scale; (c) defector optimal frequency with $x_2\left(0\right)=0.5$ over five cycles as compared with frequency associated with sequential control and constant control; (d) same as (c) plotted on log-linear scale; (e) defector optimal frequency with $x_2\left(0\right)=0.1$ over five cycles as compared with frequency associated with sequential control and constant control; and (f) same as (e) plotted on log-linear scale.