Emergence of exploitation as symmetry breaking in iterated prisoner's dilemma

In society, mutual cooperation, defection, and exploitative relationships are common. Whereas cooperation and defection are studied extensively in the literature on game theory, exploitative relationships between players, in which one receives a larger benefit than the other while the game itself is symmetric, are little explored. In a recent seminal study, Press and Dyson demonstrated that if only one player can learn about the other, asymmetric exploitation is achieved in the prisoner's dilemma game. In their study, however, asymmetry is assumed in decision making between persons; the exploiting player one-sidedly determines and fixes the strategy and the exploited player follows it. It is unknown whether such exploitation emerges and is stably established even when both players learn about each other symmetrically and try to optimize their payoffs. Here, we first formulate a dynamical system that describes the change in a player's probabilistic strategy with reinforcement learning to obtain greater payoffs, based on the recognition of the other player. By applying this formulation to the standard prisoner's dilemma game, we numerically and analytically demonstrate that an exploitative relationship can be achieved despite symmetric strategy dynamics and symmetric rule of games. This exploitative relationship is stabilized by both the players: The exploiting player demands the other's unfair cooperation. Even though the exploited player, who receives a lower payoff than the exploiting player, has optimized the own strategy, the player accepts the other's defection to some degree. Whether the final equilibrium state is mutual cooperation, defection, or exploitation crucially depends on the initial conditions. Response to decrease the cooperation probability against a defector leads to oscillations in the probabilities of cooperation between the players and thus a complicated basin structure to the final equilibrium. In particular, any slight difference between both players' initial strategies can be amplified and fixed as a large difference in the probabilities of cooperation, leading to fixation of exploitation. In other words, symmetry breaking between the exploiting and exploited players results. Considering the generality of the result, this study provides another perspective on the origin of exploitation in society.

Figure 1

Schematic diagram for the prisoner's dilemma game and the strategy. Player 1 (horizontal) and 2 (vertical) independently choose their own actions from C and D. The resultant payoffs $T, R, P$, and $S$ are displayed. The black arrows indicate the stochastic transition from the previous action to the next one, by using the probabilistic strategy in the text.

Figure 2

Interpretation of the equilibrium state for repeated games. The horizontal (vertical) axis indicates the unconditional probability of player 1 (2) to cooperate. Blue, green, red, and magenta dots indicate the strategies $x_C, x_D, y_C$, and $y_D$, respectively. Accordingly, response function $f_x\left(y\right) \left[f_y\left(x\right)\right]$ is given by connecting $x_C$ ($y_C$) with $x_D$ ($y_D$). The crossing point of response functions (black dot) agrees with $(x_{\mathrm{e}},y_{\mathrm{e}})$, which is each player's probability to cooperate in the equilibrium of repeated game.

Figure 3

(a) Final state of learning dynamics in the case of $(T,R,P,S)=(5,3,1,0)$. Many sets of fixed $(x_{\mathrm{e}}^{*},y_{\mathrm{e}}^{*})$ satisfy $0\le x_{\mathrm{e}}^{*},y_{\mathrm{e}}^{*}\le 1$ with asymmetry between them. (b) Final state of learning dynamics in the case of $(T,R,P,S)=(5,4.5,1,0)$. Only two sets, $(x_{\mathrm{e}}^{*},y_{\mathrm{e}}^{*})=(0,0)$ and (1,1), are reachable. For both the cases, initial states are uniformly chosen as $(x_C^o,x_D^o,y_C^o,y_D^o)=((2i-1)/2N,(2j-1)/2N,(2k-1)/2N,(2l-1)/2N)$ with $(i,j,k,l)=1,\cdots ,N$ and $N=10$. Accordingly, $x_{\mathrm{e}}^o$ and $y_{\mathrm{e}}^o$ can take values in [0,1]. See the Supplemental Material [22] for the animation of each dynamics.

Figure 4

(a) The region of stable fixed points in which both eigenvalues are negative. (b) Both players payoffs mapped from fixed points. $(T,P,S)=(5,1,0)$ are fixed in all figures, and $R$ is $2.5,3.0,3.5,3.999,4.001$ and 5.0 from left to right. In panel (a), the horizontal (vertical) axis indicates $x_{\mathrm{e}}^{*}$ ($y_{\mathrm{e}}^{*}$). In the case of $(T+S)/2\le R<T-P+S$, in other words, $2.5\le R<4.0$, there are two-dimensional fixed points on $(x_{\mathrm{e}}^{*},y_{\mathrm{e}}^{*})$ with player asymmetry. However, all of the fixed points become unstable in the case of $T-P+S<R\le T$, in other words, $4.0<R\le 5.0$. In panel (b), the horizontal (vertical) axis indicates $u_{\mathrm{e}}^{*}$ ($v_{\mathrm{e}}^{*}$). Red broken line indicates the possible set of both the payoffs. Orange (green) dot indicates the state of pure DD (CC) in all figures. Yellow (red) dot indicates the most exploitative state from 1 to 2 (from 2 to 1).

Figure 5

Trajectories of strategies during the learning dynamics with a payoff matrix of $(T,R,P,S)=(5,3.25,1,0)$. In all figures, $(x_C^o,x_D^o,y_C^o)=(0.9,0.1,0.9)$ are fixed, with (a) $y_D^o=0.1$, (b) $y_D^o=0.25$, (c) $y_D^o=0.65$, and (d) $y_D^o=0.7$. Thus, player 1's strategy is fixed close to TFT, but player 2's strategy departs from TFT, ranging from (a) (closest) to (d) (farthest). Blue, green, red, and magenta solid lines indicate $x_C, x_D, y_C$, and $y_D$, respectively. Yellow and cyan broken lines indicate $x_{\mathrm{e}}$ and $y_{\mathrm{e}}$, respectively. Note that only player 1's trajectory is plotted in panel (a) because each of $y_C,y_D,y_{\mathrm{e}}$ is equal to $x_C,x_D,x_{\mathrm{e}}$. (a) Trajectories of case (1). The player's probabilities of cooperation increase throughout the dynamics and converge to the pure CC. (b) Trajectories of case (2). Both players first move toward the pure CC. At time 10, however, player 1 takes advantage of player 2's generous strategy (i.e., too much unconditional cooperation) and increases his or her probability of defection. Against player 1's behavior, player 2 does not increase punishment to maintain the previous high probability of cooperation, which further increases player 1's defection probability. The finite degree of exploitation from player 1 to player 2 is thus fixed. (c) Trajectories of case (3). The initial asymmetry is larger than that in panel (b). Around time 5, the same exploitation as in panel (b) emerges. From time 5 to time 10, however, player 2 increases his or her punishment of player 1 decreasing $y_D$. From time 10, both players punish each other and finally reach the pure CC. (d) Trajectories of case (4). Until time 5, the exploitation of player 2 by player 1 emerges, and from time 5 to time 10, player 2 increases his or her punishment of player 1. From time 10 to time 20, however, player 2's excessive, one-sided punishment demands player 1's unconditional cooperation, which results in the reverse exploitative relationship from that in case (b).

Figure 6

The degree of (a) exploitation $(y_{\mathrm{e}}^{*}-x_{\mathrm{e}}^{*})$ and (b) cooperation $(x_{\mathrm{e}}^{*}+y_{\mathrm{e}}^{*})/2$ in the final state of learning dynamics is plotted by a color, against the initial condition $(x_D^o,x_C^o)$ for $(T,R,P,S)=(5,4.5,1,0)$. The horizontal (vertical) axis indicates $x_D^o$ ($x_C^o$), and the player 2's strategy is fixed at $y_C^o=0.8,y_D^o=0.2$. In this case, only pure CC and DD strategies are stable fixed points for learning dynamics. The basin to the pure CC (DD) strategy is plotted by white (black) points in the right figure.

Figure 7

Panel (a) shows both players' payoffs $u_{\mathrm{e}}^{*}$ (blue) and $v_{\mathrm{e}}^{*}$ (red) when $x_D^o=0.1$ in $R=3.25$. (#) indicates the trajectory case classified in Sec. 4. Degrees of (b) exploitation of player 2 by player 1 ($:=y_{\mathrm{e}}^{*}-x_{\mathrm{e}}^{*}$) and (c) cooperation ($:=(x_{\mathrm{e}}^{*}+y_{\mathrm{e}}^{*})/2$) are plotted against the initial values of $(x_D^o,y_D^o)$. The horizontal (vertical) axis commonly indicates $y_D^o$ ($x_D^o$), and both $x_C^o$ and $y_C^o$ are fixed to 0.999. In both figures, $(T,P,S)=(5,1,0)$ is fixed, and $R$ equals 2.5, 2.75, 3.0, and 3.25 from left to right.

Figure 8

The equilibrium states for the learning dynamics in the case of $(T,R,P,S)=(5,4,0,1)$.