Understanding adaptive immune system as reinforcement learning

The adaptive immune system of vertebrates can detect, respond to, and memorize diverse pathogens from past experience. While the clonal selection of T helper (Th) cells is the simple and established mechanism to better recognize new pathogens, the question that still remains unexplored is how the Th cells can acquire better ways to bias the responses of immune cells for eliminating pathogens more efficiently by translating the recognized antigen information into regulatory signals. In this work, we address this problem by associating the adaptive immune network organized by the Th cells with reinforcement learning (RL). By employing recent advancements of network-based RL, we show that the Th immune network can acquire the association between antigen patterns of and the effective responses to pathogens. Moreover, the clonal selection as well as other intercellular interactions are derived as a learning rule of the network. We also demonstrate that the stationary clone-size distribution after learning shares characteristic features with those observed experimentally. Our theoretical framework may contribute to revising and renewing our understanding of adaptive immunity as a learning system.

Figure 1

Schematic diagram of the adaptive immune system in relation with network-based reinforcement learning. When an infection occurs, APCs engulf the pathogens and present multiple antigens as an antigen pattern $\mathbf{s}$. The Th cell population recognizes the antigen pattern $\mathbf{s}$ and biases the activities of the effector cells $\mathbf{a}$. The stochastic mapping ${\pi }_{\mathbf{n}}\left(\mathbf{a}\right|\mathbf{s})$ from $\mathbf{s}$ to $\mathbf{a}$ is regarded as the policy of the system parametrized by the abundance of the Th clones, $\mathbf{n}$. The effectiveness of the pattern of the effector activities $\mathbf{a}$ to the infection is represented as the reward $R(\mathbf{s},\mathbf{a})$. Learning of the system is achieved not by adjusting weights but by the modulation of the clone-size distribution $\mathbf{n}$.

Figure 2

Schematic diagram of the stochastic transition dynamics of the infected and uninfected states (the environment in the MDP) defined by Eq. (15). In all simulations, $q=0.9$ and $R_0=M/2$ are used.

Figure 3

(a) Trajectories of the Th clone size $\mathbf{n}$ along a learning trial. The parameter values are $N=100, K=5000, M=20$, and $P=30$. The clone size $\mathbf{n}$ is normalized by the initial abundance. This figure shows only 200 trajectories sampled evenly out of 5000 different clones at the stationary state to avoid complication of the plot. (b) The dynamics of rank-abundance distributions along the learning trial, which were calculated from the trajectories of $\mathbf{n}$ in (a). The red dots represent the stationary distribution after the learning, and the colored curves are the transient distributions calculated at training steps from $1\times {10}^3$ (blue) to $28\times {10}^3$ (yellow). (c) Statistics of learning curves of the Th cell population. The rewards normalized by its maximum value are shown as functions of the training step for 100 independent learning trials. The red curve is the average reward, and the yellow and blue regions show the range between 25th and 75th percentiles of the rewards and that between minimum and maximum of the rewards at each training step, respectively. (d) The stationary rank-abundance distributions for 100 independent learning trials in (c) are shown by gray curves. The red dots are the same as those in (b). See also Appendix pp1.

Figure 4

Heat map plots of the stationary normalized reward after learning as functions of (a) $K$ and $\{N,P\}$, (b) $K$ and $P$, (c) $N$ and $P$, and (d) $K$ and $N$. The other parameters are the same as those in Fig. 3. The stationary reward was calculated as the moving average of the last ${10}^4$ steps. (a)–(c) use the same color code.

Figure 5

(a) Trajectories of the Th clone size $\mathbf{n}$ along a learning trial for $K=100000$. The weights, $\mathbf{w}$, are also scaled to $\mathbf{w}/10$ for comparison with the experimental data. The other parameter values are the same as those in Fig. 3. The clone size $\mathbf{n}$ is normalized by the initial abundance. The figure shows only 200 trajectories sampled evenly out of $100000$. (b) The dynamics of the rank-abundance distributions along the learning trial, which were calculated from the trajectories of $\mathbf{n}$ in (a). The red dots represent the stationary distribution after the learning, and the colored curves are the transient distributions calculated at training steps from $1\times {10}^4$ (blue) to $77\times {10}^4$ (yellow). The magenta dots show the same stationary distribution as in Fig. 3, which is shown here for comparison. (c) Clone-size distributions and (d) rank-abundance distributions obtained from the model and an experiment. The green and cyan points in (c) are the clone-size distributions obtained by counting the TCR sequences of $\text{CD4}+$ T cells collected from peripheral blood of two healthy human donors (HV01 and HVD4) in Ref. [53], and the curves in (d) are the corresponding rank-abundance distributions. Red and blue points in (c) are the clone-size distributions obtained by resampling the clones from the rank distributions in (b) for the same numbers of total counts as in experiments 1 (red) and 2 (blue), and the red and blue curves in (d) are the corresponding rank-abundance distributions.

Figure 6

Learning dynamics and stationary abundance distribution under the adversarial environment. (a) Time series of the Th clone size $\mathbf{n}$ along learning. The parameter values are the same as in Fig. 3 in the main text. The clone size $\mathbf{n}$ is normalized by the initial abundance. This figure shows only 200 trajectories sampled evenly out of 5000 at the stationary state to avoid complication of the plot. (b) The dynamics of the rank-abundance distributions along the learning trial, which were calculated from the trajectories of $\mathbf{n}$ in (a). The red circles represent the stationary distribution after the learning, and the colored curves are the transient distributions calculated at training steps from $1\times {10}^3$ (blue) to $28\times {10}^3$ (yellow). (c) Statistics of learning curves of the Th cell population. The rewards normalized by its maximum value are obtained as functions of the training step for 100 independent learning trials. The red curve is the average reward, and the yellow and blue regions show the range between 25th and 75th percentiles of the rewards and that between minimum and maximum of the rewards at each training step, respectively. (d) The stationary rank-abundance distributions for the 100 independent learning trials in (c) are shown by gray curves. The red dots are the same as those in (b).

Figure 7

Learning dynamics and stationary abundance distribution for sparse $\mathbf{w}$. Each $w_{kj}$ is determined by following Eq. (E1) for $q_w=0.1$. (a) Time series of the Th clone size $\mathbf{n}$ along learning under the adversarial environment. The parameter values are the same as Fig. 3 in the main text. The clone size $\mathbf{n}$ is normalized by the initial abundance. This figure shows only 200 trajectories sampled evenly out of 5000 at the stationary state to avoid complication of the plot. (b) The dynamics of the rank-abundance distributions along the learning trial, which were calculated from the trajectories of $\mathbf{n}$ in (a). The red dots represent the stationary distribution after the learning, and the colored curves are the transient distributions calculated at training steps from $1\times {10}^3$ (blue) to $28\times {10}^3$ (yellow). The blue circles are the stationary distribution for $K=100000$. All the parameters but the sparsity are the same as those in Fig. 5 in the main text. (c) Statistics of learning curves of the Th cell population. The rewards normalized by its maximum value are obtained as functions of the training step for 100 independent learning trials. The red curve is the average reward, and the yellow and blue regions show the range between 25th and 75th percentiles of the rewards and that between minimum and maximum of the rewards at each training step, respectively. (d) The stationary rank-abundance distributions for the 100 independent learning trials in (c) are shown by gray curves. The red dots are the same as those in (b).

Figure 8

The average reward as functions of (a) the number of the Th clone types, $K$; (b) the number of the antigen types, $N$; (c) the number of the effector types, $M$; and (d) the number of the environmental state (the number of pathogens), $P$. All the other parameters are the same as those as in Fig. 3 of the main text. The average reward for each parameter value is obtained by the arithmetic average of 20 independent learning trials. The dashed lines indicate the parameter values used in Fig. 3 of the main text.

Figure 9

(a) Stationary rank-abundance distributions for different values of $P$. All the other parameters are the same as in Fig. 3 in the main text. (b) The distributions of fitness of clones for different values of $P$. Fitness is defined by $\mathrm{\Lambda }=log(n_i\left(t\right)/n_i\left(0\right))$. The skewness of the distributions are $-0.19, -0.35, -0.28, -0.79, -1.2$, and $-1.8$ for $P=3$, 6, 10, 30, 60, and 90, respectively.