Learning to control active matter

The study of active matter has revealed novel non-equilibrium collective behaviors, illustrating their potential as a new materials platform. However, most work treat active matter as unregulated systems with uniform microscopic energy input, which we refer to as activity. In contrast, functionality in biological materials results from regulating and controlling activity locally over space and time, as has only recently become experimentally possible for engineered active matter. Designing functionality requires navigation of the high-dimensional space of spatio-temporal activity patterns, but brute force approaches are unlikely to be successful without system-specific intuition. Here, we apply reinforcement learning to the task of inducing net transport in a specific direction for a simulated system of Vicsek-like self-propelled disks using a spotlight that increases activity locally. The resulting time-varying patterns of activity learned exploit the distinct physics of the strong and weak coupling regimes. Our work shows how reinforcement learning can reveal physically interpretable protocols for controlling collective behavior in non-equilibrium systems.

Figure 1

Reinforcement learning provides a framework to control active matter by modulating activity over space and time. Recent experimental advances allow local modulation of activity by light in diverse active matter systems [6, 7, 8, 9, 10, 11, 12]. We consider a framework in which a reinforcement learning (RL) agent controls the illuminated region—its location, size, and shape—in order to achieve a desired non-equilibrium organization. Particle positions and velocity are coarse-grained, and passed to an RL agent, which decides where to place the light. The active matter system responds to the agent's choices of illumination; RL receives a reward based on this response and updates its control protocol accordingly.

Figure 2

Reinforcement learning generates distinct protocols to induce directional transport in self-propelled disks at weak and strong coupling. Weak coupling: (a)–(d); strong coupling: (e)–(g). [(a), (e)] As the RL setup trains, average x momentum during a training episode increases. X momentum remains highly stochastic (grey curves), but running average of (a) 500 or (e) 1000 episodes shows clear improvement (black curve). [(b), (f)] Training changes spotlight shape and movement in distinctive ways at weak and strong coupling in order to induce rightward transport. [(c), (g)] Kymographs of spotlight intensity, particle density, and particle x velocity for the learned protocols in the weak- and strong coupling regimes. Kymographs focus on the one spatial dimension important for the target behavior, but some information is lost in the x projection. Examples of the full system dynamics are available in Sup Movies 2,3 within the Supplemental Material [37]. (d) For weak coupling, averaging over multiple aligned periods of the learned policy shows that the spotlight moves from left to right at a well-defined velocity, with traveling wave of particle density with positive x velocity carried along.

Figure 3

Weak coupling protocol can be interpreted as a purification process. (a) As spotlight moves right, left-moving particles exit the spotlight (and become inactive) sooner than right-moving particles that cotranslate with spotlight. Consequently,–x momentum of left movers is quenched at the left edge of the spotlight while +x momentum is maintained within the spotlight. At steady state, density lost by exiting particles is replenished by particles that lie ahead of spotlight's path. (b) Phase diagram for 1D model in (a). X momentum as function of normalized spotlight velocity ($\frac{v_{\gamma }}{v_p}$) and normalized spotlight length ($\frac{l_x}{L}$). X momentum is maximized for $v_{\gamma }\approx v_p$ and for intermediate $l_x$ (black star), close to parameters identified by reinforcement learning (RL) (white star, black-dotted line). Error bars represent the standard deviation of distributions for the trained protocol. (c) We trained RL on self-propelled systems with different light-enhanced activity $v_p$. Spotlight speed $v_{\gamma }$ scales with $v_p$ as predicted by theory (black line). Boxplots extend from lower to upper quartile of velocity distribution, with a line at the median.

Figure 4

Order parameters quantify how learned protocols switch from purification at weak coupling to a flocking-based strategy at strong coupling. (a) Snapshot traces of collective direction (bottom) and ratio between area of spotlight and system area (top) for the strong coupling protocol. Measurements of the two quantities are made at the same time in simulation. (b) Average spotlight area as a function of collective polarity. At strong and intermediate coupling, the spotlight provides maximum illumination when the collective polarity points rightward, and provides less illumination when the collective polarity points leftward. There is no relation between collective polarity and spotlight size at weak coupling. This transition is consistent with correlations between spotlight area and collective polarity as a function of coupling (Fig. 8). (c) Order parameters $\langle P\rangle ,\langle O\rangle$ quantitatively track nature of protocols from weak to strong coupling.

Figure 5

Increasing coupling sharpens the collective polarity distributions generated in fully-trained trained protocols. For a range of alignment couplings, we train reinforcement learning agents and evaluate the polarity distributions their converged policies generate. We compute the circular standard deviation at each instance in time and create boxplots to assess the resulting distributions. Each distribution consists of $N=2.0\times {10}^5$ circular standard deviations, drawn from the frames of the same simulations analyzed in Fig. 4. Each circular standard deviation in turn is computed from the polarities of the $n=144$ particles in the simulation. Boxplots indicate the lower and upper quartiles, with an interior line indicating the median. As coupling increases, the polarity standard deviation decreases, indicating the development of a well-defined collective polarity.

Figure 6

In a fully-activated Vicsek system, increasing coupling sharpens the collective polarity distribution. For simulations with permanently activated particles run at a range of polarities, we compute the circular standard deviation at each instance in time and create boxplots to assess the resulting distributions. Each distribution consists of $N=5.0\times {10}^3$ circular standard deviations, drawn from the frames of a simulation where all particles are activated and coupled at the corresponding coupling values. Each circular standard deviation in turn is computed from the polarities of the $n=144$ particles in the simulation. Boxplots indicate the lower and upper quartiles, with an interior line indicating the median. The system exhibits a crossover regime for polarity standard deviation at a coupling of approximately $k={10}^{-1.5}$, where the standard deviation decreases and the distribution of standard deviations becomes more peaked as well. This indicates the development of a well-defined collective polarity.

Figure 7

Learning to induce transport in a strongly coupled dilute system with flocking domains. All nondilute data are taken from simulations discussed in Fig. 4. (a) In the dilute regime, average x momentum transferred by a fully-trained reinforcement learning agent is approximately similar to the amount in the weak coupling regime. (b) Schematic of flocking domains in the dilute, strongly-coupled regime. (c) We evaluate the spotlight x velocity $v_{\gamma }$ in the dilute regime, as well as over the range of couplings discussed in Fig. 4. At weak coupling, the spotlight moves at the speed predicted by theory (black line). At intermediate and strong coupling $v_{\gamma }$ diverges from the prediction and additionally is more stochastic. In the dilute regime, $v_{\gamma }$ has an even larger spread. Boxplots extend from lower to upper quartile of velocity distribution, with a line at the median. (d) Average spotlight area as a function of collective polarity. In the dilute regime, the spotlight is larger when the average polarity points to the right.

Figure 8

Spotlight size is anticorrelated with collective motion away from the preferred direction at strong couplings. We calculate Spearman's r between spotlight area and the absolute value of the angle between collective polarity and the +x direction for increasing values of coupling. As coupling increases, spotlight area and angle magnitude become more strongly anticorrelated.

Figure 9

Learning on 4x larger systems recapitulates the same strategies found in smaller systems for both weak and strong coupling regimes. Simulations are run with 576 particles, keeping all other physical parameters constant unless specifically noted. The RL approach and hyperparameters were also identical to those used in the smaller systems. (a) In the weak-coupling regime with $k=0$, we find a qualitatively similar strategy to the one identified in Fig. 2, where the spotlight extends laterally and moves rightward at a well-defined velocity, close to the prediction of the simple model proposed in the main text; see also Sup Movie 5 within the Supplemental Material [37]. Boxplots extend from lower to upper quartile of velocity distribution, with a line at the median. Training was performed for $1.4\times {10}^4$ episodes. (b) In the strong coupling regime with $k=.1$, we find a quantitatively similar strategy to the one identified in Fig. 2, where the spotlight rectifies rightward motion by increasing activity when particles collectively point to the right; see also Sup Movie 6 within the Supplemental Material [37]. Training was performed for $4.2\times {10}^3$ episodes.