Flow Navigation by Smart Microswimmers via Reinforcement Learning

Smart active particles can acquire some limited knowledge of the fluid environment from simple mechanical cues and exert a control on their preferred steering direction. Their goal is to learn the best way to navigate by exploiting the underlying flow whenever possible. As an example, we focus our attention on smart gravitactic swimmers. These are active particles whose task is to reach the highest altitude within some time horizon, given the constraints enforced by fluid mechanics. By means of numerical experiments, we show that swimmers indeed learn nearly optimal strategies just by experience. A reinforcement learning algorithm allows particles to learn effective strategies even in difficult situations when, in the absence of control, they would end up being trapped by flow structures. These strategies are highly nontrivial and cannot be easily guessed in advance. This Letter illustrates the potential of reinforcement learning algorithms to model adaptive behavior in complex flows and paves the way towards the engineering of smart microswimmers that solve difficult navigation problems.

Figure 1

Top left: phase diagram of gyrotactic particles in a Taylor-Green vortex flow. Each circle represents one of the 25 investigated parameter pairs. Six distinct patterns emerge: confined (a)–(c) and unconfined (d)–(f). Top right: gyrotactic trajectories (red) for each of the 6 patterns plotted on a periodic domain. For the sake of representation, unconfined trajectories are reinjected in the basic $2\pi \times 2\pi$ domain. Trajectories for smart gravitactic particles, after learning the optimal policy to choose the preferred direction ${\mathbf{k}}_a$, are shown in blue. The vorticity field is shown in grey scale. Bottom: a set of representative trajectories at different learning episodes for smart gravitactic particles (blue) compared with typical trajectories for naive gyrotactic particles (red) confined in a trapping dynamics (case $C$ above).

Figure 2

Dependence of the learning gain $\mathrm{\Sigma }(E)$ in percentage vs $E$ for 10 different learning processes (grey curves). Point ($\mathrm{\Psi }=1$, $\mathrm{\Phi }=0.3$) region $C$ in Fig. 1. The value of $\mathrm{\Sigma }(E)$ that is visualized is averaged locally on a window of 500 episodes. The insets highlight which preferred directions the smart particle takes for each of the 12 states, according to three final approximately optimal strategies [these are the highest values of the $Q(s,a)$ matrices].

Figure 3

Left: final averaged learning gain ${\mathrm{\Sigma }}_{\mathrm{avg}}$ (in percentage) for different regions in the parameter space, where the average is made out of 10 different learning experiments. Right: example of the optimal actions for the smart gravitactic particle that succeeded to escape the confinement, parameters ($\mathrm{\Psi }=1$, $\mathrm{\Phi }=0.3$) in region $C$ in Fig. 1. Data obtained from an optimal training using $\epsilon$-greedy exploration. Notice that strategies obtained by permutations that respect the symmetries of the underlying flow would lead to the same learning gain.

Figure 4

Comparison between trajectories of naive gyrotactic (red) and smart gravitactic particles (blue) in a perturbed flow. Smart particles move with a policy obtained for the original flow. Above each column, we put $\beta$ values and the corresponding ${\mathrm{\Sigma }}_{\mathrm{avg}}$. Case ($\mathrm{\Psi }=0.3$, $\mathrm{\Phi }=0.1$) region $A$ in Fig. 1.