Smart inertial particles

We performed a numerical study to train smart inertial particles to target specific flow regions with high vorticity through the use of reinforcement learning algorithms. The particles are able to actively change their size to modify their inertia and density. In short, using local measurements of the flow vorticity, the smart particle explores the interplay between its choices of size and its dynamical behavior in the flow environment. This allows it to accumulate experience and learn approximately optimal strategies of how to modulate its size in order to reach the target high-vorticity regions. We consider flows with different complexities: a two-dimensional stationary Taylor-Green-like configuration, a two-dimensional time-dependent flow, and finally a three-dimensional flow given by the stationary Arnold-Beltrami-Childress (ABC) helical flow. We show that smart particles are able to learn how to reach extremely intense vortical structures in all the tackled cases.

Figure 1

(a) Sketch of the reinforcement learning agent-environment interactions. (b) Graphical summary of a typical trajectory for a smart particle in a generic two-dimensional flow. State changes occur at times $t_n,t_{n+1},t_{n+2}$, ...when the particle enters new vorticity regions that are separated by isolines of the vorticity field ${\mathrm{\Omega }}_z$ (green lines). In each state, the particle chooses its size (the action).

Figure 2

(a) An illustrative sample of the isolines delimiting the $N_s=21$ vorticity states. Values corresponding to the underlying vorticity isolines are shown in red. (b) Set of pairs ($\mathrm{St},\beta$) corresponding to the selection of the $N_a=11$ actions. Smart particles are allowed to select each one of the available actions freely (A). We also highlight four different cases corresponding to four possible naive strategies: (B) heavy particle with $(\mathrm{St}=0.58,\beta =0.36)$; (C) light particle with $(\mathrm{St}=0.83,\beta =2.4)$; (D) tracer with $(\mathrm{St}=0.01,\beta =1)$; and (E) simple bimodal policy with only two actions: The particle has one constant low density $(\mathrm{St}=0.83,\beta =2.7)$ [light] in strong negative vorticity states or one constant high density $(\text{St}=0.71,\beta =0.18)$ [heavy] in all the other states (see also text).

Figure 3

Dependence of the normalized total gain, $\tilde{\mathrm{\Sigma }}$ in Eq. (10), on the episode $E$ for 10 different learning processes (black curves). Every point represents an average over a sliding window of 500 episodes. The red line shows the maximal possible reward.

Figure 4

Left column: Particle positions along 10 representative trajectories plotted after an initial transient for the cases studied in Fig. 2 (except for tracer particles which fill space uniformly). From top to bottom: Smart particles (A), heavy (B), light (C), and simple bimodal policy (E). Middle column: probability density function of vorticity sampled along 1000 particle trajectories for each of the cases studied (red curves). The PDFs are compared with the distribution for tracer particles (shaded curve). The blue shaded region marks vorticity levels that are unique to the target region. Right column: the policies (the action $a$ the particle takes given a state $s$) used in the considered cases.

Figure 5

The normalized learning gain, $\tilde{\mathrm{\Sigma }}\left(E\right)$ in Eq. (10) against the episodes $E$ for the $\epsilon$-greedy reinforcement learning algorithm. The representation is comparable to Fig. 3 (scale on left vertical axis). The two continuous curves represent the adiabatic decreasing of both the exploration parameter $\epsilon$ and the learning rate $\alpha$ (scale on right vertical axis).

Figure 6

The normalized learning gain $\tilde{\mathrm{\Sigma }}\left(E\right)$ as a function of the time horizon of the learning process ${(1-\gamma )}^{-1}$ averaged over samples of 10 different learning trials. Error bars are estimated on the basis of the scatter inside each sample.

Figure 7

Dependence of the normalized learning gain, $\tilde{\mathrm{\Sigma }}$, for 10 different learning processes (black curves) on the two-dimensional time-dependent flow. Every point represents an average over a sliding window of 500 episodes. The three policies shown to the right are for the cases with best, middle, and worst final gain.

Figure 8

Upper row: average $\mathrm{\Omega }$ sampled along the time evolution of 100 particles for each analyzed type: A, smart (purple); B, heavy (orange); C, light (red); D, tracer (green); and E, particle with bimodal policy (magenta). The average vorticity is periodic with the period $T=2\pi /{\omega }_0$ of the underlying flow. Middle row: spatial distribution of 100 smart particles at the four different times $T_1,T_2,T_3$, and $T_4$ highlighted in the top panel. Lower row: the same but for light particles. The last column on the right shows scatter plots of $\mathrm{\Omega }$ for smart particles (top) and light ones (bottom) throughout the entire period. The shadowed gray curve represents the instantaneous maximum negative vorticity throughout the flow.

Figure 9

Result from training of particles in ABC flow with weakly broken symmetry ($2A=B=C=1$). (a) Normalized total gain $\tilde{\mathrm{\Sigma }}={\langle \left|\mathbf{\Omega }\right|}^3{\rangle }^{1/3}$ [Eq. (10)] as a function of episode $E$. The maximal vorticity in the flow is shown as black dashed line. Red dashed line shows the steady-state average ${\langle \left|\mathbf{\Omega }\right|}^3{\rangle }_{\infty }^{1/3}$ along the trajectories of many naive light particles. Solid blue line shows the corresponding average ${\langle \left|\mathbf{\Omega }\right|}^3{\rangle }_{\infty }^{1/3}$ using the best policy obtained. Black dashed line shows maximal vorticity. (b) Distribution of magnitude of vorticity $\left|\mathbf{\Omega }\right|$ for smart particles (blue peak), naive particles with $\beta =1$ (green), and naive particles with $\beta =3$ (red region and peak). (c) Frequency of optimal action for each state using the 10 final policies for the cases displayed in panel (a). Black boxes highlight the best policy in panel (a).

Figure 10

Particle trajectories in an ABC flow ($4A=2B=C=1$) starting from an identical initial condition of a naive light particle (a) and a smart particle (b). The trajectories show that a smart particle manages to find the optimal vertical principal vortices independent of the initial condition, while the naive light particle sometimes get stuck in suboptimal horizontal principal vortices. The plotted particle sizes are proportional to the value of $\beta$. The vorticity field is color coded. To allow the visualization of particle trajectories, we have represented only a slice of it.

Figure 11

Flow distribution of different components of $\mathbf{\Omega }$: ${\mathrm{\Omega }}_x$ (red), ${\mathrm{\Omega }}_y$ (green), and ${\mathrm{\Omega }}_z$ (blue) for an ABC flow with strongly broken symmetry, $4A=2B=C=1$.

Figure 12

Result from training of particles in an ABC flow with strongly broken symmetry ($4A=2B=C=1$). Data are displayed in the same manner as Fig. 9. [(a)–(c)] Normalized total gain $\tilde{\mathrm{\Sigma }}\left(E\right)$ as a function of episode during training using the components ${\mathrm{\Omega }}_x$ (a), ${\mathrm{\Omega }}_y$ (b), and ${\mathrm{\Omega }}_z$ (c) as state. [(d)–(f)] Corresponding frequency of optimal action for the final policies in panels (a)–(c).

Figure 13

Evaluation of the performance of smart particles in an ABC flow ($4A=2B=C=1$) with $N_a=2$ actions ($\beta =0$ or $\beta =3$) and ${\mathrm{\Omega }}_x$ divided into $N_s=11$ states. (a) Solid black lines indicate the best policy obtained by the smart particles. This policy coincides with the global optimal policy for the set of states and actions used. Colors indicate frequency of the optimal action for each state based on 100 training sessions. (b) Green bars show distribution of average normalized return ${\tilde{R}}_{\mathrm{tot}}$ for all $N_s^{N_a}=2048$ possible $Q$-value matrices. Purple bars show distribution of ${\tilde{R}}_{\mathrm{tot}}$ for the 100 training sessions.