Stabilizing viscous extensional flows using reinforcement learning

The four-roll mill, wherein four identical cylinders undergo rotation of identical magnitude but alternate signs, was originally proposed by G. I. Taylor to create local extensional flows and study their ability to deform small liquid drops. Since an extensional flow has an unstable eigendirection, a drop located at the flow stagnation point will have a tendency to escape. This unstable dynamics can, however, be stabilized using, e.g., a modulation of the rotation rates of the cylinders. Here we use reinforcement learning, a branch of machine learning devoted to the optimal selection of actions based on cumulative rewards, in order to devise a stabilization algorithm for the four-roll mill flow. The flow is modelled as the linear superposition of four two-dimensional rotlets and the drop is treated as a rigid spherical particle smaller than all other length scales in the problem. Unlike previous attempts to devise control, we take a probabilistic approach whereby speed adjustments are drawn from a probability density function whose shape is improved over time via a form of gradient ascent know as actor-critic method. With enough training, our algorithm is able to precisely control the drop and keep it close to the stagnation point for as long as needed. We explore the impact of the physical and learning parameters on the effectiveness of the control and demonstrate the robustness of the algorithm against thermal noise. We finally show that reinforcement learning can provide a control algorithm effective for all initial positions and that can be adapted to limit the magnitude of the flow extension near the position of the drop.

Figure 1

Schematic representation of the four-roll mill setup where four rotating cylinders of radius $a$ and located on a square of side length $2L$ create an extensional flow near the center of the ($x,y$) coordinate system. The angular velocities of the cylinders are denoted by ${\mathrm{\Omega }}_{ij}$, with $i,j=1,2$. The goal of our reinforcement learning algorithm is to stabilize the motion of viscous drops inside a small square area shown schematically in gray, following the experiments in Ref. [3] (not to scale). The streamlines shown are from the model flow from Eq. (2) in the case where the angular velocities follow the symmetric values from Eq. (3).

Figure 2

Illustration of the learned policy for a drop starting at the dimensionless location ${\mathit{x}}_0=[-0.03,0.02]$ with the algorithmic choices $\alpha =\beta =10, \gamma =0.95, p=1, N=4$ and for rotation wiggle rooms of $70\%$ of the default angular velocity of each cylinder. (a) Drop undergoing a zigzag motion, first toward the $x$ axis and then toward the origin. The blue portion (i.e. in (11) quadrant) of the trajectory indicates when the control is done by changing the value of ${\mathrm{\Omega }}_{12}$ (its value is plotted in (b), with rotating cylinder shown as inset) while in the green curve (i.e. in (21) quadrant) the change is done by tuning the value of ${\mathrm{\Omega }}_{11}$ (its value is shown in (c), with rotating cylinder shown as inset).

Figure 3

Illustration of different policies learned after starting from six different locations: ${\mathit{x}}_1=[-0.03,0.02], {\mathit{x}}_2=[-0.01,0.03], {\mathit{x}}_3=[0.02,0.02], {\mathit{x}}_4=[0.03,-0.02], {\mathit{x}}_5=[0.01,-0.03]$, and ${\mathit{x}}_6=[-0.02,-0.04]$. The learning parameters are $\alpha =\beta =10, N=4, \gamma =0.95$, and $p=1$. In each case we use as many batches as required to get the final position of the drop below 0.0015. We plot the resulting policy in thick solid lines, while the thin lines show what the drop would do in the absence of speed control.

Figure 4

Average final distance of the drop from the stagnation point, $\overline{|{\mathit{x}}_f|}$, as a function of the batch number for different values of the discount factor, $\gamma$ (see text for the values of the other parameters).

Figure 5

Average final distance of the drop from the stagnation point, $\overline{|{\mathit{x}}_f|}$, as a function of the batch number for different values of peakedness $p$ of the reward function (see text for the values of the other parameters).

Figure 6

Average final distance of the drop from the origin, $\overline{|{\mathit{x}}_f|}$, as a function of the batch number for different values of the gradient ascent parameters $\alpha =\beta$ (see text for the values of the other parameters).

Figure 7

Average final distance of the drop from the origin, $\overline{|{\mathit{x}}_f|}$, as a function of the batch number for different values of the array size $N$ (see text for the values of the other parameters).

Figure 8

Average final distance of the drop from the stagnation point, $\overline{|{\mathit{x}}_f|}$, as a function of the batch number for different values of the length of episodes $t_{\text{max}}$ (see text for the values of the other parameters).

Figure 9

Average final distance of the drop from the origin, $\overline{|{\mathit{x}}_f|}$, as a function of the batch number for four different rotation wiggle rooms $w$ (see text for the values of the other parameters).

Figure 10

Variance and improvement during learning: Average final distance of the drop from the origin $\overline{|{\mathit{x}}_f|}$ and range of distances as a function of the batch number (see text for the values of the learning parameters).

Figure 11

Cumulative distribution function (CDF) of the distance of the drop from the origin after 1000 episodes (i.e., in batch 10; see text for the values of the learning parameters).

Figure 12

Proportion of episodes landing within a dimensionless distance 0.0078 of the origin for different values of the Péclet number ($\mathrm{Pe}$) using the algorithm trained in the absence of noise (see text for the values of the learning parameters). The original experiments in Ref. [3] had $\mathrm{Pe}\approx 1.2\times {10}^{12}$.

Figure 13

Final distances from the origin for 625 trajectories starting in the $[-0.05,0.05]\times [-0.05,0.05]$ space using the optimal policy ${\pi }_{*}$ obtained from the single dimensionless starting position ${\mathit{x}}_0=[-0.03,0.02]$. The time and learning parameters $dt=0.025, t_{\mathrm{lag}}=0.0125, t_1=t_2=0.005, \gamma =0.95, p=1, \alpha =\beta =10, N=4$, and $t_{\text{max}}=40$.

Figure 14

Same as Fig. 13 in the case where thermal noise is added to the drop trajectory with $\mathrm{Pe}={10}^5$.

Figure 15

Ratio between the extension rate $\lambda \left(t\right)$ and the reference value ${\lambda }_0$ at the center of the uncontrolled apparatus. (a) Without scaling, the extension rate routinely varies by $15\%$ around ${\lambda }_0$. (b) Using the piecewise linear scaling from Eq. (31), the fluctuations of the extension rate near the drop are significantly reduced. The time and learning parameters are $dt=0.025, t_{\mathrm{lag}}=0.0125, dt=t_1=t_2=0.005, \gamma =0.95, p=1, \alpha =\beta =10, N=4$, and $t_{\text{max}}=40$ while the Péclet number is $\mathrm{Pe}={10}^5$.