Turbulence control for drag reduction through deep reinforcement learning

Deep reinforcement learning (DRL) was applied to turbulence control for drag reduction in direct numerical simulations of turbulent channel flow. Based on the wall shear stress information only, the DRL is capable of determining the optimal distribution of wall blowing and suction, which can reduce drag by 20%, comparable with previous wall-shear-based controls such as suboptimal control [Lee et al., J. Fluid Mech. 358, 245 (1998)] and neural-network-based control [Lee et al., Phys. Fluids 9, 1740 (1997)]. However, our DRL-based control can determine the optimal amplitude of the wall actuation, which was not possible in previous controls. More importantly, from an analysis of the optimal actuation fields, two distinct types of drag reduction mechanisms are identified: the first cancels the near-wall sweep and ejection events, whereas the second mechanism suppresses the streamwise vortices near the wall, which is similar to that of the suboptimal control. This study demonstrates the successful application of DRL to turbulence control and its physical interpretation.

Figure 1

Structure of TD3 algorithm with parameter space noise. Critic net consists of two identical networks, Critic net1 and Critic net2. By comparing the $q$ predicted in Critic net1 and Critic net2, the two critic weights ($\theta$) are updated using the smaller $q$ of the two. For the efficient exploration, we form a noise actor (exploration policy) by adding Gaussian noise (exploration noise) to the weight ($\psi$) of the main actor (main policy). Here, $a_t^{\prime }=a_t$ with space noise.

Figure 2

Comparison of the learning behavior with respect to the noise introduced by exploration. The blue line depicts the pressure gradient changes when random noise is used as the exploration noise, whereas the red line indicates the pressure gradient when the OU-process is used as the exploration noise.

Figure 3

Structures of the actor and critic networks. The actor network consists of only convolutional layers, and the critic network consists of convolutional layers and fully connected layers. Here, Conv denotes convolutional layers, and Avg pooling denotes average pooling.

Figure 4

Learning performance of the normalized reward in terms of episode for (a) $\gamma =0.95$ and (b) $\gamma =0.99$.

Figure 5

Test results of performance of the trained DRL models in terms of the mean pressure gradient to maintain a constant flow rate: (a) test results at ${\text{Re}}_{\tau }=180$ of the DRL models trained at ${\text{Re}}_{\tau }=180$ and (b) test results at ${\text{Re}}_{\tau }=360$ of the DRL models trained at ${\text{Re}}_{\tau }=180$. $u_{\tau }$ used for nondimensionalization of the pressure gradient and time is that of the unmanipulated flow. The inset shows early behavior of the pressure gradient by all models.

Figure 6

rms velocity fluctuations normalized by the wall shear velocity of the unmanipulated flow at ${\text{Re}}_{\tau }=180$. Parts (a), (b), and (c) illustrate the rms values in the streamwise, wall-normal, and spanwise directions, respectively. Parts (d), (e), and (f) provide the rms values of the fluctuation vorticity.

Figure 7

Comparison of the wall actuation profile of the tested models for the same flow field (a), (c) along the streamwise direction and (b),(d) along the spanwise direction.

Figure 8

Scatter plot between the actuations of (a) DRL-($\frac{\partial u}{\partial y}$), (b) DRL-($\frac{\partial u}{\partial y},\frac{\partial w}{\partial y}$), (c) OPP model, and (d) SUB model against the actuation of the DRL-($\frac{\partial w}{\partial y}$) model.

Figure 9

Streamwise wall shear stress field (flood contours) and the actuation field (line contours) for (a) DRL-($\frac{\partial u}{\partial y}$), (b) DRL-($\frac{\partial u}{\partial y},\frac{\partial w}{\partial y}$), (c) SUB model, and (d) DRL-($\frac{\partial w}{\partial y}$). Parts (a)–(d) show overlap of the streamwise wall shear field (flood contour) and each model's action field (line contour). Spanwise gradient of the spanwise wall shear stress field (flood contours) and the actuation field (line contours) for (e) SUB model and (f) DRL-($\frac{\partial w}{\partial y}$). Among line contours, the dashed line denotes negative values.

Figure 10

Scatter plot between shear stress and actuation with correlation coefficient: (a), (b) scatter plots of the actuation of DRL-($\frac{\partial u}{\partial y}$) and DRL-($\frac{\partial u}{\partial y},\frac{\partial w}{\partial y}$) with respect to the normalized streamwise shear stress, and (c), (d) scatter plots of the actuation of DRL-($\frac{\partial w}{\partial y}$) and SUB model with respect to the normalized $z$-gradient of the spanwise shear stress.

Figure 11

Vertical velocity field (flood contour) with $u\text{−}v$ velocity vector field and the wall actuation of the DRL-($\frac{\partial u}{\partial y},\frac{\partial w}{\partial y}$) model. In the 1D actuation profile, the red line indicates normalized actuation, and the blue line indicates normalized streamwise wall-fluctuation shear stress.

Figure 12

Streamwise vorticity field (flood contour) with $v\text{−}w$ velocity vector field and the wall actuation of the DRL-($\frac{\partial w}{\partial y}$) model. In the 1D actuation profile, the red line indicates normalized actuation, and the green line indicates normalized $z$-gradient of the spanwise wall-swall shear stress.

Figure 13

Two-point correlation contour between $\frac{\partial w}{\partial y}{|}_w(0,0)$ and ${\omega }_x(\mathrm{\Delta }y,\mathrm{\Delta }z)$ in the $y\text{−}z$ plane for (a) unmanipulated field and (b) DRL-($\frac{\partial w}{\partial y}$)-controlled field, respectively. Two-point correlation between $\frac{\partial u^{\prime }}{\partial y}{|}_w(0,0)\mathrm{and}u(\mathrm{\Delta }x,\mathrm{\Delta }y)$ in the $x\text{−}y$ plane for (b) unmanipulated field and (d) DRL-($\frac{\partial u}{\partial y}$, $\frac{\partial w}{\partial y}$)-controlled field.