Learning in two dimensions and controlling in three: Generalizable drag reduction strategies for flows past circular cylinders through deep reinforcement learning

We investigate drag reduction mechanisms in flows past two- and three-dimensional cylinders controlled by surface actuators using deep reinforcement learning. We investigate 2D and 3D flows at Reynolds numbers up to 8000 and 4000, respectively. The learning agents are trained in planar flows at various Reynolds numbers, with constraints on the available actuation energy. The discovered actuation policies exhibit intriguing generalization capabilities, enabling open-loop control even for Reynolds numbers beyond their training range. Remarkably, the discovered two-dimensional controls, inducing delayed separation, are transferable to three-dimensional cylinder flows. We examine the trade-offs between drag reduction and energy input while discussing the associated mechanisms. The present paper demonstrates discovery of transferable and interpretable control strategies for bluff body flows through deep reinforcement learning with limited computational cost.

Figure 1

Illustration of the RL setup. Top: Snapshot of the vorticity field at $T=200$ for the two-dimensional flow at $\text{Re}=4000$. Bottom left: Sketch of the actuators and 2D simulation domain. Bottom right: Sketch of the actuators and 3D simulation domain, top view.

Figure 2

Results for policy $\pi$ (no regularizer weight) for a two-dimensional flow past a cylinder at $\text{Re}=4000$, with maximum actuation velocity up to $15\%$ of the cylinder velocity. Top left: Cumulative reward as a function of episodes simulated, plotted with a $95\%$ confidence interval. Top center: Drag coefficient as a function of time for the converged policy. Top right: Actions as fraction of cylinder velocity as a function of time. Bottom left: Time-averaged cylinder pressure coefficient, before and after actuators are activated. Bottom center: Angle of the points with zero shear (separation points) on cylinder surface. Bottom right: Vorticity field at $T=300$.

Figure 3

Summary of results when a regularizer is introduced into the reward function for $\text{Re}=4000$. Top row: Mean drag coefficient, mean squared lift coefficient, and mean flow separation angle as functions of the energy cost for policy $\pi$ (no regularizer) and policy ${\pi }^w$ (with regularizer). Bottom: Comparison of actions taken by the two policies, expressed as a fraction of the cylinder velocity, for $c=0.10$.

Figure 4

Summary of results for policy $\pi$ with varying maximum actuation velocities (varying $c$) and a fixed Reynolds number of 4000. Top left: Drag coefficient time series for different values of $c$. Top right: Actions time series for $c=0.05$, expressed as fraction of cylinder velocity. Bottom: vorticity field at several time instances, for $c=0.05$.

Figure 5

Summary of results for different Reynolds numbers and $c=0.15$. Top left and top center: Drag coefficient time series. Top right: Mean separation angle as a function of the Reynolds number for the uncontrolled case and a controlled case. Middle left and center: Vorticity field at $\text{Re}=8000$ before ($T=200$) and after actuation ($T=300$). Middle right: Actions as fraction of cylinder velocity for Re=8000. Bottom left and center: Vorticity field at $\text{Re}=500$ before ($T=200$) and after actuation ($T=300$). Bottom right: Cylinder pressure coefficient for $\text{Re}=500$ before and after actuators are activated.

Figure 6

Action pairs across Reynolds numbers and maximum actuation velocities for the two policies $\pi$ and ${\pi }^w$. For each plot, the horizontal axis corresponds to ${\theta }_a\in \{0^o,{45}^o,{90}^o,{135}^o\}$ and the vertical axis to ${\theta }_a\in \{{180}^o,-{45}^o,-{90}^o,-{135}^o\}$.

Figure 7

Flow past three-dimensional cylinder at $\text{Re}=1000,2000,$ and 4000. Drag coefficient time series for baseline case and for three different policies.

Figure 8

Left column: Slice through the $\mathit{xy}$ plane showing vorticity magnitude at $T=30$ for the uncontrolled cases (top to bottom: $\text{Re}=1000,2000,4000$). Right column: Slice through the $\mathit{xy}$ plane showing vorticity magnitude at $T=30$ for the controlled cases (top to bottom: $\text{Re}=1000,2000,4000$).

Figure 9

Vorticity magnitude volume rendering for $\text{Re}=2000$; left column shows the top views and right column shows the side views. The uncontrolled case corresponds to the first row, while the second row shows the controlled case.