Efficient reinforcement learning with partial observables for fluid flow control

Even if the trajectory in a viscous flow system stays within a low dimensional subspace in the state space, reinforcement learning (RL) requires many observables in the active control problem. This is because the observables are assumed to follow a policy-independent Markov decision process in the usual RL framework and full observation of the system is required to satisfy this assumption. Although RL with a partially observable condition is generally a difficult task, we construct a consistent algorithm with the condition using the low dimensional property of viscous flow. Using typical examples of active flow control, we show that our algorithm is more stable and efficient than the existing RL algorithms, even under a small number of observables.

Figure 1

Learning history of the blowing and suction problem. $\langle C_D\rangle$ is the time-averaged drag coefficient $C_D$ within one episode.

Figure 2

Time variation of the drag coefficient $C_D$ (lines). The dotted lines represent the time-averaged value for each case. Each color represents the same case as in Fig. 1. The inset shows the time series of normalized mass flow rate $Q_1^{*}$ [16].

Figure 3

Configuration of the cylinder forcing problem. Reinforcement learning optimizes the policy $\beta$, the direction of forcing, to maximize the average speed of the cylinder $\langle V_x\rangle$.

Figure 4

Learning history of the cylinder forcing problem. $\langle V_x\rangle$ is the time-averaged $V_x$ per episode.

Figure 5

Trajectories of the observables $V_y,L$, the action $\beta$, and the reword $V_x$ under the optimal policies for different numbers of observables $N_{\mathrm{o}}$.

Figure 6

Configuration of the swimming problem. The hinge is driven by the external torque $T$ acting on the plate 1. Reinforcement learning optimizes $T$ to maximize the average speed of the hinge $-\langle {\dot{x}}_h\rangle$.

Figure 7

Learning history of the swimming problem for different number of observables. $\langle {\dot{x}}_h\rangle$ is the time-averaged ${\dot{x}}_h$ within an episode.