Deep reinforcement learning for feedback control in a collective flashing ratchet

A collective flashing ratchet transports Brownian particles using a spatially periodic, asymmetric, and time-dependent on-off switchable potential. The net current of the particles in this system can be substantially increased by feedback control based on the particle positions. Several feedback policies for maximizing the current have been proposed, but optimal policies have not been found for a moderate number of particles. Here, we use deep reinforcement learning (RL) to find optimal policies, with results showing that policies built with a suitable neural network architecture outperform the previous policies. Moreover, even in a time-delayed feedback situation where the on-off switching of the potential is delayed, we demonstrate that the policies provided by deep RL provide higher currents than the previous strategies.

Figure 1

(a) $N=1$ case. Top: Potential $U$ and trained value network $V_{\varphi }$ as a function of position $x$ are denoted by blue and orange lines, respectively. Bottom: The solid line denotes the probability of switching on the potential ($p_{\mathrm{on}}$) as a function of $x$ for the greedy policy. The dotted line represents $p_{\mathrm{on}}$ of the trained MLP policy. (b) Illustration of a MLP with two hidden layers for the policy network ${\pi }_{\theta }$.

Figure 2

(a) DeepSets architecture for the policy network ${\pi }_{\theta }$. $H$ is the number of hidden units for each layer. (b) Decision boundaries from a trained MLP (left) and trained DeepSets (right) for $N=2$. The white contour denotes where the mean force $f(x_1,x_2)$ is zero. The red contour is $p_{\mathrm{on}}=0.5$ from the trained policy network ${\pi }_{\theta }$. The color gradient represents the trained value network $V_{\varphi }$. (c) Current ${\mathbb{E}}_{\alpha }\left[\dot{x}\right]$ as a function of $N$ for each policy $\alpha$. Throughout this Letter, error bars represent the standard deviation of the current measured from the realized trajectory ensemble over the period $t=50L^2/D$.

Figure 3

(a) Architecture of policy network ${\pi }_{\theta }$ augmented with an RNN. (b) Time-delayed feedback results for the greedy, MND, MLP (only for $N=1$), DeepSets (for $N>1$), and RNN policies at increasing $N$. The black dotted lines denote the current of the periodic switching policy.