Learning swimming escape patterns for larval fish under energy constraints

Swimming organisms can escape their predators by creating and harnessing unsteady flow fields through their body motions. Stochastic optimization and flow simulations have identified escape patterns that are consistent with those observed in natural larval swimmers. However, these patterns have been limited by the specification of a particular cost function and depend on a prescribed functional form of the body motion. Here, we deploy reinforcement learning to discover swimmer escape patterns for larval fish under energy constraints. The identified patterns include the C-start mechanism, in addition to more energetically efficient escapes. We find that maximizing distance with limited energy requires swimming via short bursts of accelerating motion interlinked with phases of gliding. The present, data efficient, reinforcement learning algorithm results in an array of patterns that reveal practical flow optimization principles for efficient swimming and the methodology can be transferred to the control of aquatic robotic devices operating under energy constraints.

Figure 1

Interaction between flow solver and reinforcement learning. (a) Geometrical model of a 5-day- postfertilization zebrafish larva [36]. The six curvature control points are indicated from top to bottom. (b) Simulation environment. The swimmer is placed inside a square domain and is simulated using the numerical flow solver described in Appendix pp1. (c) This is coupled with a deep reinforcement learning algorithm, that receives a state from the flow solver and sends back an action, deciding how the swimmer should act within the simulation.

Figure 2

Temporal evolution of the vorticity field for the optimized and learned escape patterns. (a) RL burst-coast escape pattern obtained by RL swimmer using energy budget $E_0$. (b) C-start escape pattern obtained by simulating the optimized parameter set reported in [36]. Orange regions represent positive flow vorticity and blue regions represent negative vorticity.

Figure 3

Comparison of the equal-energy RL burst-coast and C-start escape patterns. (a) Swimmer midpoint curvature ${\kappa }_{1/2}$ and normalized distance (in terms of swimmer length) as a function of time. The energy expended by the RL escape (blue) and C-start escape (red) is in both cases equal to $E_0$. (b) Vorticity fields and midline profiles for the RL escape (left) and the C-start escape (right). For both escapes, the midline of the swimmer is plotted over time within two intervals: The preparatory interval ($t\le 30\mathrm{m}\mathrm{s}$) and the propulsive interval ($30\mathrm{m}\mathrm{s}\le t\le 50\mathrm{m}\mathrm{s}$)

Figure 4

Influence of Reynolds number on escape efficiency. (a) The normalized distance per unit energy $\tilde{d}/\tilde{E}$ is plotted as a function of the mean Reynolds number ${Re}_{\overline{U}}$. $\tilde{d}=d/d_0$ and $\tilde{E}=E/E_0$. In blue: The burst coast escape sequence recorded by evaluating the RL control policy with energy budget $E_0$ at ${\nu }_0$, simulated for different viscosities. In red: The C-start escape sequence obtained by optimizing at ${\nu }_0$ [36], simulated at different viscosities. (b) A zoomed-in segment of (a) on the range ${\text{Re}}_{\overline{U}}\in [0,150]$. Displays the critical Reynolds number ${\text{Re}}_{\overline{U}}^{\text{crit}}$ at which the burst coast sequence becomes more efficient than the C start.

Figure 5

Influence of energy budget on escape distance and learned strategy. (a) Normalized escape distance as a function of normalized energy expenditure for ten escapes with energy budgets linearly spaced between $\frac{1}{3}{E}_0$ and $3E_0$. (b) Swimmer midpoint curvature ${\kappa }_{1/2}$, and swimmer speed as a function of time for the ten escapes. The higher energy escapes are represented by dark blue while the lower energy escapes are represented with light blue. The lightest blue curve corresponds to an energy budget $\frac{1}{3}{E}_0$, and the darkest blue corresponds to an energy budget $3E_0$. All quantities are plotted for escapes of duration $118.8\mathrm{m}\mathrm{s}$ and are simulated using the same RL policy (for training details see Appendix pp2). (c) Vorticity fields in the wake of four escapes of energy budgets, from left to right, $\frac{1}{3}{E}_0$, $\frac{9}{10}{E}_0$, $\frac{6}{5}{E}_0$, and $3E_0$. Snapshots of the vorticity field are taken at $t=75\mathrm{m}\mathrm{s}$ for all four escapes. The dotted, circled region denotes a secondary vortical structure which emerges at higher energy budgets.

Figure 6

Escape speed, drag coefficient, thrust force, and power consumption for equal-energy learned burst-coast pattern vs optimized C-start pattern. The energy expended by the RL escape (blue) and C-start escape (red) is in both cases equal to $E_0$. (a) Speed and drag coefficient of the burst-coast pattern learned through RL (blue) vs the C-start pattern (red). The speed is given in swimmer lengths per second ($L\text{/s}$) and the drag coefficient is defined as $C_d$ = $\frac{F_d}{\rho U^{2}L/2}$, where $F_d$ is the drag force experienced by the swimmer, $\rho$ is the density of water, $U$ is the swimmer speed relative to the fixed laboratory reference frame, and $L$ is the swimmer length. (b) Thrust force and deformation power of the burst-coast pattern (blue) vs the C-start pattern (red). For details on the computation and nondimensionalization of the drag force $F_d$, thrust force, and deformation power, see Appendix pp1.

Figure 7

Characterization of Lagrangian coherent structures at different energy budgets. Left: Vorticity fields overlayed onto the finite time Lyapunov exponent field (FTLE) of three different escapes. From left to right are the C start, the artificial swimmer with energy budget $E_0$, and the artificial swimmer with energy budget $3E_0$. The FTLE is visualized on a spectrum from white to black. The darker colors represent a higher value of the FTLE. The LCS are the “ridges” of the FTLE field, i.e., the darkest contours on the visualization. All FTLE fields are computed with the same integration length and displayed at the same time instant (see Appendix pp3). (a) A visualization of the lump of fluid ejected by the swimmer, the average vortex diameter $b$, and the peak C-bend midpoint curvature ${\kappa }_C$ for escapes with energy budgets linearly spaced between $\frac{1}{3}{E}_0$ and $3E_0$, shown to scale. The lumps of fluid are extracted by localizing the high ridge (LCS) values in the respective FTLE field. The average vortex diameter $b$ is normalized by the swimmer length $L$ as a function of the energy budget.