Deep learning for in situ data compression of large turbulent flow simulations

As the size of turbulent flow simulations continues to grow, in situ data compression is becoming increasingly important for visualization, analysis, and restart checkpointing. For these applications, single-pass compression techniques with low computational and communication overhead are crucial. In this paper we present a deep-learning approach to in situ compression using an autoencoder architecture that is customized for three-dimensional turbulent flows and is well suited for contemporary heterogeneous computing resources. The autoencoder is compared against a recently introduced randomized single-pass singular value decomposition (SVD) for three different canonical turbulent flows: decaying homogeneous isotropic turbulence, a Taylor-Green vortex, and turbulent channel flow. Our proposed fully convolutional autoencoder architecture compresses turbulent flow snapshots by a factor of 64 with a single pass, allows for arbitrarily sized input fields, is cheaper to compute than the randomized single-pass SVD for typical simulation sizes, performs well on unseen flow configurations, and has been made publicly available. The results reported here show that the autoencoder dramatically outperforms a randomized single-pass SVD with similar compression ratio and yields comparable performance to a higher-rank decomposition with an order of magnitude less compression in regard to preserving a number of important statistical quantities such as turbulent kinetic energy, enstrophy, and Reynolds stresses.

Figure 1

Singular value decomposition and its low-rank approximation.

Figure 2

Architecture of the proposed deep convolutional autoencoder.

Figure 3

Velocity magnitudes for the full simulation and lossy restarts of decaying isotropic turbulence with ${\text{Re}}_{\tau }=89$. Snapshots are shown of (a) the flow field at the beginning of the restart ($t=7$) and (b) the various cases at the end of the simulation ($t=10$).

Figure 4

Investigation of the statistical properties of the various lossy restarts: (a) kinetic energy spectra for the various simulations at the beginning and end of the restart as well as the integral length scale [see Eq. (18)] for the lossy restarts compared to the full simulation as a function of time and (b) PDF of longitudinal velocity gradients for the various simulations at the beginning and end of the restart as well as the KL divergence of the PDFs for the lossy restarts from the full simulation as a function of time. In both (a) and (b) the left and middle plots depict the kinetic energy spectra and the PDFs of the longitudinal velocity gradients at the beginning and end of the restarted simulations. The rightmost plots depict a measure of the errors in these statistical quantities as a function of time.

Figure 5

Comparison of the computational costs for the three compression methods as a function of the input data size.

Figure 6

Velocity magnitudes for the full simulation and lossy restarts of decaying Taylor-Green vortex. Snapshots are shown of (a) the flow field at the beginning of the restart ($t=14$) and (b) the various cases at the end of the simulation ($t=20$).

Figure 7

(a) Total and (b) turbulent kinetic energies and (c) total enstrophy for the full and restarted Taylor-Green vortex simulations as a function of time.

Figure 8

Velocity magnitudes for the full simulation and lossy restarts of the channel flow problem at the beginning of the restart.

Figure 9

Time-averaged normalized streamwise velocity as a function of the normalized wall distance for the full and restarted simulations.

Figure 10

Reynolds stresses and turbulent kinetic energy normalized by the squared friction velocity as a function of the normalized wall distance for the various lossy restart cases. The corresponding values of the full simulation are shown by the faded lines in each plot.

Figure 11

Production $\mathcal{P}$ and dissipation $\varepsilon$ of turbulent kinetic energy for the full and restarted simulations as a function of the normalized wall distance.