DiSTNet2D: Leveraging Long-Range Temporal Information for Efficient Segmentation and Tracking

Extracting long tracks and lineages from videomicroscopy requires an extremely low error rate, which is challenging on complex data sets of dense or deforming cells. Leveraging temporal context is key to overcoming this challenge. We propose DiSTNet2D, a new deep neural network architecture for two-dimensional (2D) cell segmentation and tracking that leverages both mid- and long-term temporal information. DiSTNet2D considers seven frames at the input and uses a postprocessing procedure that exploits information from the entire video to correct segmentation errors. DiSTNet2D outperforms two recent methods on two experimental data sets, one containing densely packed bacterial cells and the other containing eukaryotic cells. It is integrated into an ImageJ-based graphical user interface for 2D data visualization, curation, and training. Finally, we demonstrate the performance of DiSTNet2D on correlating the size and shape of cells with their transport properties over large statistics, for both bacterial and eukaryotic cells.

Figure 1

Overview of the method. (a) Model output for a eukaryotic data set frame pair. For each frame, EDM is the map of the Euclidean distance to the edge of each cell, and GDCM is the map of the geodesic distance to the center. For each pair of frames, for both forward ($t\to t+1$) and backward ($t+1\to t$) directions, $\mathit{dX}$ and $\mathit{dY}$ are the cell center displacements from the previous frame for each axis. Displacements toward the right (or downward) are indicated in blue, while displacements toward the left (or upward) are shown in yellow. $P(\text{Link}\text{multiplicity}=0)$ and $P(\text{Link}\text{multiplicity}>1)$ are the probabilities that the link multiplicity is zero (no linked cell in the other frame) and strictly greater than one (several linked cells in the other frame, i.e., in the case of a division), respectively. Note that only one frame pair ($t, t+1$) is represented, but the model inputs a larger temporal neighborhood and predicts these maps for more frame pairs. For a better readability, contours of segmented cells are represented in the eight rightmost images. (b) Model output for a bacterial data set frame pair. In this case, $P(\text{Link}\text{multiplicity}>1)$ is null, reflecting the absence of cell division or merging events in this data set. (c) Segmentation procedure: A watershed transform is applied to the EDM using regional maxima as seeds, which likely produces oversegmentation. A Gaussian function is applied to each pixel of the predicted GDCM and a watershed algorithm on the Laplacian transform is used to detect centers, which are used to reduce oversegmentation (see main text for details). (d) Tracking procedure: The centers of each cell at $t$ are shifted by the predicted displacement between $t$ and $t+1$ ($dX$ and $dY$). Each cell at $t$ is associated with the cell at $t+1$ in which the shifted center falls (the panel illustrates simple links; for more complex links, like division and fusion links, see Fig. S4 [24]).

Figure 2

Postprocessing uses temporal information on a large timescale to correct segmentation errors. (a) Diagrams presenting the procedure to correct wrong links. (b) Illustrative example of two distinct cells that are transiently detected as one object (undersegmented in frame $t=65$). Because objects involved in the merge link are sometimes seen separated, we can assume the object in frame $t=65$ should be split. (c) Illustrative example of one cell that is transiently detected as two objects (oversegmented in frame $t=79$). Because the cell is detected as a single object throughout the entire video (200 frames), except for one frame in which it is seen as two objects, we can assume that the two objects should be merged into one. (d) Illustrative example of a more complex error that implies more lineages. In frame $t=145$, one cell is undersegmented and one cell is oversegmented. Here again, temporal information at the scale of the entire video enables us to correct the segmentation errors.

Figure 3

Model architecture. The encoder is fed by successive frames (green, blue, and red rectangles) and produces encoded features (green, blue, and red cubes). Features are processed in pairs (corresponding to successive frames) by the pair blender module, which produces feature pairs. Encoded features and feature pairs are blended together by the blender module (see Fig. S7 [24] for details). The segmentation extractor generates three segmentation features corresponding to each frame for both EDM and GDCM, that are decoded by two distinct decoders to produce images of the same size as the input image. Likewise, the tracking extractor generates two tracking features corresponding to each frame pair for both displacement and link multiplicity, that are decoded by two distinct decoders. For simplicity only three frames have been represented but we considered seven frames in this work, and only one segmentation decoder and one tracking decoder are represented instead of two.

Figure 4

Temporal insights of the DiSTNet2D's performance, calculated with data set PhC-C2DH-PA14. (a) Influence of DNN time window in predictions. DNN time window was varied by changing the gap between considered frames ($\delta$) from 1 to 10, while keeping the number of considered frames constant ($N=7$). (b) Robustness to acquisition subsampling. We compared the three methods (trained with the complete training data set) on computationally generated time-subsampled versions of the evaluation data set. Note that for DiSTNet2D, $\delta$ was set to 1 for all points, which explains the difference in segmentation errors compared to Table 1, where $\delta =3$.

Figure 5

(a) Mean-squared displacement measured on bacterial cells at low surface fraction ($\varphi =0.466$, blue dots) and high surface fraction ($\varphi =0.719$, orange dots). Guide lines have exponent 1 (dashed lines) and 2 (solid line). (b) Cell speed as a function of cell length, using the same color code. Error bars are standard errors of the mean (error bars are hidden behind dots for the high surface fraction data set). Each dot is the mean speed of all cells binned by length, with a bin size of 0.088 $\mu \mathrm{m}$ (one pixel). Averaging is weighted by the duration of trajectories. The solid lines represent the number of cells in each bin, weighted by the duration of trajectories (a value of 1 is attributed to a cell tracked for 1000 frames). (c) Histogram of angles between the velocity vector and the major axis of the HBEC cell body (obtained by ellipse fitting), for five time intervals (0–15 h, 15–30 h, 30–45 h, 45–60 h, 60–75 h). Each interval includes $N=40939$, 53 284, 76 305, 115 860, and 138 749 data points (respectively). No chirality is measured in the data. Inset: Average cell density for each time interval. (d) Norm of the velocity vector with respect to the angle between the velocity vector and the major axis of the HBEC cell body, for the same five time intervals.