Deep neural networks for classifying complex features in diffraction images

Intense short-wavelength pulses from free-electron lasers and high-harmonic-generation sources enable diffractive imaging of individual nanosized objects with a single x-ray laser shot. The enormous data sets with up to several million diffraction patterns present a severe problem for data analysis because of the high dimensionality of imaging data. Feature recognition and selection is a crucial step to reduce the dimensionality. Usually, custom-made algorithms are developed at a considerable effort to approximate the particular features connected to an individual specimen, but because they face different experimental conditions, these approaches do not generalize well. On the other hand, deep neural networks are the principal instrument for today's revolution in automated image recognition, a development that has not been adapted to its full potential for data analysis in science. We recently published [Langbehn et al., Phys. Rev. Lett. 121, 255301 (2018)] the application of a deep neural network as a feature extractor for wide-angle diffraction images of helium nanodroplets. Here we present the setup, our modifications, and the training process of the deep neural network for diffraction image classification and its systematic bench marking. We find that deep neural networks significantly outperform previous attempts for sorting and classifying complex diffraction patterns and are a significant improvement for the much-needed assistance during postprocessing of large amounts of experimental coherent diffraction imaging data.

Figure 1

Panels (a) and (b) show capsule-shaped particles whose orientation and size differs. The scattering images are calculated using a multislice Fourier transform (MSFT) algorithm that simulates a wide-angle x-ray scattering experiment which includes 3D information about the particle [6, 20]. The two incoming beams (indicated by the arrow on the left-hand side) produce very different scattering images, yet the dominant feature, an elongated bent streak, is distinctly visible in both calculations. A handcrafted algorithm is typically not able to identify the similarity between the two scattering patterns and would classify these two images in two distinct classes, although they belong to the same capsule shape class. A deep neural network can learn these complicated similarities on its own when we provide a few manually selected diffraction patterns that contain this feature.

Figure 2

Characteristic examples for all the classes assigned to the 7264 images by a researcher, except for the empty class. The top row of every class shows a representative diffraction pattern and the bottom row in panels (b)–(d) shows a stylized drawing of the characteristic feature of this class. The bottom row in panel (a) shows an illustration of the name-giving particle shape for the spherical/oblate and prolate classes. The shapes are derived from the analysis of the data in Langbehn et al. [41], and they serve as a form of superordinate classes. They are mutually exclusive to each other, and all diffraction patterns are part of one of these two classes. Also, both superordinate classes have subclasses. For example, panel (b) shows the spherical/oblate subclasses round, elliptical, and double rings. While a diffraction pattern can be part of the round and the double rings classes, it cannot be part of the round and the elliptical classes. For the prolate superordinate class, we find analog subclass rules, although there is no exclusivity rule as it was with the round and elliptical class. Therefore, an image belonging to bent can also be in the streaks class. Furthermore, all spherical/oblate and prolate patterns cannot only be part of their respective subclass but can also be part of one or more of the classes in the nonexclusive other subclass categories, shown in panel (d). These classes describe general features within the image which are to some extent independent of the particle shape. We derived the superordinate classes from these general features. These complicated interclass relationships demonstrate the capabilities of a researcher to interconnect mostly distinctive appearing features into a consistent description and ultimately leading to a valid physical interpretation. A hand-crafted algorithm could not account for these relationships normally, but now these interconnections can serve as an additional evaluation metric for the neural network. Since there is no diffraction pattern which belongs to the spherical/oblate and prolate class simultaneously, we can check if the neural network mislabeled a diffraction pattern according to these rules. We can then interpret this as a reliable indicator for a failed generalization of the network. The physics behind these patterns are quite complicated as well, but for a rigorous interpretation and analysis of these patterns, please see Langbehn et al. [41].

Figure 3

Schematic visualization of a convolutional neural network. It shows the hierarchical structure of the network with the function hierarchy $z_1,\cdots ,z_n$ above each layer. Depicted as input is a diffraction image, which is getting expanded by nine trainable convolutional kernel into nine feature maps. Note that only one kernel, producing the last feature map, is shown. The output of the first layer is then passed through multiple convolutional layer; this is the feature extraction part of the neural network. Ultimately, a fully connected layer with a logistic function as activation function produces the predictions. Every layer consists of two stages, also indicated by the brackets underneath $z_1,\cdots ,z_n$. The first stage is an affine transformation and the second one is a nonlinear function, called an activation function. The operation that is used as affine transformation is then the name-giving component for the layer, e.g., a convolutional layer uses a convolution as affine transformation. The choice of the activation function is subject to empirical optimization with various choices possible. Section 3a1 describes the affine transformations in more detail and Sec. 3a2 covers the basics on activation functions.

Figure 4

Panels (a) to (d) showing the various stages of added noise to a standard scattering image. Panels (e) to (h) are the calculated correlation maps with the upper triangle of order $n=8$ and lower triangle from the full CCF calculation.

Figure 5

Showing the gradcam$++$ results for two distinct classes from the helium data set. Panel (a) shows five randomly selected images from the streak class and panel (b) shows five images from the bent class. We chose these classes due to their distinct and distinguishable characteristic shapes which can easily be identified using the contour maps provided by the gradcam$++$ algorithm. For each class, we plot the schematic from Fig. 2 also at the beginning of each row. gradcam$++$ contour levels are plotted as dashed lines and used as transparency value for the images from which we calculated them. This way regions with strong gradients are also brighter.

Figure 6

Schematic for a convolutional operation inside a convolutional layer in panel (a) and for a classic skip connection found in the ResNet architecture in panel (b). Panel (a) illustrates the local receptive fields and shared weights concept. The convolutional filter has size $3\times 3$ and stride 2 and is sliding over the input image of size $7\times 7$, which produces an output, called feature map, of size $3\times 3$. The stride is the distance the filter is moving in each step which is implied by the gray shading every two pixels in the input image. Using a local receptive field describes the inclusion of nearby pixels, and weight sharing means using the same filter weights for the whole input image. The calculation at the bottom is for the second entry in the feature map. (b) A classical skip connection is shown with two convolutional layers that approximate a sparse residue which gets added to the identity at the output.