Stochastic characterization and reconstruction of material microstructures for establishment of process-structure-property linkage using the deep generative model

In material design, microstructure characterization and reconstruction are indispensable for understanding the role of a structure in a process-structure-property relation. The significant contribution of this paper is to introduce a methodology for the characterization and generation of material microstructures using deep generative networks as the first step in the establishment of a process-structure-property linkage for forward or inverse material design. Our approach can be divided into two parts: (i) characterization of material microstructures by a vector quantized variational auto-encoder, and (ii) determination of the correlation between the extracted microstructure characterizations and the given conditions, such as processing parameters and/or material properties, by a pixel convolutional neural network. As an example, we tested our framework in the generation of low-carbon-steel microstructures from the given material processing. The results were in satisfactory agreement with the experimental observation qualitatively and quantitatively, demonstrating the potential of applying the proposed method to forward or inverse material design. One of the advantages of the proposed methodology lies in the capability to capture the stochastic nature behind the microstructure generation. As a result, this methodology enables us to build a process-structure-property linkage while quantifying uncertainties, which not only makes a prediction more robust but also shows a way toward enhancing our understanding of the stochastic competitive phenomena behind the generation of material microstructures.

Figure 1

Schematic of the procedure of the proposed framework. (Training 1) Characterization of microstructures by VQVAE [27]. As a result of VQVAE, we can extract index lists corresponding to input images, which are the input of PixelCNN in this framework. (Training 2) Determination of the correlation between extracted features and given conditions, such as process parameters and/or material properties, by PixelCNN [28, 29]. In its application to low-carbon-steel microstructures, the natural logarithm of the cooling rate was given to PixelCNN as an example of conditions $\mathbf{h}$. (Prediction) Microstructure generation using the trained networks. First, by using the trained PixelCNN, the index list is sampled from the probability distribution conditioned by the given parameter. Then, each index is replaced by the discrete latent vector in the codebook. Finally, the decoder can generate the corresponding microstructure from the set of latent vectors.

Figure 2

Schematic of architecture of VAE.

Figure 3

Schematic of pixel dependences assumed in PixelCNN. The central red pixel $x_i$ is dependent on the green pixels $x_1,x_2,\cdots ,x_{i-1}$.

Figure 4

Examples of original microstructure images. The size of the images is $1024\times 768\mathrm{pixel}$, and one pixel is $0.34\mu \mathrm{m}\times 0.34\mu \mathrm{m}$ in these microstructure images. The upper left, upper right, lower left, and lower right images show the steel microstructures cooled at $1.0,3.0,10.0,\mathrm{and}30.0{}^{\circ }\mathrm{C}/\mathrm{s}$, respectively. The total number of prepared images is 160, and they comprise 40 microstructure images for each of the four cooling rates, $1.0,3.0,10.0,\mathrm{and}30.0{}^{\circ }\mathrm{C}/\mathrm{s}$.

Figure 5

Schematic of procedure of creating the training dataset from the original microstructure images. $128\times 128\mathrm{pixel}$ patches are cropped and then converted to grayscale images.

Figure 6

Image reconstruction by VQVAE. (a) Microstructure images reconstructed by VQVAE. (b) Corresponding original microstructure images. This result shows that VQVAE can reproduce sharp and clear microstructure images.

Figure 7

Original microstructure images and generated microstructure images corresponding to the given cooling rates. Parts (a), (c), (e), and (g) are original microstructures for each of the four cooling rates $1.0,3.0,10.0,\mathrm{and}30.0{}^{\circ }\mathrm{C}/\mathrm{s}$, respectively. Parts (b), (d), (f), and (h) are generated microstructures for each of the four cooling rates $1.0,3.0,10.0,\mathrm{and}30.0{}^{\circ }\mathrm{C}/\mathrm{s}$, respectively. Each set shows nine microstructure images.

Figure 8

Microstructure interpolation. Parts (a)–(c) are generated microstructure images corresponding to the cooling rates $2.0,6.5,\mathrm{and}20.0{}^{\circ }\mathrm{C}/\mathrm{s}$, respectively, which were not included in the original dataset. Regarding microstructure morphology, trained VQVAE seems to capture the trend relative to the change in cooling rate. Each set shows four microstructure images.

Figure 9

Original microstructures observed in the experiment and microstructures predicted by our approach. Parts (a) and (b) are, respectively, original and predicted microstructures for cooling rate 3.0 ${}^{\circ }\mathrm{C}/\mathrm{s}$. We can see that the predicted microstructures have similar topology to the original microstructure images. This result validates the capability of the proposed method to predict the microstructures corresponding to interpolated conditions. Each set shows nine microstructure images.

Figure 10

Box plots of the ferrite volume fractions in (a) the microstructure patches cropped from the original images, and (b) the microstructure patches generated by the introduced methodology. The black lines and green triangles in the boxes denotes median and mean values of sets of images for each cooling rate, respectively. Larger variance corresponds to higher spatial variation of microstructures.

Figure 11

Original microstructure patches [(a) and (c)] and schematic figures for the calculation of the mean free path [(b) and (d)]. The example test line passes through four white grains in (b) and eight white grains in (d). The mean free path corresponds to the average of the segments of the test lines shown as the red lines in these figures.

Figure 12

Box plots of the average grain sizes (a) in each patch cropped from the original images, and (b) in each generated microstructure patch by the introduced methodology. The black lines and green triangles in the boxes denote median and mean values of sets of images for each cooling rate, respectively. The mean values are corresponding to the overall averages for each cooling rate. Note that one pixel is $0.34\mu \mathrm{m}\times 0.34\mu \mathrm{m}$ in the microstructure images.

Figure 13

Stochastic relationship between the cooling rate and the ferrite volume fraction in the microstructure patches predicted by the proposed methodology. The black lines and green triangles in the boxes denote median and mean values of sets of images for each cooling rate, respectively.

Figure 14

Stochastic relationship between the cooling rates and the average grain sizes in the microstructure patches predicted by the proposed methodology. The black lines and green triangles in the boxes denote median and mean values of sets of images for each cooling rate, respectively. Note that one pixel is $0.34\mu \mathrm{m}\times 0.34\mu \mathrm{m}$ in the microstructure images.

Figure 15

Plots of microstructure images in the latent space with the corresponding cooling rates, visualized by PCA [32]. (a) Plot of images embedded in the latent space. (b) Plot of latent vectors colored for cooling rates.

Figure 16

A plot of the vectors included in the codebook colored based on the coordinate along the first main axis (horizontal direction) in PCA.

Figure 17

Generated microstructure images and sampled index lists. Parts (a), (c), (e), and (g) are generated microstructures for each of the four cooling rates $1.0,3.0,10.0,\mathrm{and}30.0{}^{\circ }\mathrm{C}/\mathrm{s}$, respectively. Parts (b), (d), (f), and (h) are sampled index lists for each of the four cooling rates $1.0,3.0,10.0,\mathrm{and}30.0{}^{\circ }\mathrm{C}/\mathrm{s}$, respectively. We can see the correspondence between the generated microstructure images and sampled index lists. Each set shows nine microstructure images and index lists.