Robustness of the cluster expansion: Assessing the roles of relaxation and numerical error

Cluster expansion (CE) is effective in modeling the stability of metallic alloys, but sometimes cluster expansions fail. Failures are often attributed to atomic relaxation in the DFT-calculated data, but there is no metric for quantifying the degree of relaxation. Additionally, numerical errors can also be responsible for slow CE convergence. We studied over one hundred different Hamiltonians and identified a heuristic, based on a normalized mean-squared displacement of atomic positions in a crystal, to determine if the effects of relaxation in CE data are too severe to build a reliable CE model. Using this heuristic, CE practitioners can determine a priori whether or not an alloy system can be reliably expanded in the cluster basis. We also examined the error distributions of the fitting data. We find no clear relationship between the type of error distribution and CE prediction ability, but there are clear correlations between CE formalism reliability, model complexity, and the number of significant terms in the model. Our results show that the size of the errors is much more important than their distribution.

Figure 1

Symmetry-allowed distortions for two different unit cells. The atomic positions of the cell on the left do not have any symmetry-allowed degrees of freedom, but the aspect ratio of the unit cell is allowed to change. For the unit cell on the right, the horizontal positions of the atoms in the middle layer may change without destroying the symmetry. (The unit cell aspect ratio may also change.)

Figure 2

Relaxation scheme. The top images show the original unrelaxed configurations, while the bottom figures show the relaxed configuration. The left images (a) shows the relaxation where the hexagon is contracted as shown by the black arrows in the bottom left figure. The relaxation in the right images (b) is restricted to displacement of the blue atoms as shown by the black arrows in the bottom right figure.

Figure 3

Cluster expansion fits for Ni-Cu alloy using DFT or EAM potential. Each bar represents the average percent error and error bar (standard deviations) for 100 independent CE fits. The blue bars represent the unrelaxed CE fits, while the red bars represent the relaxed CE fits. The colored number represents the average number of coefficients used in the CE models. When the configurations are relaxed, we find that the CE fits are often worse (higher prediction error and higher number of $J\mathrm{s}$) than the unrelaxed system. However, we show that in one case (Ni-Cu EAM) the unrelaxed and relaxed CE fits are identical (same error and same number of coefficients) and this is due to a small relaxation.

Figure 4

CE fitting and relaxation of Ni-Cu alloys (DFT calculations) using FCC derivative superstructures and BCC derivative superstructures. Shown in Figs. 4 and 4 are the 100 CE fits and the histogram of the clusters used for the FCC lattices, while plots 4 and 4 are for the BCC lattice. The errors and coefficients are shown in 4 and 4 for the FCC structures and in 4 and 4 for the BCC lattice. The plot shows that the number of clusters used in fitting is small when cluster expansion fitting is good (error is on average 6.03% for FCC derivative structures). However, the CE fitting of the BCC parent lattice is worse at 16.70% compared to FCC at 6.03%. More coefficients are used when CE fails. The increased number of $J\mathrm{s}$ and error indicate a bad CE fitting model as shown by plots 4 and 4. Figure 4 shows only a few significant terms with many other clusters used sparingly in the fits.

Figure 5

Plot 5 displays the CE fitting error vs the number of coefficients, while plot 5 highlights the relationship between number of coefficients and relaxation. The dashed line approximates what we consider as the maximum acceptable error for a CE model (10%). The dashed line in Fig. 5 marks the estimated threshold for acceptable relaxation level. Each symbol represents 100 independent CE fittings for each Hamiltonian. Higher error correlates with a higher number of coefficients.

Figure 6

Relationships between relaxation and CE fitting reveal a heuristic for determining the quality of a CE model. This graph shows the CE fitting error vs normalized mean square displacement (NMSD). Each mark represents 100 individual CE fittings for each system (potentials and parameters). As the NMSD (relaxation) increases, the CE fitting error increases for various systems and potentials. Using the relaxation metric, the quality/ reliability of the CE fits can be divided into three regions: good, maybe, and bad CE model. The solid black lines indicates these three areas.

Figure 7

Distributions from real relaxations using classical and semiclassical potentials, as well as DFT calculations. The distributions are all normalized to fall within 0 and 1. The widths $\mathrm{\Delta }$ were calculated by taking the difference between the $25\mathrm{th}$ and $75\mathrm{th}$ percentiles.

Figure 8

The analytic, equal width distributions used for adding error to the toy model CE fit.

Figure 9

Comparison of the predictive error in CE fits as the shape of relaxation error changes. (A) refers to the analytic distribution while (S) refers to simulated distribution. The fits are ordered from lowest to highest distribution width. Fits were averaged over 100 randomly selected subsets with 500/2000 data points used for training; the remaining 1500 were used to verify the model's predictions. The black and red colored symbols represent 2% and 15% error levels, respectively. The circles and triangles represent the analytical and simulated distributions, respectively. Higher error produces higher prediction errors.

Figure 10

Prediction error over 65% of the structures for the “toy” cluster expansion (at 15% error added). The systems are ordered by $\mathrm{\Xi }$, which is the total number of unique clusters used by any of the 100 CE fits for the system. This ordering shows a definite trend with increasing $\mathrm{\Xi }$.

Figure 11

Prediction error over 65% of the structures for the “toy” CE model (at 15% error added). The errors are ordered by $\mathrm{\Lambda }$, the number of significant terms in the expansion. As expected, the values are close to the known model complexity (5 terms) and the ordering once more appears random.

Figure 12

Plot of predictive error over 65% of the structures for the “toy” problem (at 15% error added). The systems are ordered by $\notin$ the number of clusters that were used less than 25 times across all 100 CE fits. These are considered exceptions to the overall fit for the system. As for Fig. 10, there is a definite trend toward higher error for systems with more exceptional clusters.

Figure 13

For a reliable CE model, the number of coefficients converges as a function of the training set size. A total of 500 structure were available for training. The number of coefficients in a fit and its error bars give us an indication of the predictive power of CE with only a small training set. The black curve represents a good CE fit; only 25 to 50 (or 5 to 10%) of training structures were needed. On the other hand, the red and blue curves show that CE fails to fit the data due to a slowly converging expansion. The error bars on the blue points indicate extremely bad fitting.