Kernel methods for interpretable machine learning of order parameters

Machine learning is capable of discriminating phases of matter, and finding associated phase transitions, directly from large data sets of raw state configurations. In the context of condensed matter physics, most progress in the field of supervised learning has come from employing neural networks as classifiers. Although very powerful, such algorithms suffer from a lack of interpretability, which is usually desired in scientific applications in order to associate learned features with physical phenomena. In this paper, we explore support vector machines (SVMs), which are a class of supervised kernel methods that provide interpretable decision functions. We find that SVMs can learn the mathematical form of physical discriminators, such as order parameters and Hamiltonian constraints, for a set of two-dimensional spin models: the ferromagnetic Ising model, a conserved-order-parameter Ising model, and the Ising gauge theory. The ability of SVMs to provide interpretable classification highlights their potential for automating feature detection in both synthetic and experimental data sets for condensed matter and other many-body systems.

Figure 1

(a) Average test set accuracy of support vector machines with polynomial kernel $K(\mathbf{\sigma },{\mathbf{\sigma }}^{\prime })={(\mathbf{\sigma }·{\mathbf{\sigma }}^{\prime })}^k$ trained on Monte Carlo sampled configurations from the 2D Ising model. For each number of training samples, the accuracy is averaged over 100 independent training and test sets. (b) The SVM classifies samples according to $\mathrm{sgn}(d\left(\mathbf{\sigma }\right))$. The decision function $d\left(\mathbf{\sigma }\right)$ for the SVM with a quadratic polynomial kernel is evaluated by Monte Carlo sampling at different temperatures and compared to the squared magnetization per spin $m^2$. The arbitrary scale factor and offset in the SVM decision function are fixed by matching the decision function to $\langle m^2\rangle$ at $T=1.6$ and $T=2.9$.

Figure 2

The decision function for an SVM with quadratic polynomial kernel, Eq. (3). The averaged $C_{\mathit{x}}={\langle C_{\mathit{x}}^{\left(\mathit{a}\right)}\rangle }_{\mathit{a}}$ is displayed for different values of regularization $\gamma ={10}^6$ and $\gamma ={10}^{-6}$. Clearly, for large regularization ($\gamma ={10}^6$), the decision function is essentially the $m^2$ order parameter of the 2D Ising model.

Figure 3

(a) Average test set accuracy vs number of training samples for support vector machines with polynomial kernel $K(\mathbf{\sigma },{\mathbf{\sigma }}^{\prime })={(\mathbf{\sigma }·{\mathbf{\sigma }}^{\prime })}^k$ trained on Monte Carlo sampled configurations from the conserved-order-parameter Ising model at different temperatures. (b) The spatial dependence of the SVM decision function coefficients ${\langle C_{\mathit{x}}^{\left(\mathit{a}\right)}\rangle }_{\mathit{a}}$ learned by an SVM with quadratic kernel and regularization coefficient $\gamma =112.9$ for system size $L=30$ shows very good agreement with the analytical form (5) devised in Ref. [5].

Figure 4

(a) Comparing the decision function averaged over Monte Carlo samples at different temperatures learned by SVMs with quadratic kernel, trained on 30000 samples with $L=30$, to the analytical order parameter $S$ [Eq. (4)] assuming different critical temperatures $T_c^{\prime }$ and amount of regularization $\gamma$. (b) Dependence on regularization $\gamma$ of the test set accuracy (averaged over ten sets of 10000 samples) of quadratic SVMs trained on 30000 samples for $L=30$.

Figure 5

(a) Test set accuracy of SVMs with polynomial kernels of order $k$ in classifying ground states versus infinite temperature states of the 2D Ising gauge theory for system size $L=5$. Only in the feature space of fourth-order polynomials is the SVM algorithm performance better than a random algorithm. The performance approaches 100% with increasing number of training samples. (b) Test set accuracy for $k=4$ and different system sizes. For larger system sizes, more training samples are needed to learn the correct decision function.

Figure 6

Histogram of the coefficients $C_{\mathit{a}\mathit{b}\mathit{c}\mathit{d}}$ in the decision function learned by a SVM with fourth-order polynomial kernel in classifying ground states versus infinite temperature states of the 2D Ising gauge theory for $L=5$. There are 600 large coefficients (in absolute value), which correspond to the 25 plaquettes and the permutations of their spins.