Spin Nematics, Valence-Bond Solids, and Spin Liquids in $\mathrm{SO}(N)$ Quantum Spin Models on the Triangular Lattice

We introduce a simple model of $\mathrm{SO}(N)$ spins with two-site interactions which is amenable to quantum Monte Carlo studies without a sign problem on nonbipartite lattices. We present numerical results for this model on the two-dimensional triangular lattice where we find evidence for a spin nematic at small $N$, a valence-bond solid at large $N$, and a quantum spin liquid at intermediate $N$. By the introduction of a sign-free four-site interaction, we uncover a rich phase diagram with evidence for both first-order and exotic continuous phase transitions.

Figure 1

Equal time structure factors for SN order [$S_{\mathrm{SN}}(\mathbf{k})$] and susceptibility for VBS order [${\chi }_{\mathrm{VBS}}(\mathbf{k})$] shown for $N=10$ and $N=14$, for the ${\widehat{H}}_J$ model Eq. (1) with $L=48$. The Bragg peaks for SN (VBS) weaken (sharpen) with increasing $N$. The illustration of the Brillouin zones shows the location of the ordering vectors of both order parameters. Quantitative finite-size scaling of these orders is shown in Fig. 2.

Figure 2

Crossing plots of the ratios $R_{\mathrm{SN}}$ and $R_{\mathrm{VBS}}$ as a function of the discrete variable $N$ for the ${\widehat{H}}_J$ model. It is seen that spin nematic order is present for $N\le 10$. VBS order, on the other hand, is present for $N>12$. $N=12$ appears to be on the verge of VBS order. Interestingly, $N=11$ has no SN or VBS order. In the text, we present evidence that this phase is a quantum spin liquid (QSL). The inset in the upper panel shows $R_{\mathrm{SN}}$ scales to zero at $N=11$, despite nonmonotonic behavior at intermediate $L$.

Figure 3

Phase diagram of ${\widehat{H}}_{JQ}$ [Eqs. (2) and (4)] for different values of $N$. The left panel shows the phase diagram for small $N$, where a first-order SN-VBS transition is found for $6\le N\le 9$ (see Fig. 4). As $N$ is increased, we find the first-order transition weakens. The right panel shows how an intermediate QSL phase emerges for $N=10$ and $N=11$. Transitions from the QSL to both SN and VBS phases are continuous on the large systems studied; see Fig. 5.

Figure 4

First-order SN-VBS transition in $H_{JQ}$ at $N=7$. The upper panel shows the VBS order parameter and the stiffness as a function for $Q/J$ for different $L$ indicating a direct SN-VBS transition. The lower panel shows MC histories (and histograms in the inset) at $Q/J=1.26$, providing clear evidence that the SN-VBS transition at $N=7$ is direct and first order.

Figure 5

Crossings of $R_{\mathrm{SN}}$ (above) and $R_{\mathrm{VBS}}$ (below) signaling the location of the onset of long-range SN and VBS orders at $N=10$ (left) and $N=11$ (right). At $N=10$, $R_{\mathrm{SN}}$ and $R_{\mathrm{VBS}}$ cross at close but significantly different couplings, $Q_c=0.100(5)$ and $Q_c=0.117(2)$, respectively. At $N=11$, $R_{\mathrm{SN}}$ appears to have crossed at $Q/J<0$ (we cannot study this region because of the sign problem), whereas $R_{\mathrm{VBS}}$ crosses at $Q_c=0.042(3)$. From the location of the crossings, for both $N=10$ and $N=11$, we can infer an intermediate phase which is neither SN nor VBS, as shown in Fig. 3. We present arguments that this phase is a QSL. No direct evidence for first-order behavior is found at either of the transitions, though a weakly first-order SN-QSL cannot be ruled out. The QSL-VBS transitions show good scaling behavior with unconventional critical exponents.