Magnetic ordering phenomena of interacting quantum spin Hall models

The two-dimensional Hubbard model defined for topological band structures exhibiting a quantum spin Hall effect poses fundamental challenges in terms of phenomenological characterization and microscopic classification. In the limit of infinite coupling $U$ at half filling, the spin model Hamiltonians resulting from a strong-coupling expansion show various forms of magnetic ordering phenomena depending on the underlying spin-orbit coupling terms. We investigate the infinite-$U$ limit of the Kane-Mele-Hubbard model with $z$-axis intrinsic spin-orbit coupling as well as its generalization to a generically multidirectional spin-orbit term which has been claimed to account for the physical scenario in monolayer ${\text{Na}}_2{\text{IrO}}_3$. We find that the axial spin symmetry which is kept in the former but broken in the latter has a fundamental impact on the magnetic phase diagram as we vary the spin-orbit coupling strength. While the Kane-Mele spin model shows a continuous evolution from conventional honeycomb Néel to $XY$ antiferromagnetism which avoids the frustration imposed by the increased spin-orbit coupling, the multidirectional spin-orbit term induces a commensurate to incommensurate transition at intermediate coupling strength, and yields a complex spiral state with a 24 site unit cell in the limit of infinite spin-orbit coupling. From our findings, we conjecture that in the case of broken axial spin symmetry there is a large propensity for an additional phase at sufficiently large spin-orbit coupling and intermediate $U$.

Figure 1

Intrinsic spin-orbit terms with amplitude $\lambda$ according (a) to (1) for the Kane-Mele model and (b) to (2) for the SI model with multidirectional SOC amplitude $\tilde{\lambda }$.

Figure 2

(a) Phase diagram of the Kane-Mele-Hubbard model as obtained in Ref. 13. The transition from a topological insulator (TI) to $XY$-plane antiferromagnet (AFM) was derived within slave-rotor theory which underestimates $U_c$. (b) Mean-field phase diagram of the sodium-iridate Hubbard model as obtained in Ref. 33. The transition from TI to a valence bond solid (VBS) phase that links to AFM was derived within slave-spin theory which overestimates $U_c$. The phase diagram of (b) is qualitatively similar to (a) apart from the additional “QSH${}^{☆}$ phase.” At $\lambda =\tilde{\lambda }=0$ and not too large $U$ the semimetal (SM) phase of graphene is present. See main text for details.

Figure 3

Characteristic behavior of the flowing ($\Lambda$-dependent) susceptibility in a magnetically ordered phase. While the RG flow is smooth above some critical value ${\Lambda }_{\text{c}}\approx 0.45$, a numerically unstable regime is found below that value. This feature signals a magnetic instability which becomes a divergence in the thermodynamic limit. The specific case shown here represents the largest component of the susceptibility of ${\mathcal{H}}_{\text{SI}}$ at $J_{\tilde{\lambda }}=0.4J_1$ where the system favors antiferromagnetic order.

Figure 4

Magnetic susceptibilities at the critical scale $\Lambda ={\Lambda }_{\text{c}}$ for various values of $J_{\lambda }$ ($J_1=1$) in the Kane-Mele spin model, resolved for in-plane ($x,y$) and out-of-plane ($z$). Top row: ${\chi }^x\left(\mathbf{k}\right)={\chi }^y\left(\mathbf{k}\right)$ (left panel) and ${\chi }^z\left(\mathbf{k}\right)$ (right panel) for $J_{\lambda }=0.1$. Bottom row: ${\chi }^x\left(\mathbf{k}\right)={\chi }^y\left(\mathbf{k}\right)$ (left panel) and ${\chi }^z\left(\mathbf{k}\right)$ (right panel) for $J_{\lambda }=0.5$. The susceptibility weight along $z$ significantly decreases for large $J_{\lambda }$. For higher $J_{\lambda }$, the remainder $z$ susceptibility deviates from the Néel AFM structure.

Figure 5

Magnetic susceptibilities of the SI spin model for various values of $J_{\tilde{\lambda }}$ ($J_1=1$). All susceptibilities shown refer to a magnetic field in $z$ direction. The $x$ and $y$ components of the susceptibility are obtained by $k$-space rotations of ${120}^{\circ }$ in clockwise or counterclockwise direction, respectively. (See Sec. 6b for more details.) While the Néel peaks initially persist for finite $J_{\tilde{\lambda }}$, the peaks start to move due to the onset of incommensurability (Fig. 6). For large $J_{\tilde{\lambda }}$, new susceptibility peaks emerge which link to the change of unit cell structure of magnetic order.

Figure 6

Dependence of the ordering vector $Q_y$ on $J_{\tilde{\lambda }}$ in ${\mathcal{H}}_{\text{SI}}$. The inset illustrates the evolution of the ordering peaks in the Brillouin zone (thick hexagon: second Brillouin zone, thin hexagon: first Brillouin zone; see also Fig. 5). In the limit $J_{\tilde{\lambda }}\to \infty$, the system converges again towards a commensurate ordering vector.

Figure 7

The SI spin model at $J_{\tilde{\lambda }}\to \infty$: (a) Magnetic susceptibility displayed in the first Brillouin zone of the triangular sublattice. The two ordering peaks correspond to the peaks in Fig. 5 which emerge at $J_{\tilde{\lambda }}\gtrsim 5$ and $k_x=0$. In the limit $J_{\tilde{\lambda }}\to \infty$, these maxima reach the same height as the ones at ${\mathbf{Q}}_{\infty }=(\pm \frac{2\pi }{3},\pm \frac{2}{3}\frac{2\pi }{\sqrt{3}})$. (b) Mapping of the SI spin model at $J_{\tilde{\lambda }}\to \infty$ to the antiferromagnetic Heisenberg model on the triangular lattice: The lattice is divided into four sublattices denoted by $•$, $xy$, $xz$, and $yz$. As shown in Eq. (11) the transformation from ${\mathbf{S}}_i$ to ${\tilde{\mathbf{S}}}_i$ depends on the sublattice where $i$ resides. The exchange couplings follow the convention shown in Fig. 1.

Figure 8

Schematic phase diagram for the SI Hubbard model as conjectured from our strong-coupling results of a Néel to spiral transition at $J_{\tilde{\lambda }}=0.53$ (upper $x$ scale), corresponding to $\tilde{\lambda }/t\approx 0.73$ (lower $x$ scale). While we cannot ultimately assess the nature of a possible intermediate phase aside from TI and AFM for strong interaction and strong spin-orbit coupling (yellow phase), its existence is likely.