Chiral Spin Density Wave Order on the Frustrated Honeycomb and Bilayer Triangle Lattice Hubbard Model at Half-Filling

We study the Hubbard model on the frustrated honeycomb lattice with nearest-neighbor hopping $t_1$ and second nearest-neighbor hopping $t_2$, which is isomorphic to the bilayer triangle lattice, using the SU(2)-invariant slave boson theory. We show that the Coulomb interaction $U$ induces antiferromagnetic (AF) chiral spin density wave ($\chi \mathrm{SDW}$) order in a wide range of $\kappa =t_2/t_1$ where both the two-sublattice AF order at small $\kappa$ and the decoupled three-sublattice 120° order at large $\kappa$ are strongly frustrated, leading to three distinct phases with different anomalous Hall responses. We find a continuous transition from a $\chi \mathrm{SDW}$ semimetal with the anomalous Hall effect to a topological chiral Chern insulator exhibiting the quantum anomalous Hall effect, followed by a discontinuous transition to a $\chi \mathrm{SDW}$ insulator with a zero total Chern number but an anomalous ac Hall effect. The $\chi \mathrm{SDW}$ is likely a generic phase of strongly correlated and highly frustrated hexagonal lattice electrons.

Figure 1

(a) The isomorphic $t_1-t_2$ honeycomb lattice and bilayer triangle lattice. (b),(c) Spin configurations of the four-site, triple-$Q$, tetrahedron AF $\chi \mathrm{SDW}$ order.

Figure 2

Phase diagram: shaded regions represent distinct $\chi \mathrm{SDW}$ phases: Semimetal with the AHE, $C=2$ Chern insulator with the QAHE, and $C=0$ Chern insulator with the ac AHE. All phase boundaries are lines of discontinuous transitions except for the continuous transition across the blue line.

Figure 3

Band dispersion (top panel, in unit of $t_1$) and FS (bottom panel) at $t_2/t_1=0.5$ (a), $t_2/t_1=0.85$ (b), and $t_2/t_1=1.0$ (c). Red hexagons in the bottom panel mark the original zone boundary, on which lie six vH points connected by the wave vectors ${\overrightarrow{Q}}_{1,2,3}$ shown in (a). The outer FS crosses the vH points in (b) where the blue hexagon indicates the $2\times 2$ reduced zone boundary that intersects the inner hole FS.

Figure 4

Evolution of spin chirality and anomalous Hall response as a function of $U/t_1$ at $t_2/t_1=0.85$. The calculated dc AHE response corresponds to the green line, while the $C=2$ QAHE is marked by the superposed red line. The hysteretic spin chirality (blue line) is shown at the transition between the $C=2$ and $C=0$ Chern insulators. Inset: ac AHE response at $U/t_1=8.0$.

Figure 5

Band structure (in unit of $t_1$) endowed with corresponding Chern numbers by $\chi \mathrm{SDW}$. (a) $C=2$ Chern insulator, $U/t_1=7.15$. (b) $C=0$ Chern insulator, $U/t_1=8.0$.