Gate-Tunable Antiferromagnetic Chern Insulator in Twisted Bilayer Transition Metal Dichalcogenides

A series of recent experimental works on twisted ${\mathrm{MoTe}}_2$ homobilayers have unveiled an abundance of exotic states in this system. Valley-polarized quantum anomalous Hall states have been identified at hole doping of $\nu =-1$, and the fractional quantum anomalous Hall effect is observed at $\nu =-2/3$ and $\nu =-3/5$. In this Letter, we investigate the electronic properties of AA-stacked twisted bilayer ${\mathrm{MoTe}}_2$ at $\nu =-2$ by $k$-space Hartree-Fock calculations. We identify a series of phases, among which a noteworthy phase is the antiferromagnetic Chern insulator, stabilized by an external electric field. We attribute the existence of this Chern insulator to an antiferromagnetic instability at a topological phase transition between the quantum spin hall phase and a band insulator phase. Our research proposes the potential of realizing a Chern insulator beyond $\nu =-1$, and contributes fresh perspectives on the interplay between band topology and electron-electron correlations in moiré superlattices.

Figure 1

(a) Real-space model for ${\mathrm{tMoTe}}_2$ at 3.89°. Orange (green) circles represent MX (XM) stackings. (b) The moiré band structure with an out-of-plane electric field of ${\mathrm{\Delta }}_D=5\mathrm{meV}$. The orange (green) arrows indicate the states are mainly from the bottom (top) layer, and the up (down) arrows indicate spin up (down) states. (c) Phase diagram of $k$-space Hartree Fock calculation vs displacement field ${\mathrm{\Delta }}_D$ and dielectric constant $\epsilon$ at twist angle 3.89°. In the calculations, an $18\times 18$ mesh in the reciprocal space is used and 196 bands are included. Inset left (right) is the illustration of the in-plane (out-of-plane) spin texture of the AFMxy (AFMz) phase. (d) The quasiparticle band structure of the AFMz Chern insulator phase at ${\mathrm{\Delta }}_D=32\mathrm{meV}$ and $\epsilon =33$ as denoted as the blue star in (c). (e) DMRG calculation of the Kane-Mele-Hubbard model with $U=13t_1$. The calculation is performed with bond dimensions 9600, 12 000, and 15 000. The inset shows a linear fit of order parameter over the two-site DMRG truncation errors [38] for the three bond dimensions with ${\mathrm{\Delta }}_D=10.2t_1$. Most fits yield invisible error bars except for ${\mathrm{\Delta }}_D=9.6t_1$ and $9.8t_1$ (with cross labels) with an error comparable to the values. For these two, the 15 000 bond dimension data are plotted instead. The challenges may be a consequence of nearby weak first-order transition or quantum criticality. (f) Sketch of the gap opening process of the quasiparticle band structure from the nonmagnetic phases at the QSH-BI phase boundary (black lines) to the AFMz phase (red lines). Spin orientation of the bands is denoted by black arrows.

Figure 2

(a) The noninteracting band structure at twist angle 6° without out-of-plane electric field. (b) The $k$-space Hartree-Fock phase diagram at twist angle 6°. In the calculations, an $18\times 18$ mesh in the reciprocal space is used and 196 bands are included. The AFMxy and AFMz phase is still defined by the magnetic moments at MX and XM stackings, although charge density also appears at the MM stacking, where the Mo atoms sit on top of the Mo atoms.

Figure 3

The phase diagram of the continuum model at $\epsilon =20$.