Band mixing in the quantum anomalous Hall regime of twisted semiconductor bilayers

Remarkable recent experiments have observed fractional quantum anomalous Hall (FQAH) effects at zero field and unusually high temperatures in twisted semiconductor bilayer $t$MoTe$_2$. Intriguing observations in these experiments such as the absence of integer Hall effects at twist angles where a fractional Hall effect is observed, do however remain unexplained. The experimental phase diagram as a function of twist angle remains to be established. By comprehensive numerical study, we show that band mixing has large qualitative and quantitative effects on the energetics of competing states and their energy gaps throughout the twist angle range $\theta\leq 4^\circ$. This lays the ground for the detailed realistic study of a rich variety of strongly correlated twisted semiconductor multilayers and an understanding of the phase diagram of these fascinating systems.

Due to the emergence of narrow flat topological bands, moiré superlattices have become an arena for an intricate interplay between topology and correlations.The hallmark example of topological phases is the quantum Hall effect [8,9], both integer and fractional, occurring in Landau levels which arise when two dimensional electrons are subject to strong magnetic fields [10].There has been numerous theoretical and experimental research into lattice analogues of the quantum Hall effect in the absence of external magnetic fields, known as the quantum anomalous Hall effect, as these might provide an experimentally feasible way to realize high temperature topological phases [11][12][13][14][15][16][17][18][19].Materials exhibiting fractional quantum anomalous Hall effect (FQAH) effect have often been called fractional Chern insulators (FCIs) in the literature [15,20].
The integer quantum anomalous Hall effect (QAH) has been already experimentally observed in different moiré systems [21][22][23].In addition, local incompressibility measurements [24] on twisted bilayer graphene point towards the existence of fractional quantum Hall states that survive down to small magnetic fields (∼ 5T), below which, topologically trivial charge density wave (CDW) states are found instead.
In a very recent exciting development, the first evidence of FQAH states has been observed in twisted MoTe 2 based on both thermodynamic and transport measurements [25][26][27][28].Twisted transition metal dichalcogenide bilayers host topological moiré bands with spin/valley contrasting Chern numbers [29].At small twist angles, strong exchange and correlation effects in topological narrow bands [30] drive spontaneous spin/valley ferromagnetism and FQAH states [31,32].
Motivated by the current state of art, we provide a detailed study of the many-body interacting problem of AA stacked twisted TMD homobilayers at both integer (n = 1) and fractional (n = 1/3 and n = 2/3) hole doping of the underlying moiré valence bands.Using unbiased exact diagonalization techniques (ED) that includes multiple bands, we uncover novel significant effects of band mixing on various topological and correlated phases, which also enables a new electronic phase.
For integer filling n = 1, we demonstrate robust QAH states over a wide range of twist angles.However, for realistic interaction strength at dielectric constant ϵ = 5, we find the interacting QAH energy gap as a function of twist angle to be strongly renormalized compared to the non-interacting case, highlighting ubiquitous correlation effects due to band mixing.Remarkably, we also find that strong interactions could drive spontaneous layer polarization (i.e.ferroelectricity) that is only enabled by multiband effects.
Moving to fractional fillings n = 1/3 and n = 2/3 where previous single-band projected ED studies [32][33][34][35][36] have predicted various correlated phases such as FQAH and CDW states, we further show that band mixing plays a crucial role in the stability of the aforementioned phases.In particular, band mixing weakens the FQAH state at n = 1/3 in favor of the competing CDW state, while at n = 2/3, FQAH states remain robustly present over a wide range of twist angles.We perform the entanglement spectrum study to characterize and distinguish FQAH and CDW states in finite-size systems.
Model.To model the K valence bands of twisted bilayer TMDs, we use a continuum description [29].The non-interacting Hamiltonian around a single valley (or spin due to spin-valley locking [37]) written in the basis of the two layers is given by arXiv:2309.16548v2 [cond-mat.mes-hall] 1 Apr 2024 Where V ± (r) captures the intralayer moiré potential and T (r) represents the interlayer tunneling.Fourier expanding both functions to the lowest harmonics and taking into account symmetry constraints restrict their form to take In the Hamiltonian H ↑ , the corners of the moiré Brillouin zone are chosen to be κ ± = 4π 3a M (− √ 3/2, ∓1/2) and the moiré recirpocal lattice vectors are given by where a M = a 0 /(2 sin(θ/2)) is the moiré lattice constant for twist angle θ and a 0 is the lattice constant of the monolayer.The Hamiltonian around the opposite valley H ↓ is related to H ↑ by a time-reversal transformation.
(3) We are concerned with the problem of interacting holes in the moiré valence bands of H ↑(↓) .The full momentum space many-body Hamiltonian reads where c † kασ (c kασ ) are creation (annihilation) operators of holes in a Bloch state |kασ⟩ where α and σ =↑, ↓ are band and spin indices respectively.
ϵ ασ (k) is the single-particle energies and are the two-body interaction matrix elements between the different Bloch states.Our two-body interaction is the dual-gated screened Coulomb interaction V (q) = 2πe 2 tanh d g |q|/(ϵ|q|) for dielectric constant ϵ which controls the interaction strength.Unless stated otherwise, we choose the distance d g from the sample to the gates to be d g = 5 nm.
We diagonalize the Hamiltonian (4) on different equalaspect-ratio clusters [38] keeping a finite number N b of valence bands.By the variational principle, our calculations provide an upper bound on the exact ground state energy of Eq. 4 that becomes tighter upon increasing N b .We utilize generic twisted boundary conditions,  [38] with periodic boundary conditions, (θ1, θ2) = (0, 0).The index κ labels the different momentum points.
parameterized by (θ 1 ∈ [0, 2π), θ 2 ∈ [0, 2π)) along the two axes of the cluster.Robust ferromagnetism across a broad range of filling factors n ≤ 1 has been observed experimentally [25,26] and in previous numerical studies [31,33].With this motivation, we perform all calculations in the full spin-valley sector.However, the presence of full spin/valley polarization in ground and low-lying excited states throughout the entire twist angle range studied in this work should not be taken for granted and requires further study.
Results.We begin our analysis by investigating integer hole filling n = 1 where previous studies have mainly used self-consistent Hartree-Fock methods [39,40].For weak interaction strength ϵ −1 = 0.1, we find the many-body ground state to be a QAH state for twist angles spanning from θ = 2.0 • to θ = 4.0 • .The interacting QAH state is smoothly connected to the non-interacting limit with holes completely filling the first valence band which has a Chern number |C| = 1.The many-body spectrum shown in interacting energy gap and can be either smaller or larger depending on the twist angle.
As the interaction strength is increased to the realistic value ϵ −1 = 0.2, we observe the appearance of a ferroelectric phase which exhibits strong layer polarization.The signs of ferroelectricity stem from the existence of twodegenerate many-body ground states (Fig. 1(d)) which are manifestation of spontaneously breaking C 2y symmetry and realize two layer-polarized states on either the top or the bottom layer.
The ferroelectric phase dominates over the QAH for smaller twist angles.In Fig. 1(b), we plot E gap = E 3 − E 2 as a function of twist angle for realistic interaction strength ϵ −1 = 0.2 and find that it is non-vanishing up until θ ≈ 2.8 • , after which, the system undergoes a transition to a QAH state.
Similar to [41].which focused on smaller twist angles, the emergence of layer polarization can be intuitvely understood from a real-space picture also in the regime that we consider.Here, the first three valence bands have a total Chern number C = 0 and admit a real-space description in terms of three Wannier orbitals which are maximally localized on the high symmetry positions, R M X , R X M and R M M forming a triangular lattice of three sublattices [40].R α β denotes atomic positions in the moiré unit cell where the α atom (metal M or chalcogen X) of the top layer is aligned with the β atom.The two Wannier orbitals centered on the R M X and R M X sites are mainly localized in the top and bottom layers respectively while the orbital at the R M M site carries equal weight in both layers.When the interaction strength is strong, it becomes energetically favorable for the holes to minimize repulsion by localizing on one of the two layer-polarized sublattices.
The emergence of ferroelectric phase strongly affects the phase space of QAH state.For realistic interaction strength ϵ −1 = 0.2, the QAH state now appears at θ > 2.8 • .Interestingly, its topological gap (within the fully polarized sector) increases monotonously (see Fig. 1(b)) with the twist angle, at least up to θ = 4 • .This contrasts sharply with the noninteracting case where the band gap ∆ 12 decreases in this angle range.For comparison, we also performed ED calculation using a different set of continuum model parameters in the literature [34], and found that at the QAH phase n = 1 only appears in a narrow twist angle range for ϵ −1 = 0.1, and is entirely absent for ϵ −1 = 0.2 [38].
Next, we consider fractional hole fillings n = 1/3 and n = 2/3.For n = 1/3, single-band exact diagonalization studies [33,35,36] have revealed an intriguing interplay between FQAH and CDW states as a function of twist angle while more robust FQAH states for a wider range of twist angles were found at n = 2/3.
To contrast the two fillings, we focus first on twist angle θ = 2.5 • and ϵ −1 = 0.2.As shown in Fig. 2(a) and 2(c), the many-body spectrum in both cases displays three quasi-degenerate ground states at three distinct total momentum sectors.In this cluster geometry, the ground state momentum sectors expected of FQAH and CDW states happen to be identical and, therefore, distinguishing between the two candidate phases requires further analysis.
In order to pin-point the underlying phase, we calculate the particle entanglement spectrum (PES) [42,43] of the quasi-degenerate states.The density matrix is de- denotes the three quasi-degenerate ground states.We perform a cut in the particle-space corresponding to minority particles in a single band, holes for n = 1/3 and electrons for n = 2/3.By tracing out N B particles and keeping N A ones, the entanglement spectrum consists of the eigenvalues {ξ i } of exp(−ρ A ) where ρ A = Tr N B ρ is the reduced density matrix.In order to trace out electrons from the first valence band and avoid additional entanglement structure from the filled higher bands, we calculate the entanglement spectrum { ξ i } obtained from projecting the density matrix ρ onto the first valence band.
As evident from Fig. 2(b) and 2(d), the PES exhibits a well-separated low-lying spectrum at both fillings.For n = 2/3, we find the number of low-lying states to be consistent with a Laughlin-like FQAH which satisfies a counting rule (1,3) of the number of admissible configurations (at most 1 particle in each 3 consecutive orbitals [44][45][46]).In contrast, the counting of the fewer low-lying states at n = 1/3 is consistent with a CDW state [47].
Moreover, we find the CDW state at n = 1/3 and the FQAH state at n = 2/3 to exist for a wide range of twist angles.In Fig. 3, we plot the energy gap E gap = E 4 − E 3 as a function of the twist angle θ for both ϵ −1 = 0.1 and ϵ −1 = 0.2.In addition to signatures of a weak FQAH state at n = 1/3 for θ < 2.0 • [38], we observe robust CDW and FQAH states at fillings n = 1/3 and n = 2/3 respectively for θ ≥ 2.0 • , using the realistic interaction strength ϵ −1 = 0.2.For weak interaction ϵ −1 = 0.1, we find the system at n = 2/3 to become metallic for θ ≥ 3.5 • .
We also study the stability of the discussed phases for various twisted boundary conditions.While, we find the phases at n = 1 and n = 1/3 to be insensitive to the choice of boundary conditions, we observe significant effects of twisting the boundary conditions at n = 2/3, see Fig. 4 where we fix θ = 3.6 o .Although FQAH states should be insensitive to changes in the boundary conditions in the thermodynamic limit, these states may exhibit a strong dependence in the small systems available to exact diagonalization.At zero flux (standard periodic boundary conditions) we find an apparent two-fold ground state degeneracy, seemingly at odds with FQAH expectations, see Fig. 4(a).Crucially, however, we notice that a three-fold set of states originating from the predicted FQAH momenta transform into each other under twisting boundary conditions while evolving essentially separately from the rest of the spectrum as shown in Fig. 4(b).Moreover, with anti-periodic boundary conditions in one direction (θ 2 = π), these three states are separated from the rest of the spectrum and their corresponding PES shows a well developed entanglement gap with the predicted FQAH counting below the gap as shown in Fig. 4(d).In fact, this non-trivial counting persists if we consider the PES resulting only the two low lying states at (θ 1 , θ 2 ) = (0, 0) as evident from Fig. 4(c).We take this as strong evidence for a n = 2/3 FQAH state in the large system limit.Discussion.Inspired by recent experiments we have shown that multiband effects are crucial for understanding integer and fractional QAH states as well as their competitors at n ≤ 1 in twisted semiconductor bilayer tMoTe 2 .
Our results have several important implications.First, it follows that the optimum twist angle for QAH states is filling dependent, i.e. there is no unique magic angle for all fillings.For instance, at twist angles θ ≲ 2.8 • , an integer QAH states is missing at n = 1 (Fig. 1(b)) while a fractional FAQH state at n = 2/3 prevails (Fig. 3(d)).Second, a new intriguing phase is enabled by band mixing: spontaneous layer polarized state at n = 1 (Fig. 1(b),(d)).Third, in addition to the particlehole symmetry breaking within a band, multiband effects provide a second key ingredient in understanding why twisted bilayer tMoTe 2 may exhibit the FAQH effect at n = 2/3 but not at n = 1/3 (Figs. 2 and 3).Fourth, the theoretical multiband effects uncovered here, and their relation to experimental results provide important means of distinguishing between the greatly varying available sets of model parameters obtained via first principles calculations.We expect that these effects of multiband mixing will carry over mutatis mutandis to related setups.This thus provides key input for future studies of a rich variety of strongly correlated twisted semiconductor multilayers.
Note added.During the preparation of this work, a related study of twisted bilayer MoTe 2 reported two-band ED results at n = 2/3 [48].Upon submission a related preprint appeared [49].

SUPPLEMENTAL MATERIAL EXACT DIAGONALIZATION
In this paper, we employ exact diagonalization techniques on finite clusters.Let a 1 and a 2 be two basis vectors for the moiré superlattice.The finite cluster is spanned by two vectors T 1 and T 2 given by where m 1 , n 1 , m 2 and n 2 are integers.Generic twisted boundary conditions are imposed by requiring T T1 ψ kασ (r) = e iθ1 ψ kασ (r) and T T2 ψ kασ (r) = e iθ2 ψ kασ (r) where T is the translation operator and ψ kασ (r) is a Bloch state of momentum k, band α and spin σ.
In momentum space, this amounts to a discrete set of momentum points given by k = (α+θ 1 /2π)g 1 +(β +θ 2 /2π)g 2 for integer α and β where g 1 and g 2 are the discretization vectors of the momentum grid, given by g

RESULTS USING REF [34] PARAMETERS
In this sections, we study the continuum model of tMoTe 2 using the parameters introduced in Ref [34].We focus at filling n = 1.As shown in Fig. S3, we find signs of QAH states only for weak interaction ϵ −1 = 0.1 at twist angles θ > 3.5 • while we find robust ferroelectricity for θ ≤ 3.5 • and for the realistic interaction ϵ −1 = 0.2 at all twist angles between θ = 3.0

FIG. 2 .
FIG. 2. ED results at twist angle θ = 2.5 • and ϵ −1 = 0.2.(a) and (c) show the many-body spectrum in the CDW and FQAH phases respectively while (b) and (d) show the PES and the projected (onto the first valence band) PES respectively obtained by keeping NA = 2 particles.There are 18 states and 42 states below the line in (b) and (d), consistent with a CDW and a FQAH state respectively.Calculations were done on the 12 site cluster [38] using periodic boundary conditions (θ1, θ2) = (0, 0) in (a)-(b) and anti-periodic boundary conditions, (θ1, θ2) = (π, π) in (c)-(d).The index κ labels the different momentum points.

FIG. 4 .
FIG.4.Effect of twisting the boundary conditions for θ = 3.6 • , n = 2/3 , N b = 2 and ϵ −1 = 0.2.(a) The many-body spectrum with periodic boundary conditions, (θ1, θ2) = (0, 0).(b) The evolution of the many-body spectrum upon twisting the boundary condition in one direction (θ2) and keeping the other direction unchanged (θ1 = 0).The colors blue, green and orange label the three different momentum sectors where FQAH are expected while the color red labels the rest of the sectors.(c) The projected particle entanglement spectrum at (θ1, θ2) = (0, 0) calculated from the two quasi-degenerate ground states (see (a)).(d) The projected particle entanglement spectrum at (θ1, θ2) = (0, π) calculated from the three quasi-degenerate ground states.NA = 2 particles are kept in (c) and (d).There are 42 states below the line in (c) and (d), consistent with a FQAH state.