Spontaneous fractional Chern insulators in transition metal dichalcogenides Moire superlattices

Moir{\'e} superlattice realized in two-dimensional heterostructures offers an exciting platform to access strongly-correlated electronic states. In this work, we study transition metal dichalcogenides (TMD) Moir{\'e} superlattices with time-reversal-symmetry and nontrivial spin{/valley}-Chern numbers. Utilizing realistic material parameters and the method of exact diagonalization, we find that at a certain twisting angle and fractional filling, gapped fractional topological states, i.e., fractional Chern insulators, are naturally {stabilized} by simply introducing the Coulomb repulsion. In contrast to fractional quantum Hall systems, where the time-reversal symmetry has to be broken explicitly, these fractional states break the time-reversal symmetry spontaneously. {We show that the Chern number contrasting in the opposite valleys imposes a strong constraint on the nature of fractional Chern insulator and the associated low energy excitations.} We also propose to realize the non-abelian Moore-Read state in TMD Moir{\'e} superlattice sandwiched between nonlinear dielectric media.

Introduction.-When two layers of two-dimensional materials are placed atop of each other with slight misalignment, it creates a superlattice with periodicity much larger than the atomic lattice parameter. Because of the large lattice periodicity, one can fill or empty the entire band by electrode gating. This Moiré superlattice provides a tunable platform to control the electronic band structure [1, 2], and therefore enables access to a plethora of interesting quantum states. Because the band width in these systems can be tuned to be extremely narrow [2], these Moiré superlattices open up a new pathway to stabilize various strongly-correlated phases such as superconductivity and correlated insulators . Furthermore, such electronic band structure can also be topologically nontrivial, e.g., with a nonzero integer Chern number [28][29][30][31][32]. Combined with their strong coupling nature, such Moiré superlattices offer a promising route to realize the long-sought fractionalized topological order [33][34][35][36][37][38][39].
Encouraged by such exciting experimental progress, here we explore the feasibility of the second category in TMD Moiré superlattices. In particular, we focus on systems like MoTe 2 , which may host topologically nontrivial bands with non-zero spin/valley-Chern numbers [53]. In contrast to a partially filled Chern band [44][45][46][47][48][49][50][51], because these systems preserve the time-reversal symmetry, two types of fractional states are in principle allowed (a) time-reversal invariant fractional topological insulators [54] and (b) FCIs via spontaneously breaking the time-reversal symmetry. The key focus of this study is whether Coulomb repulsion could stabilize some of these fractional states in TMD Moiré superlattices.
In this work, we show that by simply increasing the Coulomb interaction strength in such TMD Moiré superlattices, the system undergoes a quantum phase transition that spontaneously breaks the time-reversal symmetry by polarizing electrons into one of the two valleys. Further increase of Coulomb interaction will trigger a second quantum phase transition, and stabilize a FCI at a fractional filling. For excitations, our numerical studies observe both (intravalley) fractional excitations from the fractional topological order and (intervalley) valley-wave excitations from the spontaneous symmetry breaking. We argue that the symmetry breaking state and low-energy excitations are constrained by the valley contrasting Chern number in TMD Moiré superlattice.
Model.-We consider twisted homobilayer TMD materials. For each single layer, the low energy electronic states reside at the valence band maxima at ± valleys. Contrary to bilayer graphene systems where the valley and spin degrees of freedom are both present, in TMD each valley in the top valence band has fixed spin orientation due to strong spin-orbit coupling and the broken inversion symmetry [55]. With a small twist angle between two layers, the + valley for the top and bottom layers are shifted to and in the Moiré Brillouin zone (MBZ) respectively, [ Fig. 1(b)]. For convenience we choose the rhombus-shaped MBZ and set the point = ( + )/2 as the origin. We employ the continuum model [2] in which the Moiré Hamiltonian for the + valley is: Here * is the effective mass. The form of Moiré potential, Δ , , , is dictated by the 3 crystalline symmetry and a combination of 2 rotation followed by switching the two layers, and can be parameterized by [53]: . Here = 0 / is the Moiré lattice constant for a small twisted angle and 0 is the lattice parameter of TMD. The Hamiltonian for the valley − can be obtained by the time-reversal symmetry − ( , ) = + (− , ) * . To be specific, we focus on twisted MoTe 2 homobilayer with typical parameters (ℏ 2 /2 * 2 0 , , , ) = (495 meV, 8 meV, −8.5 meV, −89.6 • ) [53].
The top valence band of a TMD single layer splits into multiple Moiré bands due to the Moiré potential. As shown in Fig.  1(c) and (d), when the twist angle is close to 0 = 1.38 • , the top Moiré band becomes nearly flat. The flatness of a band can be characterized by a ratio of the gap between the nearest bands to its band width. For the top Moiré band, the ratio can be as large as 13. When < 3.1 • , The top Moiré band is topological characterized by a valley/spin Chern number = ±1 due to the skyrmion lattice pseudo spin textures of the Moiré potential [53]. The Chern number for the opposite valley/spin is opposite as required by time-reversal symmetry. Thus, at the single-particle level, such TMD homobilayer realizes a quantum valley/spin Hall insulator.
We then introduce screened Coulomb interaction and project it to the nearly flat top Moiré band [56]: where = ± is the valley index, and , ( ) = , | , + is the form factor originated from the projection. Here ( ) = 4 tanh( )/ √ 3 is the dimensionless screened Coulomb potential with the separation between the electrode and Moiré superlattice, which is set to = 2 in the calculations.
is system area and cell is the number of the unit cells in the calculations. The coefficient of ( ) is chosen to make equal to the bare Coulomb potential between two particles separated by .
( ) is the annihilation operator for single particle state | , . We neglect the weak intervalley impurity scattering process associated with a large momentum transfer, and therefore the Hamiltonian also has a valley (1) symmetry. In this model, there are two competing symmetry breaking states: an intervalley coherent state that breaks the valley (1) symmetry and an valley/spin polarized state that breaks the time reversal symmetry. At half filling of the topmost band (account for valley degree of freedom), our Hartree-Fock analysis and exact-diagonalization results both suggest that a valley-polarized state is energetically favored, which spontaneously breaks the time-reversal symmetry and leads to a interaction-induced Chern insulator [57]. At fractional filling, in principle, two types of fractional topological states might emerge, a fractional Chern insulator or a fractional topological insulator [54,58], depending on whether  the time-reversal symmetry is spontaneously broken or preserved [59], and our exact diagonalization below show that the FCI is favored and stabilized in our system.
Valley polarized FCI.-We define the filling factor = 2 / , where is the electron density occupying the top Moiré band and is the electron density for the full filling of the two-fold degenerate top Moiré band. The factor 2 accounts for the valley degree of freedom. Using exact diagonalization, at = 1/3 we observe numerical evidence of spontaneous valley polarization and FCI in the strong interaction limit, as shown in Fig. 2(a). For 8 electrons in 4 × 6 unit cells (4 × 6 × 2 single-particle states including both valleys), the ground states are fully valley polarized with three nearly-degenerate ground states for each valley polarization, separated from the excited states by an energy gap of the order of 2 K. We calculated the many-body Chern number of each ground state [48], and the topological index is found to be 1/3, characterizing a 1/3 FCI phase. This conclusion is further supported by the total momentum for each ground state, which obeys the generalized Pauli exclusion rule [60].
The occupation number ( 1 , 2 ) of single particle states for each of the three many-body ground states are plotted in Fig. 2 ( 1 , 2 ) is uniformly distributed for different single particle states, consistent with the fact that the ground state is an incompressible liquid. The spectrum evolution under flux insertion along 2 direction is shown in Fig. 2(c). The excitation gap is maintained throughout the flux insertion process.
The topological nature of the ground states are further confirmed by our calculation of the particle entanglement spectrum (PES) [60]. To compute PES, we divide the particles into two collections of and = − particles and trace out particles to get the reduced density matrix .  The PES levels are obtained from the logarithm of eigenvalues of , and are labeled by the total momentum of the remaining particles, as shown in Fig. 2(d). There is a clear entanglement gap with 2730 levels below the gap for = 4, consistent with the counting of quasihole excitation in the = 1/3 FCI [60].
To exam the finite-size effect, we study the scaling of the many-body gap Δ with various system sizes. For a genuine FCI, Δ remain finite in the thermodynamic limit when both 1 is fixed at 3 and 2 increases from 4 to 8, but Δ increases when the system size changes from 3 × 8 to 4 × 6.
We then map out the phase diagram at = 1/3 filling as a function of the interaction strength which can be controlled by distance between the electrodes and the Moiré superlattice in experiments and the dielectric constant . The results are shown in Fig. 3(b). We find a valley non-polarized Fermi liquid at a small 1/ corresponding to a small , a Fermi liquid with valley polarization at an intermediate interaction and the FCI phase with valley polarization at strong interaction. Depending on the twisted angle , which controls the bandwidth, the valley non-polarized Fermi liquid can transit directly to FCI with valley polarization or through an intermediate Fermi liquid with valley polarization. The phase transition between the valley-polarized Fermi liquid and FCI can be described by Ginzburg-Landau theory with a Chern-Simons term [57]. The direct transition occurs near = 1.38 • where the single parti-cle Moiré band has the largest gap to bandwidth ratio [see Fig.  1(d)]. This is consistent with the quantum Hall systems with flat Landau levels, where the interaction stabilizes simultaneously the fractional quantum Hall state with spin polarization. Note that the FCI can be stabilized in a relatively broader region of the twisted angle here compared to that in magic angle twisted bilayer graphene [33][34][35][36][37][38][39], and the region of angle for FCI increases with interaction. For interaction above the dashed line in Fig. 3(b), our single-band approximation used in the numerical calculations breaks down and it requires to take other nearby bands into account.
Excitations.-Here we study the charge neutral excitations above the FCI ground states. As a consequence of the sponta- The lowest intravalley many-body excitation has lower energy than valley wave excitation for the parameters we used, i.e., the energy difference between the lowest fully-polarized excited state and the FCI state is = 0.167 meV < , see Fig. 4. Nevertheless, the valley wave excitation remains a stable excitation because the decay of the valley wave to the intravalley many-body excitations are forbidden. Intravalley many-body excitations has valley quantum number 0, while the valley wave has valley quantum number 2.
In quantum Hall ferromagnets, a pair of skyrmions has lower energy than the particle-hole bound state [61]. The system size limitation in the exact diagonalization does not allow us to study the valley skyrmion excitation in our numerical calculations. Here we use an effective Hamiltonian density for where and are given by the valley wave spectrum. The presence of the valley pseudospin anisotropy can be traced back to the opposite Chern number for the opposite valley. One cannot rotate n from one valley to opposite valley adiabatically without closing the energy gap, which implies the existence of anisotropy for n. The last term accounts for the Coulomb interaction ( − ), because a skyrmion is dressed with charge distribution ( ) = /8 , where ( ) is the Levi-Civita tensor with , being the space index and , , being the spin index. The skyrmion topological charge = ∫ 2 ( ) is quantized to an integer number. The easy axis anisotropy favors skyrmions with a small radius while the Coulomb repulsion favors skyrmions with a large radius. Their competition determines the skyrmion size [62].
FCI at = 2/5.-In twisted graphene Moiré superlattices, the Halperin (332) state is stabilized at = 2/5 due to the remaining SU(2) spin rotation symmetry in the valley polarized state [36]. In our TMD Moiré superlattices, the spin rotation symmetry is absent because of the spin-valley locking. The Chern number contrasting valley degree of freedom disfavors the (332) state. To demonstrate this explicitly, we calculate the energy spectrum, spectrum flow under flux insertion and entanglement spectrum at = 2/5, and the results are displayed in Fig. 5. The 5-fold degenerate ground states are valley polarized, and are consistent with the = 2/5 FCI state. In the fractional quantum Hall, the = 2/5 state belongs to the second hierarchical Jain state, and similarly one can assign the = 2/5 FCI as a second hierarchical FCI. Our results highlight the importance of symmetry in dictating the ground state and contrast the difference between the graphene and TMD Moire superlattices. Discussions.-We show that TMD Moiré superlattices can host fractional topological states via spontaneously breaking the time-reversal symmetry, using realistic parameters of TMD Moiré superlattices. Comparing with graphene, the spinvalley locking in TMD materials breaks the SU(2) spin rotation symmetry and eliminates the spin wave Goldstone modes, which could help stabilize the FCI states. The valley contrasting Chern number in TMD Moiré superlattices also dictates the symmetry breaking states, hence the nature of fractionalized topological states, and also the low energy excitations in the FCI. The gapped nature of these states can be detected by transport, optical measurements etc, and its topological nature can be accessed by Hall conductivity measurement. Due to the strong analogy between FCI and chiral spin liquids, it is plausible that Moiré superlattices may also help realize/stabilize exotic spin liquid phases [63][64][65]

I. Energy for valley polarized (VP) and intervalley coherent (IVC) state
Here we compare the energy for the valley polarized and intervalley coherent state. We consider half filling of the topmost band including the valley degree of freedom. The wave function for the VP state can be written as Here we choose = + valley to be fully occupied. Its energy Ψ VP | |Ψ VP can be decomposed into single particle contribution 0 = + ( ); Hartree contribution: Ha = (0) [ +,0 ( )] 2 and the Fock contribution: Fo = − , ( )| +, ( )| 2 . Here the summation of is over the whole Moiré Brillouin zone.
We note that the system Hamiltonian is invariant under the following gauge transformation We choose a gauge which fixes the valley pseudospin associated with |Ψ IVC in the direction, and the wave function of the intervalley coherent state can be written as The energy for the IVC is the sum of single particle contribution 0 = After we choose the form of |Ψ IVC by fixing a gauge, it is not guaranteed that +, ( ) −,− ( + ) is positive real for all and . Therefore the IVC always has higher energy than that of the VP state. This conclusion is further supported by more detailed Hartree-Fock calculations below.

II. Hartree-Fock calculations
We treat the interaction part of the Hamiltonian using Hatree-Fock mean-field theory [S66] in which the Hamiltonian can be written as The gap Δ , ( ) = † ( ) ( ) has to be written in terms of these new quasiparticles We now write the gap equation in the new basis, (S4) Here ( ) = † ( ) ( ) Also, filling in the new basis is given bȳ Eqs. S4 and S5 are then solved self-consistently for a fixed filling, until and mean field order parameter or all converges. But, we use a relatively relaxed condition for convergence, as Δ , ( ) can have multiple degenerate configurations, therefore, we use max(Δ − Δ +1 ) < tolerance limit, where Δ = Δ , ( ). In the numerical simulation, we observe that only diagonal element of Δ , ( ) are populated whereas the off-diagonal elements are zero, meaning that valley polarized state is the ground state. We define a valley polarization order parameter In Fig. S1, we plot dependence on the interaction ( ) for ( = 1) and fractional ( = 1/3) fillings using the parameters discussed in the main text and at = 1.38 • . In the case of filling ( = 1) shown in Fig. S1 (a), we observe that till around = 0.3 we have unpolarized metal, . . equal number of electron in both the valleys. For an intermediate interaction, 0.3 ≤ ≤ 0.8 , we observe partially polarized metal and from = 0.8 onward the system is fully polarized. Here, is the bandwidth of the non-interacting bands. The band in one valley is fully occupied and the system is a valley polarized insulator. This behavior is consistent with the results reported for twisted bilayer graphene in the Ref. [S27].
On the other hand, in the case of the fractional filling ( = 1/3) shown in Fig. S1 (b), we observe the system to be unpolarized metal till = 0.85 . In the 0.85 ≤ ≤ 1.4 regime, partially polarized metal is observed and finally above, = 1.4 , the system saturates into a completely polarized metal. Note that in this filling fraction, one can only partially fill the lower mean-field band. Hence, the system remains a metal in contrast to the case with = 1.

III. Possibility of the Halperin state
Here we ague that it is unfavorable to host the Halperin ( 1 , 2 , 1 ) states in TMD Moiré superlattice, because of the opposite Chern number in the opposite valleys. For a flat Chern band, a good starting point to understand the physics is the Landau level by neglecting the variation of the Berry curvature and band dispersion. The Chern bands in the ± valleys in TMD Moiré superlattice can be treated as Landau levels stabilized by opposite effective magnetic field, ± . The wave function for the Halperin ( 1 , 2 , 1 ) state is where ± = + and = √︁ ℏ / is the magnetic length. Introducing composite particle through the flux attachment [S67] where ( 1 − 2 ) is the angle between the vector 2 − 1 and the axis. ( ) is the composite particle wave function and ( ) is the electron wave function.

IV. Effective theory for the transition between the valley polarized Fermi liquid and fractional Chern insulator
If we take the Landau level point of view by assuming completely flat bands with uniform Berry curvature, we can write down Ginzburg-Landau free energy to describe the phase transition between the Fermi liquid with valley polarization and fractional Chern insulator (FCI), analogous to the case for the transition between the fractional topological insulator and superfluid [S68]. In this consideration, we can think that two valleys experience an opposite magnetic field = ∇ × . The Lagrangian can be written as Here is the dynamical gauge field with Chern-Simons term. is the external electromagnetic gauge fields. describes the FCI condensate and = † is valley polarization order parameter, and their spatial average value obey | | 2 + | | 2 = 0 with 0 the electron density. Here is an integer and we consider electron filling at 1/ . The term with > 0 describes the repulsion between the FCI condensate and valley polarization condensate. ( ) is the potential for the FCI condensate which depends on the electron interaction.
From this construction, it is clear why the Fermi liquid with valley polarization (VP) is favored over the the Fermi liquid with inter-valley coherent state (IVC). In VP, the effective magnetic field due to the valley contrasting Chern band cancels. In L , there is no gauge field, which is energetically favorable because of energy cost associated with the Meissner screening of if it is present. Whereas for the IVC, we need to condense IVC ∼ † + − . In this case, we will have coupling to the field: − 1 2 IVC |(− ∇ − − 2 ) IVC | 2 , which costs energy due to the Meissner effect. The transition from the Fermi liquid with VP to FCI can be understood as follows. For an intermediate interaction, the Fermi liquid with VP is favored. Due to the repulsion between the FCI and VP condensates, electrons condense into the channel and the Fermi liquid with VP is stabilized. When the interaction becomes strong, the FCI is favored by the ( ) term, and more electrons condense into the FCI state.