Antiferromagnetic ground state with pair-checkerboard order in FeSe

A monolayer FeSe thin film grown on ${\mathrm{SrTiO}}_3$(001) (STO) shows the sign of $T_c>77\mathrm{K}$, which is higher than the $T_c$ record of 56 K for bulk FeAs-based superconductors. However, little is known about the magnetic ground state of FeSe, which should be closely related to its unusual superconductivity. Previous studies presume the collinear stripe antiferromagnetic (AFM) state as the ground state of FeSe, the same as that in FeAs superconductors. Here we find a magnetic order named the “pair-checkerboard AFM” as the magnetic ground state of tetragonal FeSe. The pair-checkerboard order results from the interplay between the nearest-, next-nearest, and unnegligible next-next-nearest neighbor magnetic exchange couplings of Fe atoms. The monolayer FeSe in pair-checkerboard order shows an unexpected insulating behavior with a Dirac-cone-like band structure related to the specific orbital order of the $d_{xz}$ and $d_{yz}$ characters of Fe atoms, which could explain the recently observed insulator-superconductor transition. The present results cast insights on the magnetic ordering in FeSe monolayer and its derived superconductors.

Figure 1

(a) Schematic top view of the pair-checkerboard AFM in an FeSe layer. Each spin of the Fe atom has only one neighbor spin aligned ferromagnetically while the other neighbor spins are all aligned antiferromagnetically. The rectangle enclosed by the dashed lines denotes the magnetic unit cell. The optimized lattice constants for the orthorhombic phase are shown in the figure. (b) The charge-density difference between the pair-checkerboard AFM and the nonmagnetic state. (c) The charge-density difference between the col-linear AFM and the nonmagnetic state. The isosurface depicted by the red and green colors represents the lost and gained charge density compared to the nonmagnetic state. Se atoms in the pair-checkerboard AFM order show orbital orders which are labeled Se1 (blue) and Se2 (yellow), respectively.

Figure 2

(a) Electronic band structure of bulk FeSe with the pair-checkerboard AFM. (b), (c) Electronic band structure of monolayer FeSe in the pair-checkerboard AFM order without or with spin-orbit coupling. The Dirac point appears in the bulk FeSe and monolayer FeSe (highlighted), while including SOC would open (highlighted) a 27 meV indirect band gap in monolayer FeSe. The Fermi level is denoted by the dashed line.

Figure 3

Projected orbital-resolved partial density of states of (a) the spin-majority part and spin-minority part of the Fe atom in bulk FeSe, (b) spin-majority part and spin-minority part of the Fe atom in monolayer FeSe, and (c) type-1 Se atom and type-2 Se atom [defined in (b)], respectively. The $d$ orbitalsof Fe atoms diminish to 0 at the Fermi level in monolayer FeSe, while the spin-minority parts of the $d_{xz}$ and $d_{yz}$ orbitalscross the Fermi level in bulk FeSe. The position of the Fermi level is denoted by the dashed lines.