Spontaneous ferromagnetism and finite surface energy gap in the topological insulator Bi$_2$Se$_3$ from surface Bi$_\mathrm{{Se}}$ antisite defects

We perform ab-initio calculations on Bi$_\mathrm{{Se}}$ antisite defects in the surface of Bi$_2$Se$_3$, finding strong low-energy defect resonances with a spontaneous ferromagnetism, fixed to an out-of-plane orientation due to an exceptional large magnetic anisotropy energy. For antisite defects in the surface layer, we find semi-itinerant ferromagnetism and strong hybridization with the Dirac surface state, generating a finite energy gap. For deeper lying defects, such hybridization is largely absent, the magnetic moments becomes more localized, and no energy gap is present.

Topological insulators (TI) [1,2] has been one of the most intensively studied areas in physics in the past decade, owing to their remarkable electronic properties: the bulk is insulating but the surfaces are metallic due to a gapless Dirac surface state (DSS) protected by time reversal symmetry (TRS).
Besides an interest in discovering new TIs, considerable effort has also been dedicated to opening up an energy gap in the DSS spectrum by breaking TRS, in order to enhance the electric control and also to achieve the quantum anomalous Hall effect (QAHE) [3][4][5]. A natural route to break TRS is to introduce an effective magnetic field perpendicular to the surface of the TI [6]. Most of the studies along this route have involved doping the TI with magnetic impurities [6][7][8][9][10][11][12][13][14][15], whose magnetic moments might couple ferromagnetically through RKKY [16,17], Van-Vleck [18], or other exchange [11,12,[19][20][21] mechanisms to produce the necessary out-of-plane magnetic field. A more recent development has been the realization of intrinsic magnetism in MnBi 2 Te 4 [22][23][24][25], where the Mn atoms order antiferromagnetically with an out-of-plane magnetic anisotropy.
However, for both magnetic impurities in TI and MnBi 2 Te 4 , there seem to exist significant complications when it comes to opening a gap in the DSS, with experiments so far reporting both the presence of an energy gap [9,[26][27][28][29] and finite density of states [30][31][32][33][34][35][36][37][38][39] at the Dirac point. Also, in the case of thin films, the hybridization between the two DSSs could be the reason for a finite energy gap [40][41][42], and not TRS breaking. For magnetic impurities, a two-fluid description has been proposed [43] to account for the contradicting results. Here the DSS spectrum is indeed gapped due to TRS breaking, but at the same time the non-magnetic part of the scattering potential produces localized impurity-induced resonances [44][45][46][47][48] filling up the gap [49].
In this work, we show that a surface energy gap is generated in the most common TI, Bi 2 Se 3 , from intrinsic Bi Se antisite defects, entirely without the need of foreign magnetic atoms. By performing extensive ab-initio calculations of antisite defects, we find defect-induced low-energy resonances, which spontaneously acquire a magnetic moment and thus gap the DSS. Antisite defects and their associated resonance states have already been observed experimentally using scanning tunneling microscopy (STM) [50][51][52][53], in both surface and subsurface layers, when growing Bi 2 Se 3 in a Bi-rich environment [54][55][56][57]. An additional benefit of Bi Se defects is that they behave as compensating p-type dopants, neutralizing the naturally occurring n-type Se vacancies and thus moving the Dirac point closer to the Fermi level [57,58].
In detail, we show how Bi Se antisite defects in the TI surface produce low-energy states, with a spontaneous magnetization which even increases for lower concentrations. We find a magnetic anisotropy energy favoring an out-of-plane magnetic orientation of individual antisite defects that is two orders of magnitude larger than for common magnetic dopants. Together with an appreciable ferromagnetic exchange coupling this guarantees an out-of-plane ferromagnetic alignment between different defects. For antisite defects in the surface layer, we find semi-itinerant ferromagnetism and defect states coupling strongly to the DSS, resulting in a sizable energy gap in the DSS. On the other hand, antisite defects in the first subsurface layer display more localized magnetism with no discernible hybridization with the DSSs and consequently no DSS energy gap. This also reveals that a significant hybridization is necessary between the DSS and the defect states for the magnetic moment to be able to produce an energy gap. Taken together, our results open up an entirely original and general pathway for designing magnetic and gapped TIs, by merely tuning the synthesizing conditions and thus completely avoiding the need for external magnetic impurity atoms.
Method.-We perform electronic structure calculations, based on density functional theory, as implemented in the Vienna Ab-initio Simulation Package (VASP [59]), on Bi 2 Se 3 -slabs containing six quintuple layers (Se 1 -Bi 1 -Se 2 -Bi 2 -Se 3 ) in order to capture the TI surface, while still maintaining bulk conditions within the slab. On the surface we create a supercell by repeating the conventional surface unit cell, n × n (n = 2, 3, 4), adding one defect per supercell, resulting in defect concentrations x ∼ 25, 11, and 6%. Below we mainly report re-arXiv:2002.03962v1 [cond-mat.mes-hall] 10 Feb 2020 sults for antisite defects Bi Se1,2 , i.e. Bi replacing either the surface Se 1 or subsurface Se 2 atom, see Fig. 1(a,b), but we also study Bi Se defects in deeper layers, including the bulk. We carry out the structural and electronic optimizations using a plane-wave basis set with kinetic energy cut-off 270 eV [60], together with Projector Augmented Wave (PAW) pseudopotentials. We use the GGA for the exchange-correlation functional [61] and DFT-D3 [62] to properly account for the van der Waals corrections. Furthermore, we use a Γ-centered k × k × 1 grid to sample the Brillouin zone, where for even (odd) n we use k × n = 8(9) and k × n = 4(3) for the electronic and structural optimizations, respectively. We also use a 30Å vacuum to isolate each periodic instance of the slab. In terms of convergence criteria, we use force and energy convergence thresholds of 10 −6 eV (corresponding to 3 × 10 −2 meV/Å) and 10 −7 eV, respectively. We perform all calculations in a fully relativistic manner, always including the effects of spin-orbit coupling, and also allow for a finite magnetization in all directions. Structural distortions.-We start by performing structural optimizations of the atomic positions for each defected TI surface. This both establish the equilibrium positions of the Bi Se defects and give a structural view on the impact of antisite defects. To quantify the latter, we track the atomic distortions caused by the Bi Se defects by comparing with an equivalently relaxed pristine TI slab. In Fig. 1(c,d) we display in black text the relative change of bond lengths (in %) in the neighborhood of the Bi Se1 defect, while blue text reports the equivalent change when ignoring spin-orbit coupling. As seen, the Bi Se1 defect creates large local perturbations of the lattice structure, with bond lengths changing as much as 9% for nearest neighbor bonds. This is by all accounts a large structural change, which we at least partly can attribute to the 40% larger atomic size of the Bi atom compared to Se. In comparison, the next-nearest neighbor bonds show almost negligible distortion. If we were to ignore the spin-orbit coupling in the structural optimization we find that both the nearest and next-nearest neighbor bond show a similar change. This illustrates that spin-orbit coupling is essential to capture not just the DSS but also the correct atomic structure of antisite defects in TIs. We find similar structural patterns for the other antisite defects, including defects in the bulk of the TI, see Supplementary Material (SM) [63].
Magnetism.-We next turn to the electronic properties of antisite defects. Surprisingly we find that Bi 2 Se 3 with antisite surface defects hosts a pronounced magnetization, despite the intrinsically non-magnetic nature of antisite defects. For both Bi Se1 and Bi Se2 , we observe a highly anisotropic, out-of-plane (c-direction), magnetization. Bi Se3 (Bi on the third Se layer) also gives rise to a net magnetization, but the defect also easily migrates to the van der Waals gap during structural optimization. For antisite defects further into the bulk we find no magnetization. Interestingly, if we start with atomic structures optimized without spin-orbit coupling, we find no net magnetization for antisite defects in any layer. Thus, the large structural distortions caused by spin-orbit coupling is crucial for correctly determining the electronic ground state of antisite defects.
In Fig. 2(a) we show how the net magnetization varies as a function of defect concentration for the Bi Se1 and Bi Se2 defects. We find that both spin and orbital moments increase with decreasing concentration: the surface Bi Se1 defect shows an almost three-fold increase in the spin magnetic moment when decreasing the defect concentration from 25% to 6%, while the subsurface Bi Se2 defect shows a minor increase. The increasing, and persistent, magnetization with decreasing defect concentrations assures that the magnetization is stable in the dilute defect limit. In Fig. 2(a) we also see that the orbital moments are large, suggestive of a highly anisotropic magnetization [64,65]. In order to confirm this, we calculate magnetic anisotropy energy (MAE), i.e. the total energy difference between out-of-plane and in-plane magnetizations. Figure 2(b) shows how also the MAE increases significantly with decreasing defect concentration. Notably, the MAE is almost 20 (12) meV for the Bi Se 1 (2) antisite defect at the lowest concentration. Such MAE values are impressive, about two orders of magnitude larger than what has been achieved in TIs with the magnetic transition metal impurities Cr, V or Mn, where the MAE is only of the order of 0.1 meV [66]. We also directly calculate the exchange coupling as the energy difference between ferromagnetic and anti-ferromagnetic c-axis alignments of two Bi Se1 defects at a distance of ∼ 13Å. We find the ferromagnetic configuration to be lower in energy by 3.2 meV. This is comparable to the interactions between magnetic impurities on TIs [54,67,68]. The large MAE and exchange coupling values give an exceptionally strong preference for an out-of-plane ferromagnetic alignment of the antisite magnetic moments, thus creating optimal conditions for also opening a gap in the DSS [6,69,70].
To further understand the antisite-induced magnetic state, we analyze its spatial properties in the 6% Bi Se1,2 systems. In Fig. 2(c) we resolve the spin magnetic moments into layers of thickness h equal to the Bi-Se bond-length projected onto the c-axis, see black box in Fig. 2(d). We find that magnetism is only present in the surface layers, with a peak in the layer of the defect. For the in-plane behavior we show in Fig. 2(d) the real-space magnetization density of the surface atoms for the Bi Se1 defect. We find that the magnetization is semi-itinerant, extending with a three-fold spatial pattern from the defect to distances well beyond the primary unit-cell. To quantify the itinerancy, we study how the magnetization accumulates with distance away from the antisite defect. For this we plot in Fig. 2(e) the net magnetization within a rhombus with the same shape as the unit-cell and with thickness h and centered around the antisite defect with varying side length d. For Bi Se1 the magnetization continuously increases with d, demonstrating semi-itinerancy. However, for the Bi Se2 defect we find that the magnetization is localized since a plateau develops when d equals about half the lattice parameter of the unit-cell in the a-b plane. This difference in the spatial characteristics of the magnetization for the two antisite defects is in agreement with the concentration dependence in Fig. 2(a).
Surface energy gap.-Having established finite magnetism from intrinsically non-magnetic Bi Se defects, we turn to investigating the electronic spectrum in detail. Since we find an exceptionally strong MAE, effectively guaranteeing an out-of-plane magnetic moment, we are particularly interested in how the magnetization affects the topologically protected DSSs in Bi 2 Se 3 . In Fig. 3(a) we plot the band dispersion along the Γ-K direction for the Bi Se1 defect system at 6% concentration (blue) and compare it with the equivalent but defect-free system (red). To be able to effectively compare the two systems, we first set the Fermi level of the pristine slab to 0 eV at the Dirac point. We then align the spectrum of the defected slab such that the valence (VB) and conduction (CB) bands perfectly align for the defect and defect-free systems, see SM [63]. This is possible since both systems reach bulk conditions in the interior of the slabs. We refrain from plotting all bands belonging to the bulk but instead conceptually show their extent in the dark pink regions. We see a clear bulk band gap ranging from -0.05 eV to 0.27 eV (light pink), in agreement with previous predictions [71]. We also identify an intrinsic doping produced by the antisite defect, as the Fermi level (dotted lines) of the antisite defect system falls at a slightly higher energy (61 meV with 6% defects).
We next focus on the in-gap region, where we expect to find the DSS, but also defect states generated by the antisite defects. Initially we are interested only in the intrinsic DSSs, and therefore first exclude all bands belonging to the antisite defects. We can do this easily beyond the Γ point, as there the DSS and the antisite defect states have very different orbital and spatial characters: states belonging to the DSS is present throughout the surface, while the antisite defect states are heavily localized at the defect. At the Γ point we find finite hybridization between the DSS and some defect states, but, nonetheless, we can still remove the defect states based on their orbital weights and flat energy dispersion (due to their localization), see SM [63]. In this way we extract and plot only the DSSs for the Bi Se1 defect in Fig. 3(a). The DSS in the pristine system (red) and antisite system (blue) are very similar at higher energies, both showing a linear Dirac spectrum with the same slope. However, at low energies we find a clear 24 meV energy gap induced in the antisite system. The gap size at the Γ point is fully consistent with the slope of the DSS at higher energies. In Fig. 3(d) we plot the equivalent bands for the Bi Se2 defect, but here the DSS energy gap is negligible, despite the finite magnetization. This is another property, along with the spatial extent of the magnetization, where we observe contrasting behavior for Bi Se1 and Bi Se2 defects.
Density of states.-To gain further insight into the magnetization and DSS energy gap we investigate the density of states (DOS). In Fig. 3(b,e) we compare the DOS in the bulk (red) with that of the surface quintuple layer of the Bi Se1 and Bi Se2 antisite systems (blue), respectively. By comparing these two DOS, we find that the DOS predominantly belonging to the Bi Se 1(2) antisite defect occupy an energy window ranging from around -20 (0) meV to 120 (220) meV, with respect to the pristine Dirac point, thus filling a large part of the bulk energy gap for both types of defects. However, we also observe that the defect states co-exist (in energy) with the induced energy gap for the Bi Se1 defect, while for the Bi Se2 defect, the defect states are mainly around 60 meV above the Dirac point of that defected system. By additionally studying the orbital character of all low-energy bands near the Γ-point, we find that the non-dispersive Bi Se1 states overlapping in energy with the Dirac point strongly hybridizes with the DSS, see SM [63]. This hybridization explains why the Bi Se1 defect both generates a semi-itinerant magnetization and opens an energy gap in the DSS by effectively acting as a TRS breaking perturbation on the DSS. The strong hybridization between the magnetic Bi Se1 defect states and the DSS also provides the necessary pathway for a strong exchange coupling to align the antisite magnetic moments [72]. Here, with Bi Se being an inherent defect, it naturally has the same (s,p) orbital character as the DSS and thus also has a clear advantage over transition metal atoms with their d orbital character in generating a strong exchange coupling [66].
The semi-itinerant magnetism and finite energy gap for the Bi Se1 system should be contrasted with the behavior of the Bi Se2 system. While the Bi Se2 defect states have a finite magnetization, they have a negligible overlap in energy with the DSS around its Dirac point. As a consequence, they do not effectively couple to the DSS and thus the magnetization stay localized and the energy gap in the DSS remains vanishingly small. Thus we conclude that a mere presence of an out-of-plane magnetic defect moments does not guarantee the opening up of an energy gap in TIs, but that an effective coupling between the magnetic defect states and the DSS needs to be present as well.
The creation of in-gap defect-induced resonance states for strong potential defects has previously been established for generic 2D Dirac materials [48], including the DSS in TIs [45] and also in the presence of finite magnetic moments [49]. Our ab-initio results establish that naturally occurring surface antisite Bi Se defects act as such strong potential scatterers, inducing in-gap resonance states. This then also implies a so-called two-fluid behavior [43], with both the dispersive DSS and the localized impurity resonance states filling the TI bulk energy gap, as is clearly visible in Fig. 3.
Finally, we compare the magnetization density of states (MDOS) between the bulk and surface in Fig. 3(c,f) for the Bi Se1,2 defects. We find that the magnetization in the system is almost exclusively associated with the in-gap defect states. This also finally offers an explanation as to why the antisite defect states spontaneously become magnetized in the first place: the defect resonance states generate a large DOS at the Fermi level ρ(E F ), which necessarily becomes prone to spontaneous magnetization. In its simplest incarnation the instability towards magnetism is given by a Stoner-like criterion ρ(E F )U > 1, where U is the electron-electron interaction strength [73]. Our ab-initio results on antisite Bi Se defects on the surface of the TI Bi 2 Se 3 show that antisite defects are indeed strong enough potential scatterers to generate these low-energy defect-induced resonances, which then thanks to finite exchange interactions in the TI also become spontaneously magnetized.
Conclusions.-Our fully relativistic ab-initio calculations show that intrinsic antisite Bi Se defects in the surface layer of the TI Bi 2 Se 3 generates a finite energy gap in the topologically protected DSSs. The antisite defect produces low-energy resonance states, which spontaneously become magnetic with an exceptionally large MAE guaranteeing an out-of-plane magnetic moment. With the defect states also overlapping in energy with the Dirac point, they hybridize with the DSS and thus the surface antisite defect acts as an effective magnetic field opening an energy gap in the DSS. For antisite defects buried in the first subsurface layer we also find a finite magnetization, but the overlap with the DSS is negligible and thus no measurable energy gap is present in the DSS. These results illustrates both that naturally occurring defects can produce a magnetic TI and that magnetic defect moments require effective coupling to the DSS to open an energy gap. Moreover, the results provide an important observation on the site dependence of defects in exhibiting essential physics of TIs.

Structural distortions for Bi Se 2 in the surface and bulk
In the main text, we show that spin-orbit coupling greatly influences the local atomic structure of the Bi Se 1 defect by displaying the relative change in the bond lengths close to the antisite defect, both in the presence (black) and absence (blue) of spin-orbit coupling. Here in fig. 1 we show that the same behavior for the Bi Se 2 defect, both when it is in the surface as well as in the bulk. As seen, there are similar structural distortions locally around antisite defects for all layers. Notably, including spin-orbit coupling is very important in determining the correct atomic structure. In order to draw conclusions about the band structure of the antisite Bi 2 Se 3 we need to properly align it with that of the pristine Bi 2 Se 3 . In fig. 2 we plot the full band structure of both the pristine (red) and antisite (blue) slab following the alignment procedure described in the main text. For the defected system we plot the bands belonging to the bulk valence and conduction bands with dotted lines, while the states (defect and DSS) within the gap are shown with solid lines. We can easily make this distinction by noting their spatial extent within the slab. Clearly, we see that both the conduction and valence bands of both systems overlap essentially perfectly. This helps us to identify both the DSS and the defect states in the gap region. Extracting the DSS and energy gap for Bi Se 1 defects In order to extract the DSS in the Bi Se 1 slab, we study the orbital weights for each low-energy eigenstate. For the purpose of displaying the data we sum the orbital weights for all atoms up to the next-neighbors (within a distance of 5Å) of the Bi Se 1 defect. Then we use a circle at each energy eigenvalue, whose diameter is set by the inverse of this summed orbital weight, to display how much that particular eigenstate is delocalized over the whole surface, i.e., the larger the circle, the less localized defect character in that eigenstate. Doing this for all in-gap states, we can then connect those eigenvalues that have the largest circles to give the DSS as that is the state that is fully delocalized over the surface. The remaining bands are then identified as localized defect states. In fig. 3 we show the result, including a zoom-in around the Γ-point. We clearly observe the lower and upper branch of the DSS as they have significantly larger circles than the remaining states. They also show the characteristic linear dispersion as expected for a DSS. Note that away We follow the same procedure as in the previous section also for the Bi Se 2 antisite defect slab and show the results in fig. 4. While there are large overall similarities with the Bi Se 1 system, we here find that the defect states are spread to higher energies. In particular, near the Γ-point, we find that the DSS do not hybridize with the defect states, simply because there are no defect states there. As a consequence, we also find no energy gap in the DSS. Far away from the Γ-point we