Topological orders competing for the Dirac surface state in FeSeTe surfaces

FeSeTe has recently emerged as a leading candidate material for the two-dimensional topological superconductivity (TSC). Two reasons for the excitement are the high $T_c$ of the system and the fact that the Majorana zero modes (MZMs) inside the vortex cores live on the exposed surface rather than at the interface of a heterostructure as in the proximitized topological insulators. However, the recent scanning tunneling spectroscopy data have shown that, contrary to the theoretical expectation, the MZM does not exist inside every vortex core. Hence there are ``full'' vortices with MZMs and ``empty'' vortices without MZMs. Moreover the fraction of ``empty'' vortices increase with an increase in the magnetic field. We propose the possibility of two distinct gapped states competing for the topological surface states in FeSeTe: the TSC and half quantum anomalous Hall (hQAH). The latter is promoted by magnetic field through the alignment of magnetic impurities such as Fe interstitials. When hQAH takes over the topological surface state, the surface will become transparent to scanning tunneling microscopy and the nature of the vortex in such region will appear identical to what is expected of the vortices in the bulk, i.e., empty. Unmistakable signature of the proposed mechanism for empty vortices will be the existance of chiral Majorana modes(CMM) at the domain wall between a hQAH region and a TSC region. Such CMM should be observable by observing local density of states along a line connecting an empty vortex to a nearby full vortex.

Intensive experimental investigations of FeSeTe confirmed the existence of Dirac surface state in the normal state above T c [21]. The predicted evidence for the MZM in the vortex core of superconducting state was the zero-bias peak in scanning tunneling microscopy (STM). Indeed, the STM is a particularly suitable probe for the MZM in this material as it would exist at the surface [11,12]. Despite several observations of a zero-bias peak in cores of some vortices [22][23][24][25], an apparent contradiction to the prediction has also been observed in the increasing fraction of "empty" vortices without a zero-bias peak upon the increase in magnetic field [26,27]. A careful study [26] revealed that the "empty" vortices cannot be accounted for by a simple picture of pair-wise annihilation of MZM between two near-by vortices. Although Ref. [28] showed that a model allowing for long-range interaction among MZM's far separated can in principle explain the "empty" vortices, an alternative explanation with simpler starting point and a falsifiable prediction is desirable.
Here we provide an alternative interpretation of the observed "empty" vortices based on the role of the magnetic field on aligning local moments of Fe-interstitials. Our main physical picture is summarized in Fig. 1a-d. As it is known from the study of magnetic dopants added to TI surface states, the exchange field from magnetic impurities also gap the TI surface state to form the half quantum anomalous Hall (hQAH) state with the half-integer quantization of Hall conductivity [2,[29][30][31] (Fig. 1a -1b) Uneven distribution of interstitials can nucleate the hQAH regions on the surface of FeSeTe when the moments get aligned with magnetic fields (Fig. 1c), preventing TSC to form in that very region. Such hQAH surface state will reveal the bulk superconductivity to STM and the vortices penetrating hQAH surface will show properties of the bulk superconducting state with topologically trivial the s± pairing [32,33], i.e., becoming "empty". With increasing magnetic fields, more hQAH regions are nucleated on the surface of FeSeTe, thus providing a natural explanation of the increasing faction of empty vortices observed in experiments. Interestingly, it has been known that a boundary between hQAH and TSC should host a chiral Majorana mode (CMM) [4,[34][35][36]. Hence our key prediction is that the MZM that would have been in the vortex core transforms into the CMM located at the boundary between the hQAH and TSC on the surface of FeTeSe (Fig. 1d). In the rest of this Letter, we first present our proposal using a low energy effective theory and then support it with a numerical simulation on a microscopic model. Exchange field and low energy effective theory -Consider the low energy effective theory for the topological Dirac surface state in FeSeTe. As noted by Jiang et al. [37], the interstitial Fe atoms can provide magnetic impurities in Fe(Te 0.55 Se 0.45 ). Although the impurity moments will point in random direction at zero-field (Fig  1a), the external field applied to create vortices would align the impurity moments (Fig 1b). In the regions with higher concentration of aligned impurity moments, the exchange field generated by these moments would couple to the topological surface state as in magnetically doped TI [38][39][40][41][42][43]. Such exchange coupling can be captured by H ex (r) = −J 0 i S i · sδ(r − r i ), where s = 2 σ is the surface state electron spin, S i and r i are the spin and location, respectively, of the Fe interstitial and J 0 is the coupling constant. This exchange field will be heterogeneous depending on the distribution of the interstitials. We consider the mean field approximation for the exchange field, leading to the form H ex (r) = −I ex (r) · σ, where I ex (r) = J0 2 i S i δ(r − r i ) local is a smoothly varying field with ... local representing the average over a small region for Fe moments. In an ordinary topological insulator, such heterogeneous exchange field should result in hQAH effect with spatially varying gaps for the Dirac surface state [44]. However, non-topological bands crossing the Fermi Surface will mask hQAH states in the normal state of Fe(Te 0.55 Se 0.45 ).
Once the system develops superconductivity, the hQAH and TSC can compete as the two possible ways of gapping the Dirac surface state. Moreover, the hQAH region will reveal itself by leaving the bulk superconductivity bare when the exchange gap dominates over the superconducting gap. This can be captured by the BdG Hamiltonian for the Dirac surface state with both exchange field and the s-wave pairing in the basis where σ i and τ i are the Pauli matrices in the spin space and particle-hole space, respectively. Here we assumed an s-wave gap to be real and only consider exchange field along the z direction. It is straightforward to find upon increase in the exchange term, the superconducting gap for BdG quasiparticles closes at the critical exchange field strength of [45][46][47] When |I ex | < |I ex,c |, the TSC dominates to support the vortex core MZM, which can be explicitly obtained by choosing ∆τ x → |∆|(τ x cos θ − τ y sin θ) substitution (θ is the azimuthal angle), which places a superconducting vortex at the origin. The zero mode we obtain for |I ex | < |µ| [3], (3) where J l is the l-th Bessel function of the first type, reduces the Fu-Kane vortex zero mode by setting first I ex = 0 and then µ = 0 [3]. It can also be generalized to |I ex | > |µ| using J l (ix) = i n I l (x) for x ∈ R, where I l is the l-th modified Bessel function of the first type, provided, however, that |∆(r → ∞)| > I 2 ex − µ 2 , i.e. |I ex | < |I ex,c |, as can be seen from the asymptotic forms for the large real arguments, 2πx. On the other hand, when |I ex | > |I ex,c |, hQAH dominates without the vortex core MZM. The domain wall CMM can be demonstrated by setting µ = 0 with the domain wall at y = 0 arising from I ex (y) = I 0 Θ(y) and ∆ = ∆ 0 Θ(−y) will be considered, i.e.
with the eigenenergy E kx = vk x .
Microscopic model -Next we will support our results by the numerical simulations on the bulk model of Fe-SeTe system. For FeSeTe bulk system, the topological phase is attributed to the band inversion between two states with opposite parities at Z point. Taking |S + , + 1 2 , |S + , − 1 2 , |P − , 1 2 and |P − , − 1 2 as the basis at Z point, the topological electronic structure can be described by the Hamiltonian in a 3D lattice H T I = k ψ † k H T I (k)ψ k and Hamiltonian matrix reads where Here η are Pauli matrices in the orbital space. The mass term at Γ and Z points are We extend the Hamiltonian to include superconductivity and exchange field from impurities, the BdG Hamil- and the Hamiltonian matrix reads, where ∆ s is the intra-orbital spin singlet pairing. In the absence of exchange field, the (001) surface states will be gapped by superconductivity and form an effective p + ip pairing, where Majorana modes can be trapped in a vortex core of the surface (as described by Eq. 1 with I ex = 0). We then study the effect of exchange field on the (001) surface states by adopting the above Hamiltonian with open boundary condition along z direction.
The microscopic model reproduces the topological phase transition of the low energy effective theory. Fig. 2 demonstrates the existence of the topological phase transition of the surface states by fixing the pairing potential and increasing the exchange field strength. In Fig. 2a, with zero exchange field, the surface state is gapped by superconducting pairing. When the exchange field strength reaches the critical strength which is equal to the superconducting gap for µ = 0, the gap of the surface states closes (Fig. 2b), consistent with the condition of Eq. 2. With further increasing exchange field, the surface state gap reopens and the system is driven into the hQAH state (Fig. 2c).
Next we turn to how exchange field affects the vortex core MZM in topological surface state superconductivity. We introduce a vortex located at the center of the system by setting ∆ s (r) = |∆ s (r)|e iθ and adopt the Hamiltonian with open boundary conditions along the x, y, z directions. A lattice size of 17 × 17 × 16 is chosen for the following numerical calculations. The exchange field is only restricted to the top (001) surface of the system. With the above sample configuration, Figs. 3a and b show the distribution of the zero-energy local density of states on the bottom and top surfaces, respectively, for the exchange field exceeding the critical strength defined by Eq. 2. One can see that an "empty vortex" appears on the top surface in Fig. 3b, in sharp contrast to the "full vortex" on the bottom surface where there is no exchange field in Fig 3a. At the core of a full vortex, there is a well-defined MZM with zero-bias peak in the local density of states. On the other hand, the MZM is absent at the core of an empty vortex. Despite of the absence of MZM in the vortex core, the edges of the top surface under exchange field show a large amount of density of states that depict the presence of edge CMM. In Supplementary Materials, we study the profile evolution of zero-energy local density of states on the top surface with increasing magnetic fields, from which one find that the localized MZM gradually extends outside of the vortex and becomes localized on the the edges of (001) surface.
Experimental prediction -Based on our results that have been well established by both the effective theory and microscopic bulk model, a natural prediction is the existence of the domain wall CMM between an empty vortex and a full vortex. Consider an experimental setup shown in Fig. 1c, in which two vortices are located at the TSC and hQAH regions, respectively. Fig.4a displays the spatial profiles of zero-energy states in the vicinity of a full vortex (left) and an empty vortex (right) and Fig.4b shows the progrssion of the local density of states (LDOS) as a tip marches from a full vortex to an empty vortex. Experimentally, one can implement an STM measurement of LDOS along the line connecting a full vortex (indicating the TSC state) and an empty vortex (indicating the hQAH state). As shown in Fig. 4b, a zero-bias peak is expected to exist in the intermediate region without any vortex and can be attributed to the existence of the CMM at the domain wall between the hQAH and TSC regions. This chiral Majorana mode always possessing a zero-energy state is distinct from a normal chiral mode and the energy spectrum is related to the circumference of the region with Zeeman field (see SM). With a large thermal smearing in STM measurements, the LDOS at the domain wall exhibits a broad peak around zero energy. While the external magnetic field cannot gap out the CMM, changing its magnitude will shift the location of the CMM as the hQAH region expands while the TSC region contracts or vice versa.
Conclusion-To summarize, we proposed a new mechanism by which magnetic field can increase the fraction of "empty" vortices without MZM in Fe(Te 0.55 Se 0.45 ). Our mechanism is purely local, i.e. a vortex is "empty" because of its intersecting the surface inside the hQAH domain rather than the long-range MZM interaction effects. We postulate that these hQAH domains arise from the alignment of local moments associated with Fe inter-stitial which produces heterogeneous exchange fields exceeding the superconducting gap in isolated puddles. It has been known that there should be the CMM localized at the domain wall between regions with dominant superconducting gap and regions with dominant exchange gap. Through an explicit calculation on a minimalistic lattice model of topological bands, we showed that MZM in the vortex core of topological superconductor transforms into the domain wall CMM upon increase in the exchange field on the region supporting the vortex.
Our proposal is distinct from an earlier proposal in Ref. [28] that relies on pair-wise extinction of MZM's through tunneling between vortices. In our proposal, the MZM relocates and extends to the domain wall CMM instead of disappearing. A clear signature of the proposed mechanism will be the existence of the domain wall CMM between an "empty" vortex and a "full" vortex which can be detected through STM measurements along a line connecting an "empty" vortex to a nearby "full" vortex. Given the clear distinction between the domain wall and the vortex core as shown in Fig. 4b, our proposal suggests that the CMM detection in Fe(Te 0.55 Se 0.45 ) through the STM measurement may be relatively easy compared to the recent transport experiments [48,49]. Another prediction that should be easy to check is that we anticipate the "full" vortices and "empty" vortices to segregate as their segregation will represent the regions dominated by TS or by hQAH.

Evolution of Majorana modes on top (001) surface
We include a Zeeman field on the (001) surface to investigate its effect the on Majorana states. With increasing magnetic field, a topological phase transition on (001) surface states will occur, as shown in Fig.3 in the main text. If the magnetic field is large enough (larger than ∆ 2 0 + µ 2 ), the (001) surface becomes topologically trivial. As the other sides surface states are topologically nontrivial, chiral Majorana modes should occur.
where the basis is Ψ p = (c † p↑ , c † p↓ , c −p↓ , −c −p↑ ). Here τ and σ are Pauli matrices in Nambu and spin space and the gap function ∆(r, θ) = ∆ 0 f (r)e inθ = ∆ 0 tanh r ξ0 e inθ (n is the vorticity of the vortex). With the above basis, the time reversal operation is T = −iσ y K and the particle-hole operation C = τ y σ y K. The above Hamiltonian satisfies: T H s (p)T −1 = H s (−p) and CH s (p)C −1 = −H s (−p). In the real space, we use the substitution p x,y → −i∂ x,y and we have the following equations, As there is a rotational symmetry, the angular momentum is conserved and we can express the above BdG equations as a set of 1D radial equations separated into angular momentum modes. In the following we consider the vortex with n = 1 and assume the trial wavefunction has the following form, With the above trial wavefunction, the eigen equation is H s (r, θ, ∂ r , ∂ θ )Ψ(r, θ) = EΨ(r, θ) and the matrix form reads, where P − = −ie −iθ ∂ r − e −iθ r ∂ θ and P + = −ie iθ ∂ r + e iθ r ∂ θ . From the above eigenvalue equation, we can get, Now the radial equations can be further written as,  Here we notice that Hamiltonian matrix is not symmetric. For a Majorana state, its antiparticle is itself and the corresponding wavefunction should satisfy CΨ ∝ Ψ, which leads to ν = 0. We define ρ = r ξ0 = r v F /∆0 and follow the above definition by setting = 1 and we further have d h √ r . Therefore, the eigenfunction can be further written as,  When discretizing a Dirac equation on a lattice one encounters the problem of fermion doubling. One standard approach is to use a forward-backward difference scheme for approximating the partial derivatives in the above equations [50][51][52], with h being the discretization step. Here we use the same differential form for u σ and v σ to preserve the particle-hole symmetry. With discretization on 1D radial geometry with radius R, the above equation can be written as,