Mott Physics and Topological Phase Transition in Correlated Dirac Fermions

We investigate the interplay between the strong correlation and the spin-orbit coupling in the Kane-Mele-Hubbard model and obtain the qualitative phase diagram via the variational cluster approach. We identify, through an increase of the Hubbard $U$, the transition from the topological band insulator to either the spin liquid phase or the easy-plane antiferromagnetic insulating phase, depending on the strength of the spin-orbit coupling. A nontrivial evolution of the bulk bands in the topological quantum phase transition is also demonstrated.

Figure 1

(a) 6-site cluster tiling (dashed lines) on honeycomb lattice used for the calculations of bulk properties. A and B denote the two inequivalent sites, ${\mathbf{a}}_1$ and ${\mathbf{a}}_2$ the lattice unit vectors. (b) First Brillouin zone. (c) An illustration of tiling the ribbon used for the calculations of edge states. The superlattices (rectangle with solid lines ) are arranged periodically along the $x$ direction. For illustration, we only plot two clusters (separated by the dotted horizontal line) in each superlattice, while in the calculations eight clusters are included.

Figure 2

Qualitative phase diagram of KMH model. SM, TBI, SL and AF insulator denote the semimetal, topological band insulator, spin liquid and antiferromagnetic insulator, respectively. Above the dashedline in the AF insulator phase, the $z$ term of the AF order disappears.

Figure 3

(a) $\Omega$ as a function of $h_{\mathrm{AF}}$ for various values of $U$ at $\lambda =0$. (b) The density of states for $U=4$ and $\lambda =0$. (c) and (d): $\Omega$ vs $h_{\mathrm{AF}}$ at $\lambda =0.2t$ along the $z$ and $x$-directions, respectively.

Figure 4

Intensity plot of $A(\mathbf{k},\omega )$ for single-particle excitations in the bulk [Figs. (a), (c), (e) and (g)] and in the ribbon with the zigzag edges [Figs. (b), (d), (f) and (h)] at $\lambda =0.1t$. The white dashed curves in Figs. (c), (e) and (g) are the mean-field fits discussed in the text. From the up to down figures, $U=0$, $2t$, $3t$, $4t$. The colors represent the intensity of spectrum function as indicated by the color scale at the bottom. The inset shows the $U$ dependence of the renormalized velocity of edge states at $\lambda =0.1t$ and $0.2t$.