Kane-Mele Hubbard model on a zigzag ribbon: Stability of the topological edge states and quantum phase transitions

We study the quantum phases and phase transitions of the Kane-Mele Hubbard (KMH) model on a zigzag ribbon of honeycomb lattice at a finite size via the weak-coupling renormalization group (RG) approach. In the noninteracting limit, the Kane-Mele (KM) model is known to support topological edge states where electrons show helical property with orientations of the spin and momentum being locked. The effective interedge hopping terms are generated due to finite-size effect. In the presence of an on-site Coulomb (Hubbard) interaction and the interedge hoppings, special focus is put on the stability of the topological edge states (TI phase) in the KMH model against (i) the charge and spin gaped (II) phase, (ii) the charge gaped but spin gapless (IC) phase, and (iii) the spin gaped but charge gapless (CI) phase depending on the number (even/odd) of the zigzag ribbons, doping level (electron filling factor) and the ratio of the Coulomb interaction to the interedge tunneling. We discuss different phase diagrams for even and odd numbers of zigzag ribbons. We find the TI-CI, II-IC, and II-CI quantum phase transitions are of the Kosterlitz-Thouless (KT) type. By computing various correlation functions, we further analyze the nature and leading instabilities of these phases. The relevance of our results for graphene is discussed.

Figure 1

Honeycomb lattice of a finite-sized zigzag ribbon of the tight-binding Kane-Mele model with the ribbon size $N=4$ $(N$ being the number of zigzag chains along $x$ axis) along $y$ axis. The honeycomb lattice consists of two interpenetrating triangular lattices denoted by sublattice $A$ (dark circles) and sublattice $B$ (open circles) with lattice vectors ${\mathbf{a}}_{\mathbf{1}}$ and ${\mathbf{a}}_{\mathbf{2}}$ (dashed arrows). The zigzag ribbon shows translational symmetry along $x$ axis. The nearest-neighbor lattice vectors between nearest-neighbor $A$ and $B$ sites are denoted by ${\mathbf{e}}_{\mathbf{i}=\mathbf{1},\mathbf{2},\mathbf{3}}$ with a lattice constant $a$. The red (black) arrows within sublattice $A\left(B\right)$ represent the directions of the next-nearest-neighbor hopping term ${\lambda }_{\mathrm{SO}}$ in the KM model (see text). The gray shaded region represents for the super unit cell of the zigzag ribbon, which repeats itself along $x$ axis.

Figure 2

Energy spectrum of the finite-sized Kane-Mele model on a zigzag ribbon for (a) $N=4$ and (b) 5 of honeycomb lattice. Here, we set $t=1$ and ${\lambda }_{\mathrm{SO}}/t=0.2$.

Figure 3

Energy spectrum of the finite-sized Kane-Mele model on a zigzag ribbon for (a) $N=16$ and (b) 15 of honeycomb lattice. Here, we set $t=1$ and ${\lambda }_{\mathrm{SO}}/t=0.2$.

Figure 4

The square magnitude of the edge state wave function ${\left|\Psi \right|}^2$ of the KM zigzag ribbon at half-filling as a function of $y/b+1$ (defined in text) for $N=14$ and (a) for $ka=\pi$ and (b) for $ka=\pi \pm 0.2\pi$. Here, ${\left|\Psi \right|}^2$ (blue circles and red squares) represent for the square magnitude of the two edge state wave functions, which are degenerate eigenstates at the corresponding wave vector $k$. In (a), the two hybridized degenerate edge state wave functions $\Psi ={\Psi }_{\mathrm{hyb},i=1,2}$ (red and blue symbols) lead to the same square magnitude, $|{\Psi }_{\mathrm{hyb},1}{|}^2={\left|{\Psi }_{\mathrm{hyb},2}\right|}^2$, in (b), we make the following identifications: $\Psi \left(y\right)={\Psi }_{R,1}^{\uparrow }$ (blue) and $\Psi \left(y\right)={\Psi }_{L,2}^{\uparrow }$ (red). The solid lines are guides to the eyes in (a), and in (b) they are fits to the exponential form in Eq. (6). We set ${\lambda }_{\mathrm{SO}}/t=0.1$.

Figure 5

The square magnitude of the edge state wave function ${\left|\Psi \right|}^2$ of the KM zigzag ribbon at half-filling as a function of $y/b+1$ for (a) $N=15$ and $ka=\pi$ and (b) for $N=15$ and $ka=\pi \pm 0.2\pi$. Here, ${\left|\Psi \right|}^2$ (blue circles and red squares) represents for the square magnitude of the two edge state wave functions, which are degenerate eigenstates at the corresponding wave vector $k$. The solid lines are fits to the exponential form in Eq. (6). We set ${\lambda }_{\mathrm{SO}}/t=0.1$. Note that in (a) ${\left|\Psi \right|}^2$ is shown for only even values of $y/b$ (see text).

Figure 6

Schematic diagram for the interedge hopping term $t_{\perp }$ (red or blue dashed line).

Figure 7

(a) Energy spectrum $\epsilon$ vs momentum $k$ of the topological edge states of the finite-sized $(N$ zigzag chains) Kane-Mele model on a zigzag ribbon of honeycomb lattice near the Dirac point $ka=\pi$ for different ribbon sizes. Here, we set $t=1,{\lambda }_{\mathrm{SO}}/t=0.5$. (b) Energy gap $\Delta$ at the Dirac point as a function of $N$ for different values of ${\lambda }_{\mathrm{SO}}$.

Figure 8

Schematic diagrams for (a) the interedge umklapp $g_{\mathrm{um}}$ (red and blue arrows) and (b) the interedge spin-flip $g_{\sigma }^{\perp }$ processes.

Figure 9

The exponential decay of ${\overline{t}}_{\perp }$ as a function of odd number of zigzag chains $N$.

Figure 10

Quantum phase diagram of the Kane-Mele Hubbard model at half-filling for $N=\mathrm{even}$ as a function of $U/t$ and $t_{\perp }^2/t$. The helical topological edge states (TI phase) is stable only at $U=t_{\perp }=0$ (dark circle). For a finite ribbon size, $t_{\perp }\ne 0$, the system flows to a charge and spin gaped (charge and spin insulating or II) phase.

Figure 11

The RG flows of the Kosterlitz-Touless type for the spin sector $\left(g_{\sigma }^{\perp },g_{\sigma }^z\right)$ of the zigzag Kane-Mele Hubbard ribbon for $N=\mathrm{even}$ away from half-filling. The black circle stands for the initial (bare) couplings. The arrows indicate the directions of the RG flows upon decreasing the curt-off scale $\mu$ from ${\mu }_0$. The red line represents a line of fixed points in the TI phase, the TI-CI phase boundary is defined by the separatrix line (thick black arrow). Note that the coupling $g_{\rho }$ does not flow under RG in this case [see Eq. (34)].

Figure 12

Quantum phase diagram of the Kane-Mele Hubbard model away from half-filling for $N=\mathrm{even}$ as functions of $t_{\perp }^2/t$ and $U/t$. The helical topological edge states (TI phase) are unstable towards the charge conducting and spin insulating CI phase for $t_{\perp }^2/t>2U$. The TI-CI quantum phase transition set by the boundary $t_{\perp }^2/t=2U$ is of the Kosterlitz-Thouless (KT) type (red dashed arrows).

Figure 13

The RG flows of the Kosterlitz-Touless type for the charge sector $\left(g_{\rho },g_{\mathrm{um}}\right)$ of the zigzag Kane-Mele Hubbard ribbon at half-filling for $N=\mathrm{odd}$. The black circle stands for the initial (bare) couplings at $(g_{\rho }^0,-g_{\mathrm{um}}^0)=(U/2,{\overline{t}}_{\perp }^2/t)$. The arrows indicate the directions of the RG flows upon decreasing the curt-off scale $\mu$ from ${\mu }_0$. The red line represents a line of fixed points in the CI phase, the CI-II boundary is defined by the separatrix line (thick black arrow) and its quantum transition is of the Kosterlitz-Thouless (KT) type. The topological TI phase is stable only at the origin $U=0={\overline{t}}_{\perp }$.

Figure 14

The RG flows of the Kosterlitz-Touless type for the spin sector $\left(g_{\sigma }^{\perp },g_{\sigma }^z\right)$ of the zigzag Kane-Mele Hubbard ribbon for $N=\mathrm{odd}$ at half-filling. The black circle stands for the initial (bare) couplings at $(g_{\sigma }^{z,0},g_{\sigma }^{\perp ,0})=(-2U,{\overline{t}}_{\perp }^2/t)$. The arrows indicate the directions of the RG flows upon decreasing the curt-off scale $\mu$ from ${\mu }_0$. The red line represents a line of fixed points in the IC phase, the II-IC phase boundary is defined by the separatrix line (thick black arrow) and its quantum transition is of the Kosterlitz-Thouless (KT) type. The topological TI phase is stable only at the origin $U=0={\overline{t}}_{\perp }$.

Figure 15

Quantum phase diagram of the zigzag Kane-Mele Hubbard ribbon for $N=\mathrm{odd}$ at half-filling as a function of $U/t$ and ${\overline{t}}_{\perp }^2/t$. The helical topological edge states (TI) are unstable against any $U\ne 0$ or ${\overline{t}}_{\perp }\ne 0$, and towards the IC, CI, and II phases for $U>{\overline{t}}_{\perp }^2/\left(2t\right),U<-2{\overline{t}}_{\perp }^2/t$ and $-2{\overline{t}}_{\perp }^2/t<U<\frac{1}{2}{\overline{t}}_{\perp }^2/t$, respectively. The II-IC and II-CI phase transitions are of the Kosterlitz-Thouless (KT) type (red dashed arrows).

Figure 16

The RG flows of the Kosterlitz-Touless type for the spin sector $\left(g_{\sigma }^{\perp },g_{\sigma }^z\right)$ of the zigzag Kane-Mele Hubbard ribbon away from half-filling for $N=\mathrm{odd}$. The black circle stands for the initial (bare) couplings. The arrows indicate the directions of the RG flows upon decreasing the curt-off scale $\mu$ from ${\mu }_0$. The red line represents a line of fixed points in the TI phase, the TI-CI phase boundary is defined by the separatrix line (thick black arrow). Note that the coupling $g_{\rho }$ does not flow under RG in this case [see Eq. (34)].

Figure 17

Quantum phase diagram of the zigzag Kane-Mele Hubbard ribbon away from half-filling for $N=\mathrm{odd}$ as functions of ${\overline{t}}_{\perp }^2/t$ and $U/t$. The helical topological edge states (TI phase) are unstable towards the charge conducting and spin insulating CI phase for ${\overline{t}}_{\perp }^2/t>2U$. The TI-CI quantum phase transition set by the boundary ${\overline{t}}_{\perp }^2/t=2U$ is of the Kosterlitz-Thouless (KT) type (red dashed arrows).