Functional renormalization group study of the Kitaev-$\mathrm{\Gamma }$ model on the honeycomb lattice and emergent incommensurate magnetic correlations

The theoretical inception of the Kitaev honeycomb model has had defining influence on the experimental hunt for quantum spin liquids, bringing unprecedented focus onto the synthesis of materials with bond-directional interactions. Numerous Kitaev materials, which are believed to harbor ground states parametrically close to the Kitaev spin liquid, have been investigated since. Yet, in these materials the Kitaev interaction often comes hand in hand with off-diagonal $\mathrm{\Gamma }$ interactions—with the competition of the two potentially giving rise to a magnetically ordered ground state. In an attempt to aid future material investigations, we study the phase diagram of the spin-1/2 Kitaev-$\mathrm{\Gamma }$ model on the honeycomb lattice. Employing a pseudofermion functional renormalization group (pf-FRG) approach, which directly operates in the thermodynamic limit and captures the joint effect of thermal and quantum fluctuations, we unveil the existence of extended parameter regimes with emergent incommensurate magnetic correlations at finite temperature. We supplement our results with additional calculations on a finite cylinder geometry to investigate the impact of periodic boundary conditions on the incommensurate order, thereby providing a perspective on previous numerical studies on finite systems.

Figure 1

Phase diagram of the Kitaev-$\mathrm{\Gamma }$ model on the honeycomb lattice as a function of the angle $\alpha$, which parameterizes the ratio of Kitaev coupling $K=-cos\left(\alpha \right)$ and $\mathrm{\Gamma }$ interaction $\mathrm{\Gamma }=sin\left(\alpha \right)$. The black curve indicates the characteristic RG scale ${\tilde{\mathrm{\Lambda }}}_c$, which is an indicator for the onset energy scale of the magnetic order, see text for details. The shaded regions indicate ordered phases with finite ${\tilde{\mathrm{\Lambda }}}_c$. Observed ordered phases are ferromagnetic order (FM), incommensurate phase 1 (IC1), vortex phase 1 (V1), antiferromagnetic order (AFM), incommensurate phase 2 (IC2), and vortex phase 2 (V2). White regions indicate spin liquid phases and the absence of spontaneous symmetry breaking, mainly the FM and AFM Kitaev spin liquids around $\alpha /\pi =0$ and $\alpha /\pi =1$, respectively.

Figure 2

Flow equations for the cutoff-dependent self-energy ${\mathrm{\Sigma }}^{\mathrm{\Lambda }}\left(\omega \right)$ and the fermionic two-particle interaction vertices (dashed lines) ${\mathrm{\Gamma }}_{i_1{i}_2}^{\mathrm{\Lambda }}(1^{\prime },2^{\prime };1,2)$ in diagrammatic representation, where composite indices $k=({\omega }_k,{\alpha }_k)$ comprise frequency and spin index. Lattice sites are preserved along solid lines. The self-energy is independent of lattice site and spin index. Slashed propagator lines resemble the single-scale propagator $S\left(\omega \right)=\frac{\delta \left(\left|\omega \right|-\mathrm{\Lambda }\right)}{i\omega -{\mathrm{\Sigma }}^{\mathrm{\Lambda }}\left(\omega \right)}$, pairs of slashed propagator lines denote $G\left({\omega }_1\right)S_{\mathrm{kat}}\left({\omega }_2\right)+G\left({\omega }_2\right)S_{\mathrm{kat}}\left({\omega }_1\right)$ with $G\left(\omega \right)=\frac{\theta \left(\left|\omega \right|-\mathrm{\Lambda }\right)}{i\omega -{\mathrm{\Sigma }}^{\mathrm{\Lambda }}\left(\omega \right)}$ and $S_{\mathrm{kat}}\left(\omega \right)=-\frac{d}{d\mathrm{\Lambda }}G\left(\omega \right)$. Details are provided in Ref. [56].

Figure 3

Renormalization group flow of the on-site magnetic susceptibility ${\chi }_{ii}^{\mathrm{\Lambda }}$ for different lattice truncation ranges $N_L$. The dashed lines (if present) indicate the characteristic RG scale ${\tilde{\mathrm{\Lambda }}}_c$, which is associated with the occurrence of spontaneous symmetry breaking. The data displayed is in the (a) Kitaev spin liquid (KSL) phase at $\alpha /\pi =0$, (b) incommensurate phase 1 (IC1) at $\alpha /\pi =0.37$, (c) vortex phase 1 (V1) at $\alpha /\pi =0.72$, (d) incommensurate phase 2 (IC2) at $\alpha /\pi =1.30$, and (e) vortex phase 2 (V2) at $\alpha /\pi =1.68$.

Figure 4

Structure factors of the Kitaev-$\mathrm{\Gamma }$ model on the honeycomb lattice in the different phases. The dashed black lines indicate the first Brillouin zone, the extended Brillouin zone is denoted by the the solid lines. (a) FM KSL phase, plotted at $\alpha =0$. (b) IC1 phase at $\alpha /\pi =0.37$. (c) V1 phase at $\alpha /\pi =0.72$. (d) AFM KSL phase at $\alpha /\pi =1$. (e) IC2 phase at $\alpha /\pi =1.30$. (f) V2 phase at $\alpha /\pi =1.68$. The color code is normalized within each plot. All structure factors are plotted at the characteristic RG scale ${\tilde{\mathrm{\Lambda }}}_c$.

Figure 5

Incommensurate ordering vectors in the two incommensurate phases IC1 and IC2. The plots show the position ${\mathbf{k}}_{\max }$ of the structure factor peak within the Brillouin zone. (a) The peak in the IC1 phase moves continuously on the momentum-space line (indicated as a red arrow in the inset) from the BZ center ($\mathrm{\Gamma }$ point, corresponds to relative distance $d_{M\mathrm{\Gamma }}=0$) to the BZ edge ($M$ point, corresponds to relative distance $d_{M\mathrm{\Gamma }}=1$) as a function of the angle $\alpha$. The dashed lines indicate the phase boundaries of the IC1 phase. (b) Relative distance of the peak in the IC2 phase between the ${\mathrm{\Gamma }}^{\prime }$ point and the $M^{\prime }$ point.

Figure 6

Finite cylinder geometry. (a) The rhombic unit cell is indicated by dashed lines. The cylinder extends infinitely in the $a_1$ direction and is assumed to have periodic boundary conditions in the $a_2$ direction. (b) Red lines indicate lines of allowed momenta within the extended BZ for a circumference of 6 lattice sites (3 unit cells). Dashed lines indicate the first BZ. (c) Explicit breaking of symmetry by modulating the exchange constants $(\tilde{K},\tilde{\mathrm{\Gamma }})\ne (K,\mathrm{\Gamma })$ along the $x$ and $y$ bonds.

Figure 7

Bond anisotropy on a cylinder with rhombic unit cell and 6-site circumference as a function of the angle $\alpha$ (blue curve), plotted at the characteristic scale ${\tilde{\mathrm{\Lambda }}}_c$. Solid markers denote ${\zeta }_x={\zeta }_y$, open markers denote ${\zeta }_z$. The grey shaded regions correspond to the incommensurate phases IC1 and IC2, respectively. The yellow curve shows the anisotropy on an infinite lattice with small explicit symmetry breaking in the exchange constants, $\epsilon =0.01$ (see text for details).