Nesting instability of gapless U(1) spin liquids with spinon Fermi pockets in two dimensions

Quantum spin liquids are exotic states of matter that may be realized in frustrated quantum magnets and feature fractionalized excitations and emergent gauge fields. Here we consider a gapless U(1) spin liquid with spinon Fermi pockets in two spatial dimensions. Such a state appears to be the most promising candidate to describe the exotic field-induced behavior observed in numerical simulations of the antiferromagnetic Kitaev honeycomb model. A similar such state may also be responsible for the recently reported quantum oscillations of the thermal conductivity in the field-induced quantum paramagnetic phase of $\alpha \text{−}{\mathrm{RuCl}}_3$. We consider the regime close to a Lifshitz transition, at which the spinon Fermi pockets shrink to small circles around high-symmetry points in the Brillouin zone. By employing renormalization group and mean-field arguments, we demonstrate that interactions lead to a gap opening in the spinon spectrum at low temperatures, which can be understood as a nesting instability of the spinon Fermi surface. This leads to proliferation of monopole operators of the emergent U(1) gauge field and confinement of spinons. While signatures of fractionalization may be observable at finite temperatures, the gapless U(1) spin liquid state with nested spinon Fermi pockets is ultimately unstable at low temperatures towards a conventional long-range-ordered ground state, such as a valence bond solid. Implications for Kitaev materials in external magnetic fields are discussed.

Figure 1

(a) Schematic spinon band structure of U(1) spin liquid state with a particlelike pocket around the center $\mathit{\Gamma }$ of the hexagonal Brillouin zone and holelike pockets around its corners $\mathbf{K}$ and ${\mathbf{K}}^{\prime }$, proposed as effective description of the field-induced exotic quantum paramagnet occurring for intermediate field strengths $h<h_{\mathrm{c}}$ below the high-field transition in the antiferromagnetic Kitaev model [79, 80]. The Fermi level is indicated as dashed line. (b) Increasing the field strength shifts the particlelike (holelike) band(s) up (down) in energy, such that for $h=h_{\mathrm{c}}$, the bands just touch the Fermi level at isolated points $\mathbf{K}, {\mathbf{K}}^{\prime }$, and $\mathit{\Gamma }$. (c) In the high-field phase, the spinon spectrum is gapped, leading to proliferation of monopoles of the compact U(1) gauge field and confinement of spinons. This describes the conventional paramagnet that is adiabatically connected to the polarized state.

Figure 2

(a) Nested spinon Fermi surface of U(1) spin liquid state for $h<h_{\mathrm{c}}$ with particlelike and holelike pockets around $\mathit{\Gamma }$ and $\mathbf{K}, {\mathbf{K}}^{\prime }$ points in the hexagonal Brillouin zone, depicted as blue and red circles, respectively. The nesting vector ${\mathbf{Q}}_{\text{nesting}}=\mathbf{K}$ is indicated as a green arrow. (b) Upon approaching the Lifshitz transition at $h=h_{\mathrm{c}}$, the spinon Fermi pockets shrink to isolated spinon Fermi points at $\mathit{\Gamma }$ and $\mathbf{K}, {\mathbf{K}}^{\prime }$, depicted as blue and red dots, respectively.

Figure 3

Diagrammatic representation of fermion (a)–(c) and gauge-field (d)–(e) self-energies, as well as vertex corrections for gauge (f)–(n) and interpocket (o)–(u) couplings, at the one-loop order. Solid and wavy inner lines denote fermion and gauge-field propagators, respectively. Diagrams (b) and (c) are independent of external momentum and therefore yield no contribution to the anomalous dimensions. Diagrams (d)–(g) involve closed particle-hole loops, and thus vanish identically in our models. The gauge-field anomalous dimensions are therefore identically zero at the one-loop order. Diagrams (h)–(n) yield finite contributions that are related by means of Ward identities to the fermion anomalous dimensions arising from (a). In the two-pocket model, the particle-particle scattering diagram (o) vanishes. In the three-pocket model, it contributes only to the flow of $\tilde{u}$, while the particle-hole scattering diagram (p) contributes only to the flow of $u$. The flows of $u$ and $\tilde{u}$ therefore do not mix. (q) and (s) Representative diagrams for classes of six and two, respectively, triangle diagrams that contribute to the flows of the interpocket interactions.

Figure 4

RG flow of gauge couplings $e$ and $g$, speed of light $c$, and mass ratio $m_1/m_0$, for both the two-pocket and three-pocket models. Arrows denote flow towards infrared. (a) Flow diagram in $e\text{−}g$ plane for fixed $c=\infty$. The separatrices (purple lines) divide the plane into distinct sectors, for which both $e$ and $g$ are relevant (red arrows) and irrelevant (blue arrows), respectively. The Gaussian fixed point $e=g=0$ is marked as a black dot. (b) Same as (a), but for fixed $c=1$, showing that the slope of the separatrices change as a function of $c>0$, but the qualitative behavior for ${(g/e)}^2\ll 1$ and $(g/e)\gg 1$ remains unchanged. (c) Flow diagram in $e\text{−}c$ plane for fixed ratio $g/e=1$ in the gauge-coupling-irrelevant sector [blue sector in (a) and (b)], showing that $c$ flows to infinity in this sector. (d) Same as (c), but for $g/e=1/3$ in the gauge-coupling-relevant sector [red sector in (a) and (b)], showing that $c$ initially increases, but eventually flows to a finite value $c_{★}\in (0,\infty )$ in the deep infrared. (e) Flow diagram in $e\text{−}\frac{m_1}{m_0}$ plane for fixed ratio $g/e=1$ in the gauge-coupling-irrelevant sector [blue sector in (a) and (b)], showing that the mass imbalance is irrelevant, in the sense that $m_1/m_0$ flows to one. (f) Same as (e), but for $g/e=1/3$ in the gauge-coupling-relevant sector [red sector in (a) and (b)], showing that the mass imbalance is also irrelevant in this sector.

Figure 5

RG flow of interpocket interactions $u$ and $\tilde{u}$ in the three-pocket model for $m_1/m_0=1$. Arrows denote flow towards infrared. (a) Flow diagram in $u\text{−}\tilde{u}$ plane for fixed $e=g=0$, illustrating the Gaussian fixed point at $(u_{★},{\tilde{u}}_{★})=(0,0)$, and the division of parameter space into three distinct regions: For $u<0$ and $\tilde{u}>0$, both couplings are marginally irrelevant. For $u>0$ and $\tilde{u}>-u$, there is a runaway flow towards $u\to \infty$ and $\tilde{u}/u\to 0$, while for $\tilde{u}<0$ and $u<-\tilde{u}$, there is a runaway flow towards $\tilde{u}\to -\infty$ and $u/\tilde{u}\to 0$. (b) Integrated flow in full parameter space spanned by $(e,g,c,u,\tilde{u})$ and projected onto the $u\text{−}\tilde{u}$ plane, in the gauge-coupling-irrelevant sector [blue sector in Figs. 4 and 4]. Here we have chosen fixed initial values of $(e,g,c)=(0.01,0.01,1)$ and different initial values of $(u,\tilde{u})$ along the dashed line depicted in the figure. Despite the marginal irrelevance of $e$ and $g$, there is a runaway flow for all initial values of $u$ and $\tilde{u}$. The thick purple curve separates the upper right subsector, in which $u\to \infty$ and $\tilde{u}/u\to 0$ (heavy blue), from the lower left subsector, in which $\tilde{u}\to -\infty$ and $u/\tilde{u}\to 0$ in the infrared limit (light blue). (c) Same as (b), but in the gauge-coupling-relevant sector [red sector in Figs. 4 and 4] for fixed initial values of $(e,g,c)=(0.03,0.01,1)$. Again, there is a runaway flow for all initial values of $u$ and $\tilde{u}$, with leading divergence $u\to +\infty$ (heavy red) or $\tilde{u}\to -\infty$ (light red).

Figure 6

Mean-field potential in two-pocket model as function of order parameter $\phi$. For $h^2/r=0$, the minimum is at $\phi =0$, while it moves to positive $\phi >0$ for all finite $h^2/r>0$. This corresponds to a spinon gap $\mathrm{\Delta }\propto h^2/r$, as schematically shown in the inset.

Figure 7

Candidate quantum paramagnetic ground states of models for U(1) spin liquids with nested spinon Fermi pockets on the honeycomb lattice. For the three-pocket model with spinon Fermi pockets around $\mathit{\Gamma }$ and $\mathbf{K}, {\mathbf{K}}^{\prime }$ points [79, 80], the low-temperature instability is towards a state with ordering wave vector $\mathbf{Q}=\mathbf{K}$, such as (a) the plaquette valence bond solid (VBS) [89, 90, 134] or (b) the columnar VBS [87, 88, 129, 132, 133]. Both states feature six-site unit cells (dotted hexagons), with the columnar VBS being characterized by a fixed pattern of dimer singlets, while the plaquette VBS is characterized by a pattern of plaquette singlets, each of which consists of an antisymmetric combination of the Kekulé structure, as depicted in the insets. The columnar VBS is also known as Kekulé VBS [131, 133, 138, 139] or Read-Sachdev [87, 88] state in the literature. For the two-pocket [79] and four-pocket [81] models with spinon Fermi pockets around $\mathit{\Gamma }$ and $\mathbf{M}$ points, the divergent interpocket interaction can alternatively also drive an instability towards a state with ordering wave vector $\mathbf{Q}=\mathbf{M}$, such as (c) the zigzag VBS with four-site unit cell (dotted rectangle). In this state, neighboring spins along zigzag chains on the honeycomb lattice form dimer singlets. It breaks lattice rotational symmetry and can be understood as a four-sublattice version of the staggered VBS [135, 136].