Possible nematic spin liquid in spin-1 antiferromagnetic system on the square lattice: Implications for the nematic paramagnetic state of FeSe

The exotic normal state of iron chalcogenide superconductor FeSe, which exhibits vanishing magnetic order and possesses an electronic nematic order, triggered extensive explorations of its magnetic ground state. To understand its novel properties, we study the ground state of a highly frustrated spin-1 system with bilinear-biquadratic interactions using an unbiased large-scale density matrix renormalization group. Remarkably, with increasing biquadratic interactions, we find a paramagnetic phase between Néel and stripe magnetic ordered phases. We identify this phase as a candidate of nematic quantum spin liquid by the compelling evidences, including vanished spin and quadrupolar orders, absence of lattice translational symmetry breaking, and a persistent nonzero lattice nematic order in the thermodynamic limit. The established quantum phase diagram naturally explains the observations of enhanced spin fluctuations of FeSe in neutron scattering measurement and the phase transition with increasing pressure. This identified paramagnetic phase provides a possibility to understand the novel properties of FeSe.

Figure 1

Different quantum phases in the spin-1 $J_1-J_2-K_1-K_2$ model on the square lattice. (a) Quantum phase diagram for $J_2=0.7$ in the $K_1-K_2$ plane. With varying $K_1$ and $K_2$, the system has a stripe and a Néel AFM phase. Between these two phases, we find a paramagnetic (PM) phase with lattice rotational symmetry breaking, which is between the red dash lines. The blue dash-dot line is the semiclassical phase boundary between the stripe and Néel AFM phase. (b)–(d) Magnetic order parameter $m^2\left(\vec{q}\right)$ in momentum space for the different phases. In the stripe (b) and Néel phase (c), $m^2$ has a peak at $\vec{q}=(0,\pi )$ and $(\pi ,\pi )$, respectively. In the paramagnetic phase, $m^2$ is featureless as shown in (d).

Figure 2

$K_1$ and $K_2$ dependence of magnetic order parameters for the $J_1-J_2-K_1-K_2$ square model with $J_2=0.7$ on the $\mathrm{RC}6$-12 cylinder. (a) and (b) Néel order parameter $m^2(\pi ,\pi )$ and stripe order parameter $m^2(0,\pi )$, respectively.

Figure 3

Finite-size scaling of magnetic order parameters. (a) and (b) Size extrapolations of stripe order $m^2(0,\pi )$ and Néel order $m^2(\pi ,\pi )$ versus $1/L$, respectively. We have the system with $J_2=0.7,K_1=0.0$ on the $\mathrm{RC}L-2L$ cylinders with $L=4–10$. Dashed lines are polynomial fits up to fourth order. (c) and (d) Log-log plots of the two magnetic orders versus width $L$.

Figure 4

The absence of stripe AFQ order. (a) Stripe $(\pi ,0)$ AFQ correlation $\langle {\mathbf{Q}}_i·{\mathbf{Q}}_j\rangle$ for $J_2=0.7,K_1=0.0,K_2=0.36$ on the RC8-16 cylinder. The solid green circle in the middle denotes the reference site. The solid blue and shaded red circles denote the positive and negative AFQ correlations, respectively. (b) $K_1,K_2$ dependence of stripe AFQ order parameter $Q^2(\pi ,0)$ on the RC6-12 cylinder. (c) Finite-size scaling of $Q^2(\pi ,0)$ up to width $L=10$.

Figure 5

Lattice symmetry breaking in the intermediate phase. (a) $J_1$ bond energy $\langle {\vec{S}}_i·{\vec{S}}_j\rangle$ for $K_1=0.0,K_2=0.36$ on the RC8-16 cylinder. Here we only show the middle $8\times 8$ sites. (b) Finite-size scaling of bond nematic order ${\sigma }_1$. The inset shows the cylinder length dependence of ${\sigma }_1$ for $K_2=0.36$ and different $L_y$.

Figure 6

Spin gap ${\mathrm{\Delta }}_T$ in the different phases. (a) ${\mathrm{\Delta }}_T$ versus $K_2$ for $J_1=0.7,K_1=0.0$ on different cylinders. (b) Finite-size scaling of spin gap ${\mathrm{\Delta }}_T$ in different phases. To avoid edge excitations, spin gap is obtained by sweeping the middle $L\times L$ sites with total spin $S=1$ based on the ground state of the long $\mathrm{RC}L-L_x$ cylinders with $L_x=24$ and $L=4,6,8$.

Figure 7

A schematic figure for the 45-deg tilted cylinder (TC) on the square lattice. Here the cylinder width is $L_y=4$ and the length is $L_x=12$, which is denoted as TC4-12. For the TC cylinder, the cylinder width is $\sqrt{2}{L}_y$.

Figure 8

Log-linear plots of the spin correlations in the intermediate $K_2$ regime for $J_2=0.7,K_1=0.0$ on the RC6 and TC4 cylinders. The red squares denote the RC6 cylinder, and the blue circles denote the TC4 cylinder.

Figure 9

Bulk energy versus cylinder width $W_y$ on the RC and TC cylinders. The system has $J_2=0.7,K_1=0.0$ and different $K_2$. For (a) $K_2=0.0$, the system is in the $(0,\pi )$ magnetic order phase. For (b) $K_2=0.36$, the system is in the intermediate regime. For (c) $K_2=0.5$, the system is in the $(\pi ,\pi )$ magnetic order phase. The blue circles are the bulk energy for the RC6, RC8, and RC10 cylinders. The red squares denote the energy for the TC4 and TC6 cylinders. For the RC cylinder, cylinder width $W_y=L_y$; for the TC cylinder, $W_y=\sqrt{2}{L}_y$. In the two magnetic order phases, the energies on the two geometries approach each other. However, in the intermediate regime, the TC cylinder appears to have the higher energy than the RC cylinder on our studied system size.

Figure 10

$K_2$ coupling dependence of (a) ground-state energy and (b) entanglement entropy for $J_2=0.7,K_1=0.0$ in different long cylinders with $L_x=24$.

Figure 11

Spin structure factor $S\left(\vec{q}\right)$ for $J_2=0.75,K_1=0.0$ and different $K_2$ on the RC8-16 cylinder. The structure factor is obtained by the Fourier transform from the spin correlations of the middle $8\times 8$ sites. For $K_2=0.4, S\left(\vec{q}\right)$ has the stripe characteristic peak at $\vec{q}=(0,\pi )$. For $0.48\lesssim K_2\lesssim 0.6, S\left(\vec{q}\right)$ is featureless, consistent with the nonmagnetic intermediate phase.

Figure 12

Spin structure factor $S\left(\vec{q}\right)$ for $J_2=0.8,K_1=0.0$ and different $K_2$ on the RC8-16 cylinder. We obtain the data following the way described in the caption of Fig. 11. Here, for $J_2=0.8$, we also find the featureless $S\left(\vec{q}\right)$ for $0.6\lesssim K_2\lesssim 0.75$.

Figure 13

Size dependence of lattice nematic order ${\sigma }_1$ for the parameter points in the intermediate phase regime for $J_2=0.75,0.8,K_1=0.0$.

Figure 14

Néel AFM order parameter $m^2(\pi ,\pi )$ (a) and stripe AFM order parameter $m^2(0,\pi )$ (b) versus $J_2$ and $K_1$ interactions for the $J_1-J_2-K_1$ square model on the RC6-12 cylinder. In both figures, the red dash line denotes the phase transition to the stripe AFM order. The red dots in (a) denote the phase transition from Néel to the intermediate ferroquadrupolar phase, which are determined from the finite-size scaling of magnetic order parameters as shown in Fig. 15.

Figure 15

Finite-size scaling of magnetic order parameters for the $J_1-J_2-K_1$ square model on the $\mathrm{RC}L-2L$ cylinders with $L=4,6,8$. (a) and (b) Néel and stripe magnetic order parameters $m^2(\pi ,\pi )$ and $m^2(0,\pi )$ versus $1/L$, respectively. Lines are polynomial fits.

Figure 16

FQ phase in the $J_1-J_2-K_1$ model. (a) $J_2,K_1$ dependence of FQ order parameter $Q^2(0,0)$ on RC6-12 cylinder. (b) Finite-size scaling of $Q^2(0,0)$ in different phases. (c) Finite-size scaling of lattice nematic order ${\sigma }_1$ for $K_1=-0.8$ and different $J_2$.