Mott transition and magnetism of the triangular-lattice Hubbard model with next-nearest-neighbor hopping

The variational cluster approximation is used to study the isotropic triangular-lattice Hubbard model at half filling, taking into account the nearest-neighbor $\left(t_1\right)$ and next-nearest-neighbor $\left(t_2\right)$ hopping parameters for magnetic frustrations. We determine the ground-state phase diagram of the model. In the strong-correlation regime, the ${120}^{\circ }$ Néel- and stripe-ordered phases appear, and a nonmagnetic insulating phase emerges in between. In the intermediate correlation regime, the nonmagnetic insulating phase expands to a wider parameter region, which goes into a paramagnetic metallic phase in the weak-correlation regime. The critical phase boundary of the Mott metal-insulator transition is discussed in terms of the van Hove singularity evident in the calculated density of states and single-particle spectral function.

Figure 1

Schematic representations of (a) the triangular-lattice Hubbard model with the nearest-neighbor $\left(t_1\right)$ and next-nearest-neighbor $\left(t_2\right)$ hopping parameters, (b) the ${120}^{\circ }$ Néel order, and (c) the stripe order. The arrows represent the directions of electron spins on the A, B, and C sublattices defined by different colors.

Figure 2

(a) The reference system of the 12-site cluster used in our analysis; the three-sublattice system corresponding to the ${120}^{\circ }$ Néel order (left) and the two-sublattice system corresponding to the stripe order (right). (b) The first Brillouin zone of our triangular-lattice Hubbard model: $\mathrm{\Gamma }(0,0),\mathrm{K}(4\pi /3,0)$, and $\mathrm{M}(\pi ,\pi /\sqrt{3})$.

Figure 3

Calculated ground-state phase diagram of our model in the strong-correlation regime $\left(U/t_1=60\right)$. Top: the order parameters of the ${120}^{\circ }$ Néel- and stripe-ordered phases. Solid (open) symbols indicate that the state is stable (metastable). Bottom: the ground-state energies (per site) of the ordered phases compared with that of the disordered phase. Inset: the enlargement of the energy difference $\mathrm{\Delta }E$ between the ${120}^{\circ }$ ordered and disordered phases.

Figure 4

Calculated ground-state phase diagram of our model in the intermediate- to weak-correlation regime, which includes the ${120}^{\circ }$ Néel-ordered, stripe-ordered, nonmagnetic insulating, and paramagnetic metallic phases. The circle and triangle at $U/t_1=60$ indicate the calculated phase boundaries of the ${120}^{\circ }$ Néel order and stripe order, respectively, shown in Fig. 3.

Figure 5

Calculated charge gap $\mathrm{\Delta }$ (top) and order parameters $M_{{120}^{\circ }}$ and $M_{\mathrm{str}}$ (bottom) of our model as a function of $U/t_1$.

Figure 6

Calculated DOS of our model in the metallic state (left) and insulating state without long-range magnetic orders (right). $\eta /t_1=0.1$ is assumed. The vertical line in each panel indicates the Fermi level.

Figure 7

Calculated single-particle spectral function $A(\mathit{k},\omega )$ in the paramagnetic state of our model. The wave vector $\mathit{k}$ is chosen along the line connecting $\mathrm{\Gamma }$, K, and M points of the Brillouin zone [see Fig. 2]. $\eta /t_1=0.1$ is assumed. The noninteracting band dispersion is also shown by a thin solid curve in each of the upper panels. The Fermi level (indicated by the vertical line) is set at $\omega /t_1=0$.

Figure 8

Calculated generalized magnetic susceptibility in the noninteracting limit ${\chi }_0\left(\mathit{q}\right)$ defined in Eq. (11). The corresponding Fermi surface is shown in each panel, where the first Brillouin zone is indicated by a hexagon.

Figure 9

(a) Cluster-size and cluster-shape dependencies of the phase boundaries between the ${120}^{\circ }$ Néel-ordered, stripe-ordered, nonmagnetic insulating, and paramagnetic metallic phases. Unlabeled lines are the results shown in Fig. 4. (b) Schematic representations of the clusters used in the calculations; in the six-site cluster calculation, we combine two of them to reproduce the ${120}^{\circ }$ Néel order.