Microscopic Model for Fractional Quantum Hall Nematics

,

A general mechanism believed to give rise to a FQHN from a proximate incompressible FQH state is the softening of the magnetoroton mode in the long-wavelength limit [3,4], Fig. 1.The magnetoroton mode is a collective density wave excitation [36][37][38] that occurs in many FQH states, including the Laughlin [39] and Moore-Read [40] states.At long wavelengths, the mode is described by the Girvin, MacDonald, and Platzman (GMP) ansatz [41,42].Unlike the topological properties of FQH states, such as the Hall conductance, the long-wavelength limit of the GMP mode has a geometric character: in an effective field theory, it behaves as a quadrupole that can be described by a quantum metric [43][44][45][46][47][48][49][50].
Experiments in the second LL with tilted magnetic fields [51] and hydrostatic pressure [52,53] have observed anisotropic resistance that was attributed to nematicity.These experiments indeed suggest that nematic order can be proximate to incompressible FQH states or even coexist with them.In addition, a Raman scattering experiment at filling factors ν=5/2 and 7/2 has provided evidence of an interplay between nematicity and electron pairing [54].A major challenge in bridging the FIG. 1. Schematic of the phase diagram and the FQHN transition for bosons, driven by varying the shortest-range potential from its Coulomb value by δV0.In the nematic phase, the charge gap remains open, while the neutral gap has closed due to the presence of a Goldstone mode associated with the spontaneous breaking of continuous rotation symmetry.Upon further softening of the shortest-range repulsion, the system becomes fully gapless, e.g., by also breaking translation symmetry (CDW).arXiv:2401.17352v3[cond-mat.str-el]10 Jun 2024 gap between these experimental observations and effective theories of FQHNs has been the lack of a microscopic model.Early work [55] introduced a class of variational Laughlin-like wave functions that break the continuous rotational symmetry.More recently, Ref. [56] proposed a toy model exhibiting some signatures of the FQHN.For a similar class of short-range interacting models, Ref. [57] formulated conditions for the quadrupole excitation to become gapless.However, the identification of FQHNs in realistic models has been lacking.
In this paper, we consider a quintessential model of the FQH effect -Coulomb interaction projected to the lowest LL (LLL) [58][59][60].Contrary to the general belief that the softening of the shortest range component of the Coulomb interaction results in a destruction of the FQH liquid and direct transition to a CDW, we argue that this model realizes the scenario of the FQHN transition as shown in Fig. 1.Using exact diagonalization (ED), we demonstrate the hallmarks of the FQHN in this model: the GMP mode goes soft at long wavelengths and the neutral gap closes, while the charge gap remains open.Furthermore, we demonstrate the symmetry-breaking nature of the transition and show that the nematic susceptibility and the Hall viscosity both diverge in its vicinity.The results of exact numerics are supported by estimates of the gaps in larger systems using the GMP ansatz and composite fermion (CF) theory [61].Below we present illustrative results for the Laughlin state of bosons at filling factor ν b =1/2, which is relevant for recent experiments in cold atom setups [62].However, in the Supplemental Material (SM) [63], we demonstrate that our results equally apply to the ν=1/3 Laughlin state of electrons.
Model.We consider a standard model for the FQH effect where N electrons, interacting pairwise via the Coulomb potential, are confined to a spherical surface [64], with a Dirac monopole at the center, emanating a radial magnetic flux of strength 2Qhc/e.The radius of the sphere is R= √ Qℓ B , where ℓ B = ℏc/eB is the magnetic length.The Hamiltonian is translationand rotation-invariant, which implies that the total orbital angular momentum L and its z-component M are good quantum numbers.The magnitude of the planar wavevector k is given by k=L/R.The rotational invariance of the interaction implies that it can be expressed in terms of Haldane pseudopotentials {V m }, where m=0, 1, 2, 3, . . . is the relative angular momentum of any pair of particles [58,64].Hence, m is constrained by the statistics of the particles: for bosons, only even pseudopotentials V 0 , V 2 , V 4 , etc. are relevant.
We will consider the filling factor ν b =1/2 for bosons, at which the Laughlin state [39] is realized if we set the magnetic monopole flux to 2Q=2N −S, with S=2 denoting the Wen-Zee shift [65].Starting from the Coulomb interaction, we soften the shortest-range component of the interaction potential and therefore the model is parametrized by a single number δV 0 =V 0 −V C 0 , where V C 0 is the Coulomb interaction's V 0 value [66].Spectral properties.We first analyze the energy spectra of the ν b =1/2 bosonic state obtained using ED, Fig. 2(a)-(b).Throughout, energies are expressed in units of Coulomb energy, E C ≡e 2 /ϵℓ B .While there is a gapped magnetoroton mode for the pure Coulomb interaction [Fig.2(a)], the dispersion of the collective mode is significantly altered by reducing the V 0 component of the interaction.As we add a delta interaction of strength δV 0 =−0.4, the wave number corresponding to the roton minimum changes from kℓ B ∼1.5 to k→0 [Fig.2(b)].The softening of the magnetoroton mode implies that the neutral gap decreases as V 0 is reduced and potentially closes in the long-wavelength limit.
In Fig. 2(c)-(d) we show the gaps as a function of δV 0 .We evaluate two types of gaps for accessible system sizes and perform their extrapolation to the thermodynamic limit in 1/N : (i) the neutral gap is defined as the difference in the lowest two energies at the ground state flux, 2Q=2N −2; (ii) the charge or transport gap, i.e., the energy to create an individual quasihole (occurs at flux 2Q+1) and quasiparticle excitation (occurs at flux 2Q−1).An alternative way to estimate the charge gap is to extrapolate the gap ∆ L=N , the energy difference be- tween the ground state and the lowest-lying state with orbital angular momentum L=N , at the ground state flux.The L=N roton state is formed by a quasiparticle and a quasihole on opposite poles of the sphere.The interaction between these localized excitations vanishes in the thermodynamic limit, hence the charge gap is simply the sum of their individual energies.
The results in Fig. 2(c) show a sharp drop in both the charge and neutral gaps as the interaction is softened by δV 0 ≈−0.4.More precisely, at δV 0 =−0.43, the neutral gap drops to zero (or slightly below, due to the uncertainty of the extrapolation), however, the charge gap, while significantly reduced compared to the pure Coulomb point, remains non-zero.This is demonstrated by the raw scaling data for gaps as a function of 1/N in Fig. 2(d).The reliability of the extrapolation is confirmed by independently calculating ∆ L=N , which extrapolates to the same value as the charge gap, Fig. 2(d).
Variational wave functions.To access system sizes beyond ED, we employ GMP [41] and composite fermion (CF) [67] ansatz wave functions.Fig. 3 shows the gap estimates from such wave functions.We approximate the ground state by the bosonic Laughlin wave function [39] which has a high overlap (upwards of 0.85 for all N ≤14) with the exact ground state across the range δV 0 ∈[−0.43,0].The exact spectra, Fig. 2(b), show that the gap can be upper-bounded by the GMP excitation that carries angular momentum L=2 on the sphere [68].On the other hand, to estimate the charge gap, we use the CF exciton wave functions, which accurately describe the entire magnetoroton branch [69,70].The CF exciton wave function of the Laughlin state in the long wavelength limit is identical to the GMP ansatz.We estimate the charge gap by the energy of the L=N member of the CF-exciton mode [61,71].
We independently evaluate the Coulomb and V 0 =1 pseudopotential gaps for many systems using the Monte Carlo method and extrapolate these gaps to the thermodynamic limit [63].By superposing these gaps, we extract the variational gap for an arbitrary combination of the Coulomb and V 0 pseudopotential.As shown in Fig. 3, the neutral and charge gaps have different slopes as a function of δV 0 , the former decaying faster.Thus, there is a region of parameter space 0.43≤−δV 0 ≤0.84 where the neutral gap vanishes while the charge gap remains finite.This variational estimate of the FQHN phase boundaries is consistent with the ED results above.
Symmetry breaking and geometric response.As an order parameter for the FQHN transition, we utilize the deformed Laughlin states which partly break rotational symmetry [55,72]: where we have suppressed the usual Gaussian factor and the parameter α controls the breaking of rotational symmetry by "splitting" the zeros of the wave function.For small α, these wave functions, as well as related ones in Refs.[43,72], describe an incompressible fluid rather than an FQHN.Nevertheless, we can use their variational energy to define a "mean-field" order parameter: we keep the interaction fully rotation-invariant, while we evaluate the expectation value of the energy for the anisotropic wave functions.For δV 0 >δV critical 0 , we expect the energy minimum to correspond to α=0, i.e., the isotropic Laughlin wave function, while for δV 0 <δV critical 0 the minimum should shift to a non-zero α * >0.In the vicinity of a second-order phase transition, mean-field theory predicts a critical component β=1/2 for the order parameter α * ∝(δV critical 0 −δV 0 ) β [4].This scenario is confirmed in Fig. 4(a), which shows the energy of Ψ ν b =1/2,α states for the LLL-projected Coulomb interaction at several values of δV 0 .The isotropic Laughlin state (recovered at α=0) indeed yields the minimum of energy when |δV 0 |≤0.44=−δVcritical 0 .This estimate of the FQHN transition point is in good agreement with our previous exact results for the closing of the neutral gap.Upon further reducing V 0 , the lowest energy state appears at a nonzero α * value.The scaling of α * near criticality is presented in the inset of Fig. 4(a), which is consistent with the mean-field exponent 1/2.
To characterize the geometric nature of the FQHN transition, we have computed the nematic susceptibility introduced in Ref. [56], see Fig. 4(b).This calculation is performed in the torus geometry [73] by varying the aspect ratio.Fig. 4(a) shows that deep in the Laughlin phase the nematic susceptibility is zero while close to the transition point, δV 0 =−0.43, it jumps to a non-zero value and thereafter varies erratically with system size.Moreover, we have computed the Hall viscosity η A on the torus [74][75][76] as the system is tuned towards the FQHN critical point [63].For gapped FQH states, the Hall viscosity is quantized by the shift, η A =(ℏρ/4)S [77], where ρ is the fluid density.The extracted value of S is plotted in the inset of Fig. 4(b) as a function of δV 0 .Deep in the gapped phase, δV 0 ≳−0.3, the shift is robustly quantized to the Laughlin value S=2.Near the transition but still in the gapped phase, the smaller systems depart from the quantized value, which is attributed to an increase in the correlation length.Finally, as we hit the FQHN critical point, the quantization of η A breaks down completely.In the SM [63], we show that the onset of the transition is also signaled by the dynamical response of the system to a geometric quench.
Experimental implications.The key ingredient of our model -the softening of the V 0 pseudopotential of the Coulomb interaction -can realistically arise due to the finite width of the 2DEG or screening by electrostatic gates.LL mixing, in particular, leads to a large reduction of V 0 [78][79][80].Moreover, the node in the single-particle wave functions in higher LLs tends to expose the magnetoroton mode in the long-wavelength limit [81], thus providing a natural setting for FQHNs [82].One of the main experimental challenges is distinguishing the FQHN from a CDW.Resonant inelastic Raman scattering [36,54,83] or surface acoustic waves [38] can map out the magnetoroton mode and confirm whether it closes in the longwavelength limit.Recent advances in the scanning tunneling microscopy of FQH states [84][85][86][87] could provide further insights into the FQHN formation, generalizing the previous observation of nematicity of free electrons in bismuth [31].
Beyond solid-state systems, interactions in synthetic systems of alkaline atoms can be tuned by coupling to the highly excited Rydberg levels.The Rydberg blockade radius that simulates the V 0 interaction can be tuned in these platforms and both FQH and crystalline phases have been shown to arise in these models [88,89].The long-range interactions in Rydberg-dressed atoms as well as dipolar gases [90] make them a promising platform for FQH physics [91].Other platforms where bosonic FQH states can be stabilized are polaritons where an artificial magnetic field and LLs can be generated by rotating the medium through which light propagates [92].A suitably chosen medium could potentially allow to engineer a desired interaction [93,94].
Conclusions.In summary, we have presented a microscopic model that exhibits key features of the FQHN transition.Intriguingly, the model shows clearer FQHN signatures compared to short-range models, such as the one in Ref. [56].We speculate that this is due to the tendency of the Coulomb interaction to "pull down" the magnetoroton mode below the spectral continuum in the long-wavelength limit, as seen in Fig. 2. By contrast, in short-range models, the long-wavelength limit of the magnetoroton mode is clearly inside the continuum [95].Nematic susceptibility from Ref. [56] for bosons at ν b =1/2 on the torus with a square unit cell for several system sizes indicated in the legend.As V0 is reduced and the critical point is approached, we observe a sharp increase and strong finite-size fluctuations in the nematic susceptibility.Similar behavior is found for the shift S shown in the inset.We extract the shift from the Hall viscosity η A on the torus near the square aspect ratio.In the gapped FQH phase, the shift is quantized to the S=2 value expected in the Laughlin state.Similar to the nematic susceptibility, as the FQHN critical point is approached, we observe large fluctuations in S, which is no longer quantized.
The necessary conditions for the gaplessness of the L=2 neutral excitation in short-range models have been derived in Ref. [57], and it would be interesting to generalize those results to long-range interactions.
While we have provided multiple pieces of evidence for the FQHN critical point, the nature of the phase past criticality is not fully understood.At large negative δV 0 , we expect a CDW to become the ground state.On the sphere, the study of CDW phases is hindered by frustration effects, requiring very large systems to observe the expected closing of the charge gap.On a torus, by varying the aspect ratio, we have indeed found a proximate phase with a manifold of ground states consistent with CDW ordering [63].Thus, the FQHN phase found above could be proximate to a CDW.One important question is: How general are the results above?In the SM [63], we show that fermions at ν=1/3 behave similarly to bosons at ν b =1/2 considered above.Moreover, other incompressible states, in particular ν=2/5 and 3/7 Jain states, also have collective modes "exposed" below the continuum of the spectrum [96,97].We leave their study to future work.We note, however, that the FQHN presented above could potentially be realized even beyond non-interacting CF states.For example, we have checked that bosons at ν b =1, which realize the Moore-Read state [40,[98][99][100][101], support a similar phenomenology.Beyond the nematic instability of incompressible FQH states, it would be interesting to explore a compressible nematic which could arise in third and higher Landau levels [18,[102][103][104], e.g., due to the quantum (or thermal) melting of a stripe state [26,82,105] or a Pomeranchuk instability of the compressible ν=1/2 state [48,106].
In the main text, we have computed three types of gaps to characterize the fractional quantum Hall nematic (FQHN) transition: (i) the neutral gap is defined as the difference in the lowest two energies at the ground state flux, i.e., 2Q=2N −2 for the ν b =1/2 Laughlin state; (ii) the charge or transport gap, i.e., the energy to create an individual quasihole (occurs at flux 2Q+1) and quasiparticle (occurs at flux 2Q−1) on top of the ground state; (iii) the gap ∆ L=N -the energy difference between the ground state and the lowest-lying state with orbital angular momentum L=N -at the ground state flux.These three types of gaps were computed for the accessible system sizes N =6−12 bosons and their extrapolation to the thermodynamic limit in 1/N was performed.For the charge gap, it is important to include the neutralizing background charge correction [61,108] Since we are considering modified Coulomb interaction, the total electrostatic energy is evaluated according to the general formula in the Supplementary Material of Ref. [109], which we reproduce here for convenience: where n q is an integer taking values 0 (for the ground state) or ±1 for the quasiparticle and quasihole excitations that carry charge e * =e/2 in our case.The energy correction explicitly depends on the number of flux quanta 2Q (we have assumed we are in the lowest Landau level).The average charging energy of the full LL per pair, C, is determined by all pseudopotentials V m , which includes any modification to the V 0 .Furthermore, to minimize the finite-size effects, we multiply all energies with the density-correction factor 2Qν b /N [110].
In the case of charge gap, we do not density-correct the individual quasihole and quasiparticle energies but only apply the rescaling at the end.
We use the gap ∆ L=N as a cross-check for the extrapolated value of the charge gap in the thermodynamic limit.The L=N magnetoroton state is formed by a quasiparticle and a quasihole on opposite poles of the sphere.As the charge gap is expected to remain finite across the FQHN transition, the quasiparticle and quasihole are localized objects with a few ℓ B in diameter.By contrast, their interaction scales inversely with the diameter of the sphere, 2R∝ √ N ℓ B .Thus, in the thermodynamic limit, the quasiparticle and quasihole are infinitely far from each other and thus the interaction between them vanishes in this limit and they can be regarded as independent entities, the sum of whose energies is equivalent to the charge gap.

VARIATIONAL WAVE FUNCTIONS FOR THE MAGNETOROTON MODE
For completeness, we provide an overview of three construction schemes of variational wave functions that we have used to approximate the neutral collective mode in the main text.

Jack polynomial wave functions and Girvin-MacDonald-Platzman ansatz
Accurate model wave functions for the magnetoroton mode of the Laughlin state can be constructed in the framework of Jack polynomials [111].The construction applies equally to bosons or fermions, and below we briefly review the fermionic version, commenting on how to adapt it to bosons at the end.The Laughlin wave function at ν=1/3 is a single Jack polynomial characterized by the so-called root partition λ≡{100100100100• • •1001}.The root is a particular Fock basis state from which all the other Fock states that carry non-zero weight in the Laughlin wave function can be obtained via "squeezing" [112]: a two-particle process where one particle moves from orbital m 1 to m ′ 1 and another from m 2 to m ′ 2 , where m 1 <m ′ 1 ≤m ′ 2 <m 2 , and The root partition encodes the following clustering property, unique to the Laughlin 1/3 state: no more than a single electron can exist in three consecutive orbitals.The collective mode in the limit of momentum k→0 consists of two quasielectron-quasihole pairs, forming a quadrupole.As the momentum increases, a dipole moment develops with the separation of one quasielectronquasihole pair.This idea can be translated into an explicit set of root partitions for different states comprising the magnetoroton mode: where the filled circle indicates the position of a quasielectron, while the empty circle denotes the quasihole.The configurations above are labeled by their total angular momentum L on the sphere, and the smallest momentum that can be created is L=2.
The wave functions for the magnetoroton states |ψ L λ ⟩ are ultimately determined by imposing the following conditions [95] Here, sl µ denotes a Slater determinant with an orbital occupation pattern given by the partition µ.The first condition thus means that we are looking for states that are obtained by squeezing from one of the roots λ in Eq. (S3), which introduces a partial ordering among the partitions, denoted by ⪯.The second condition states that we are FIG.S1.Variational energies for the Jack polynomial approximations to the collective mode states from Ref. [95] for the bosonic ν b =1/2 state.For δV0̸ =0 cases, we have rescaled the energies with |δV0| for clarity.
only interested in rotationally invariant states.These two conditions uniquely fix the coefficients a µ of the Laughlin ground state wave function.For the excited magnetoroton states, to find a µ we require the third, energetic condition, involving the V1 pseudopotential operator and electron annihilation operators ĉm , which singles out the state with quasiparticles piled up at the north pole of the sphere.The resulting states are confirmed to be excellent approximations to the eigenstates of Coulomb interaction [95].Moreover, the construction can be readily generalized to other types of states that have a Jack polynomial description and a parent Hamiltonian, e.g., the Read-Rezayi sequence [113].
The conditions above can be straightforwardly modified to yield the magnetoroton states of the bosonic Laughlin state at ν b =1/2.The clustering condition now stipulates that there should be no more than one boson in every two orbitals.The first condition in Eq. (S4) should then be viewed as an expansion over permanents (rather than Slater determinants), while the last condition needs to be replaced with V0 pseudopotential.Finally, the root configurations need to be divided with a Jastrow factor, resulting in bosonic roots such as 2000101010• • •10101 (for L=2), 2001001010• • •10101 (for L=3), etc.With these modifications, the construction of bosonic magnetoroton states is fully analogous to the fermionic case.
We evaluate the variational energy of the Jack wave functions in Fig. S1 for three values of δV 0 .This shows, within the variational approximation, that the collective mode becomes gapless in the k→0 limit, supporting the results of exact numerics in the main text.
It turns out that the elaborate construction presented above, in special cases L=2, 3, reduces to the much simpler single-mode approximation (SMA) ansatz written down by Girvin, MacDonald, and Platzman [41].On the sphere, the SMA ansatz for a state with total angular momentum L is obtained by acting on the ground state with the projected density operator ρL,M [114] on the ground state denoting the Clebsch-Gordon coefficient and, due to the rotational invariance, we can set M =L.
At L=1, the density operator in Eq. (S7) reduces to the angular momentum raising operator [114], hence it annihilates the ground state, which is a singlet (L=0).Thus, the L=1 SMA trial state identically vanishes, which is also the case for the Jack construction.Moreover, at L=2, 3, the SMA wave functions only involve the elements of the basis squeezed from the same root partition that defines the Jack model wave functions in Eq. (S3).Hence, for the Laughlin 1/3 state, the equivalence between the two sets of wave functions can be proven for L=2, 3 [95].Moreover, both of these states are also equivalent to the CF exciton states discussed below.For L>3 the SMA wave functions contain unsqueezed basis components with respect to the root partitions used in the Jack model wave functions, thus the final states are no longer equivalent.The accuracy of the SMA ansatz in Eq. (S7), however, is numerically found to deteriorate rapidly at larger values of L [115], unlike the Jack wave functions which remain accurate over the entire range 2≤L≤N .

LLL-projected GMP wave function
The original SMA ansatz, Eq. (S7), is not convenient for evaluation in large systems.In this subsection, we present an equivalent real-space form of the GMP ansatz that we have used to calculate the L=2 gap.We consider general ν=1/m Laughlin states, which can be either bosonic for even m or fermionic for odd m.
In first quantization, the GMP wave function [Eq.(S7)] on the sphere takes the form [41] Ψ GMP (L) = where Y L,M (u i , v i ) are the spherical harmonics.Due to rotational invariance, we can set M =−L for simplicity, thereby To implement the LLL projection exactly and efficiently, we apply the technique developed in Ref. [68] to obtain In the last line, we defined a new function P i (L, m) This newly defined function can be evaluated through the recursion relations (S11) Here, the function h i (L) is defined as (S12) It has the following property under the action of a derivative For the bosonic Laughlin state with m=2, we calculate the energy for the V 0 interaction, which requires setting Ω 1 =Ω 2 in the wave function.In this special case, the GMP wave function can be written exactly as (S15) Here, the function hi (L), for i=1, 2, is defined as (S16)

Composite fermion exciton wave function
The bosonic Laughlin state occurs when the effective flux seen by electrons is 2Q * =N −1.We will keep using the language of composite fermions even though the composite object of an electron and a single vortex here is a boson.These states can be viewed as (CF state) × (ν=1 integer quantum Hall state).
The exact form of the ν b =1/2 composite fermion exciton wave function for total orbital angular momentum L and L z =M is given as where C L,M Q * ,m1;Q * +1,M −m1 is the Clebsch-Gordan coefficient that comes about from adding the angular momentum of the constituent CF hole (−m 1 ) and CF particle (m 2 =M −m 1 ).The wave function Ψ is given as where Φ hole m1 is the (N −1)×(N −1) Slater determinant state that describes a hole at ν=1 which is obtained by leaving the −m 1 orbital unoccupied and Ỹq,m (Ω k ) is given as where N q,m is a normalization constant given by CF exciton wave function for Ω1=Ω2 The exact form of the ν b =1/2 CF exciton wave function for total orbital angular momentum L and L z =M when coordinates of two of the bosons are identically set equal to each other, which is required to evaluate its energy for the δ-function interaction [116], is given as is the Clebsch-Gordan coefficient that comes about from adding the angular momentum of the constituent CF particle (m 1 ) and CF hole (−m 2 =M −m 1 ) [Note, the CF hole has angular momentum of opposite sign so that m 1 −m 2 =M .].The wave function Ψ is given as where Φ hole m1 is the (N −1)×(N −1) Slater determinant state that describes a hole at ν=1 which is obtained by leaving the M −m 1 orbital unoccupied and Ỹq,m (Ω) is given as GEOMETRIC QUENCH DYNAMICS The spectral properties discussed in the main text can be conveniently probed by quenching the FQH sys-tem out of equilibrium.We consider a particular type of "geometric" quench where the isotropic FQH state evolves under unitary dynamics generated by a weakly anisotropic Hamiltonian [117,118].Anisotropy induces quadrupolar deformations of the FQH fluid, which breaks the full rotational invariance on the sphere and couples the FQH ground state with the (isotropic) L=2 state of the GMP mode.In the spherical geometry, quadrupolar anisotropy for bosons can be conveniently modeled by adding a small amount of the anisotropic pseudopotential V 0,2 , introduced in Ref. [119].In Fig. S2(a) we present the geometric quench dynamics for bosons at ν b =1/2 for three values of the (isotropic) short-range pseudopotential δV 0 =0, −0.3 and −0.4,approaching the FQHN transition.For all three points, we induce the dynamics by adding a fixed amount of V 0,2 =0.01.The dynamics is characterized by the fidelity, F (t)=|⟨ψ(t)|ψ(0)⟩| 2 , where ψ(0) is the initial (isotropic) FQH ground state and |ψ(t)⟩= exp(−itH ′ )|ψ(0)⟩ is the state after time t (we set ℏ=1).The prime on the Hamiltonian indicates that the interaction includes not only the modified V 0 pseudopotential but also the anisotropic component V 0,2 .In all three cases, the dynamics are approximately harmonic and can be viewed as a two-level system.At the FQHN critical point itself, the harmonic dynamics are a finite-size effect that occurs due to the discreteness of the spectrum at fixed system size N .
The fidelity in Fig. S2(a) oscillates with an increasing period as the FQHN point is approached.To quantify this behavior, in Fig. S2(b) we perform a Fourier transform on this data and plot the extracted frequency as a function of δV 0 .For reference, we also include the extrapolated neutral gap at N →∞ from the main text.At small values of δV 0 , i.e., deep in the Laughlin phase, the oscillation frequency is significantly higher than the neutral gap.This is because the neutral gap is determined by the lowest energy excitation, while the L=2 GMP state is higher in energy at the pure Coulomb point.As V 0 is reduced, the L=2 state becomes the lowest excited state around δV 0 ≈−0.35,corresponding to the kink in the neutral gap curve.For δV 0 ≳−0.35, the oscillation frequency is in good agreement with the neutral gap, with the small discrepancy between the two attributed to finite-size extrapolation.

DISK GEOMETRY
In the main text, we have focused on closed geometries, sphere and torus, which are most convenient for studying the system's bulk properties and their extrapolation to the thermodynamic limit.In particular, we showed that our main conclusions about the existence of the FQHN critical point are largely insensitive to the choice of boundary condition.Hence, for sufficiently large systems, we expect these conclusions to hold also on a finite disk, which describes more closely the real physical system that necessarily has a boundary.
In Fig. S3 we confirm that the FQHN critical point is also realized on a finite disk in a similar parameter regime δV 0 =−0.4.The FQH Hamiltonian on a disk conserves the total angular momentum L z about the perpendicular z-axis and the bosonic Laughlin state at ν b =1/2 occurs in the sector L 0 z =N (N −1).However, block-diagonalization of the Hamiltonian in L z -invariant sectors is insufficient to reveal the neutral spectrum [120]; instead, it is necessary to consider the approximately conserved center-ofmass angular momentum MCOM : where Ri is the guiding center coordinate of the ith particle.This operator can be expressed in second quantization with the help of Clebsch-Gordan coefficients on the disk, which can be found, e.g., in Ref. [121].For sufficiently small values of L z , the spectrum of MCOM is quantized in terms of integers [120].The collective mode of a FQH state can be resolved by projecting the Hamiltonian into a subspace of states with eigenvalue 0 of MCOM and then plotting the energies as a function of ∆L≡L 0 z −L z .To make the numerics more tractable, as customary in the literature, we also assume a cutoff 2N −1 for the total number of orbitals accessible to the particles.We note that, due to the additional step of finding the kernel of MCOM , the diagonalization scheme on the disk geometry is significantly less efficient compared to the sphere or torus.
The presence of an edge can severely impact the bulk properties of small systems.Nevertheless, Fig. S3 shows that this is not the case here, and we can resolve the collective mode despite a fairly small number of particles.The softening of the Coulomb interaction by reducing V 0 FIG.S3.Energy spectrum for bosons at ν b =1/2 on the disk for pure Coulomb interaction (red circles) and softened Coulomb interaction with δV0=−0.4(blue diamonds).Data is for N =7 bosons in 13 orbitals with the center-of-mass momentum equal to zero.The spectrum is plotted relative to the momentum of the Laughlin ground state (see text).Similar to the sphere and torus, the softening of V0 pseudopotential makes the magnetoroton mode (dashed line) go soft in the long-wavelength limit (dotted line).The lines are a guide to the eye.
has a similar effect as on the sphere and torus geometries: it makes the magnetoroton mode gapless in the longwavelength limit, resulting in a nematic instability.In principle, by studying the momentum sectors L z >L 0 z , the effect of nematic fluctuations on FQH edge states could also be explored in this setup, which we leave for future work.

COMPUTATION OF THE HALL VISCOSITY
To compute the Hall viscosity, we consider a torus defined by two sides L 1 and L 2 =L 1 τ in the complex plane, where τ =τ 1 +iτ 2 is the modular parameter.We assume L 1 to be real and τ 2 >0.The total area is A=L 2 1 τ 2 .Ref. [74] has shown that the antisymmetric part of viscosity for an FQH fluid, which was named the Hall viscosity η A , can be derived from the Berry curvature F τ1,τ2 in τ space as where and |Ψ⟩ is the incompressible FQH ground state in the torus geometry.The Berry curvature for a single particle in the nth Landau level (with n=0 for LLL) is [122] The Hall viscosity for an integer number of n filled LLs can then be straightforwardly derived by adding the contributions of all single particles (which is valid for a Slater determinant state) to give For FQH states, the Hall viscosity can be computed from the overlaps of the ground states with deformations of the geometry, i.e., the values of τ .Hall viscosity is expected to be quantized within a FQH phase as [77] η where S is the so-called "shift", i.e., the offset of flux quanta needed to form a ground state on a sphere: where N is the electron number, N ϕ is the flux quanta number, and ν is the filling factor.The relation Eq. (S29) has been confirmed through various calculations [76,[123][124][125].
The Berry curvature defined in Eq. (S26) can be calculated if the exact ground state is known.To simplify the computation, we first obtain the ground states at four different points τ =(τ 1 ±δ 1 , τ 2 ±δ 2 ) using exact diagonalization.We refer to these four points τ (m=1, 2, 3, 4) in counter-clockwise order around τ .Suppose the exact ground state eigenvectors are |u m ⟩, m=1, 2, 3, 4, respectively.The Berry curvature can be evaluated as This is essentially a Wilson loop calculation of the Berry phase, with the advantage that we can avoid aligning the gauge choice at different τ .The second term in Eq. (S31) accounts for the contribution from the change of the Slater determinant basis with a variation of τ , which is a constant and equivalent to the Berry curvature of the filled LLL, directly following from Eq. (S27).
In the calculation presented in the main text, we chose τ =i, δ 1 =0.001π and δ 2 =0.001 (thus, the variation of angle is 0.001 in units of π).

CHARGE DENSITY WAVE
In the main text, we have stated, without proof, that a CDW phase is expected to occur when the interaction becomes nearly hollow core.Consequently, a melted, translationally-invariant, form of such a CDW could explain the region of our phase diagram that was identified with the FQH nematic.Here we investigate in more In all the cases, we chose the ground state in momentum sector (kx=0, ky=0).The distribution of momenta, belonging to the ground state multiplet, across the Brillouin zone and the profile of S kx,ky suggest that the system realizes a stripe at N =7, while it is closer to a bubble phase at N =8, 9, 10.
detail the signatures of a CDW phase on the torus geometry, fixing the rectangular unit cell and varying its aspect ratio.It has been shown that a careful analysis of the energy spectrum and properties of the ground state under such deformations of the torus can be used to identify different types of CDW ordering, including stripe phases at half filling of higher LLs [126], electron bubble phases [127], and Wigner crystals at low-filling factors [128], which can be viewed as a special case of one-electron bubbles.In Fig. S4 we perform a similar analysis for the present case of bosons at ν b =1/2 with the Coulomb interaction softened by δV 0 =−0.5.This value of δV 0 is sufficiently large to push the system across the nematic critical point.
The low-lying energy spectra on the torus, Fig. S4(a)-(d), display significant differences across the studied system sizes N ≤10.Nevertheless, as the torus aspect ratio is varied in the vicinity of the square unit cell, in all system sizes there appears a quasi-degenerate ground state manifold.The observed degeneracy is much larger than the ones found in incompressible FQH states [129].Instead, the momenta of degenerate states form a one-or two-dimensional lattice in reciprocal space.The quasidegeneracy can be seen clearly in Figs.S4(e)-(h), which show the momentum-resolved spectra for a fixed aspect ratio that is approximately optimal for a CDW in a given finite-size system, as seen in in Figs.S4(a)-(d).For example, for N =7 bosons, we took the aspect ratio r=2 The gaps as a function of 1/N for δV1=−0.095for the fermionic ν=1/3 state.The finite-size effects are stronger in this case compared to bosons in the main text, as can be seen from the difference of the extrapolated charge and L=N Nevertheless, the lower value of these gaps is still by a factor of 3 larger than the neutral gap, which extrapolates close to zero.and plotted the spectrum in Fig. S4(e).The momenta of the four lowest-lying states in Fig. S4(e) form a onedimensional array, similar to Ref. [126].Thus, we identify this ground state as a stripe.This identification is further corroborated by evaluating the guiding-center structure factor S kx,ky [58], which is shown in Fig. S4(i) for one of the quasi-degenerate ground states with momentum (k x =0, k y =0).The structure factor exhibits sharp peaks that are colinear and separated by a multiple of a fundamental vector k * , indicating CDW ordering along one spatial direction.
In larger system sizes, N =8, 9 in Figs.S4(f)-(g), the ground state degeneracy is larger than expected for a stripe.For example, at N =8 and aspect ratio r=1.5, the number of quasi-degenerate ground states in Fig. S4(f) is 16 and they form a two-dimensional lattice of 2-electron bubbles [127].The corresponding structure factor of the FIG.S7.Variational energies for the collective mode states for fermions at ν=1/3, approximated by Jack polynomials from Ref. [95].Data is for system sizes N = 6−12 and three values of δV1 specified in the legend.We observe similar softening of the collective mode in the long-wavelength limit like in the bosonic case in the main text.
ground state in (k x =0, k y =0) momentum sector is shown in Fig. S4(j).The structure factor also exhibits sharp peaks but now they occur in both spatial directions.The overall shape of S kx,ky is similar to the examples of bubble phases in the third LL given in Ref. [127].
Note that system size N =10, shown in Fig. S4(h), differs somewhat from N =8, 9 by a visibly smaller groundstate degeneracy.At the same time, the structure factor shown in Fig. S4(l) is broadly similar to those at N =8, 9. Thus, it is possible that N =10 also realizes a bubble phase, but the splitting between the ground states is much larger than in N =8, 9.However, it is not clear at present why the splitting should be larger in this case.
In summary, Fig. S4 demonstrates that bosons at ν b =1/2 display signatures of CDW ordering once δV 0 is sufficiently reduced to cross the nematic critical point.While the nature of the CDW is hard to pinpoint due to strong finite-size fluctuations, all studied systems display large ground state quasi-degeneracy and sharp peaks in the structure factor, consistent with a CDW.

FERMIONIC LAUGHLIN STATE
In this section, we demonstrate that our results for bosons at ν b =1/2 in the main text are qualitatively similar to results for fermions at ν=1/3, which is more experimentally relevant for solid-state materials.As will become apparent from the results below, the main difference is the stronger finite-size effects for fermions.This can be understood from the CF picture: the effective magnetic length for CFs is ℓ * B = √ 2ℓ B at ν b =1/2 while it is ℓ * B = √ 3ℓ B at ν=1/3.Thus, the quasiparticles have larger sizes in the 1/3 state compared to 1/2, requiring FIG.S8.The shift extracted from the Hall viscosity η A for fermions at ν=1/3 on the torus in the vicinity of the square aspect ratio.In the gapped FQH phase, the shift is quantized to the value expected in the Laughlin state, S=3, similar to the bosonic case in the main text.As the FQHN critical point is approached, large fluctuations in S appear and its value is no longer quantized.larger systems to minimize the finite-size effects.
In Fig. S5 we show the exact spectrum of the LLLprojected Coulomb interaction for fermions at ν=1/3 as the interaction is softened by δV 1 =−0.09.By contrast, the inset to Fig. S5 shows the spectrum of the pure Coulomb interaction.Compared to the bosonic spectra in the main text the key difference is a more pronounced magnetoroton minimum for the case of pure Coulomb interaction in the fermionic case.Nevertheless, the softening of the interaction similarly leads to the collective mode going soft in the k→0 limit, though stronger finitesize effects are present in the fermionic case as seen in the spreading of the data points in Fig. S5.
The behavior of the extrapolated charge and neutral gaps for fermions at ν=1/3 is shown in Fig. S6(a)-(b), respectively.The qualitative behavior of the gaps is similar to the bosonic case in the main text.The neutral gap closes at about δV 1 ≈−0.095.At the same point, the charge gap remains open and hovers around 0.02E C upon further reduction of V 1 .Once again, the stronger finitesize effects compared to the bosonic case are also present in the gap extrapolations in Fig. S6(b).In particular, the charge gap and L=N gap do not extrapolate to the same value as they do for the bosonic case in the main text.Nevertheless, both gaps remain by at least a factor of 3 larger than the extrapolated neutral gap at the FQHN transition (δV 1 =−0.095).
Finally, the behavior of the Hall viscosity for fermions at ν=1/3 in Fig. S8 largely mirrors that of the bosonic case.Here the stronger finite-size effect is particularly clear, even deep within the Laughlin phase (δV 1 ≥−0.1).In fact, even at δV 1 =0, the extracted shift shows a visible deviation from the Laughlin S=3 in smaller system sizes N ≤9, for which it was essentially converged in the bosonic case.

FIG. 2 .
FIG. 2. (a)-(b): Energy spectrum for bosons at ν b =1/2 in the LLL, with pure Coulomb potential (a) and softened by adding δV0=−0.4pseudopotential (b), which pushes the system close to a critical point.Data is for system sizes N =6−12.All energies include the rescaling of the magnetic length, see SM [63].(c) Charge and neutral gaps (see text for the definition) of the bosonic ν b =1/2 state as a function of δV0.(d) The scaling of the gaps as a function of 1/N for the fixed value δV0=−0.43.

FIG. 3 .
FIG. 3. Variational estimate of the neutral gap (i.e., the gap of the L=2 GMP state) and charge gap (i.e., the gap of L=N CF exciton state) as a function of δV0 for bosons at ν b =1/2.The inset shows the finite-size extrapolation of the Coulomb and V0 gaps of L=2 and L=N trial states.

FIG. 4 .
FIG. 4.(a) Energy of the wave functions in Eq. (1) for various δV0.We discretize the α range in steps of 0.02ℓB.Inset shows the value of α which minimizes the total energy at each δV0.The Coulomb energy and V0=1 energy are calculated respectively for N =15, 20, 25, 30 and the thermodynamic values are obtained from finite-size extrapolation.The red dashed line is the fitting of the data with the form of κ δV critical 0

FIG. S2 .
FIG. S2.Geometric quench dynamics for bosons at ν b =1/2 in the vicinity of the FQHN critical point.(a) The fidelity for the FQH ground state for three values of the added δV0 pseudopotential (indicated in the legend).All data is for a fixed system size N =9 and weak anisotropy V0,2=0.01.(b)The frequency of the oscillations in panel (a), extracted via Fourier transform and plotted as a function of δV0.For comparison, the line shows the neutral gap in the thermodynamic limit from the main text.

FIG
FIG. S4. (a)-(d):Energy spectra for bosons at filling factor ν b =1/2 as a function of torus aspect ratio (for a rectangular unit cell).The interaction is Coulomb with δV0=−0.5, i.e., past the nematic critical point.Large quasi-degeneracies are visible in the ground-state manifold, which is consistent with different types of CDW (see text).(e)-(h): Momentum-resolved energy spectrum at illustrative aspect ratios r=1.5 (for N =8, 9, 10) and r=2 (N =7).The spectrum is plotted relative to the ground state in each case.(i)-(l): Guiding center structure factor S kx,ky for one of the ground states in panels (e)-(h).In all the cases, we chose the ground state in momentum sector (kx=0, ky=0).The distribution of momenta, belonging to the ground state multiplet, across the Brillouin zone and the profile of S kx,ky suggest that the system realizes a stripe at N =7, while it is closer to a bubble phase at N =8, 9, 10.
FIG.S5.Energy spectrum for fermions at ν=1/3 in the LLL interacting via Coulomb interaction with added δV1=−0.09pseudopotential, which pushes the system to the vicinity of the critical point.Data is for system sizes N =6−12.Inset shows the spectrum for the pure Coulomb potential.