Rotational Symmetry Breaking in a Trigonal superconductor Nb-doped Bi2Se3

The search for unconventional superconductivity has been focused on materials with strong spin-orbit coupling and unique crystal lattices. Doped bismuth selenide (Bi$_2$Se$_3$) is a strong candidate given the topological insulator nature of the parent compound and its triangular lattice. The coupling between the physical properties in the superconducting state and its underlying crystal symmetry is a crucial test for unconventional superconductivity. In this paper, we report direct evidence that the superconducting magnetic response couples strongly to the underlying 3-fold crystal symmetry in the recently discovered superconductor with trigonal crystal structure, niobium (Nb)-doped bismuth selenide (Bi$_2$Se$_3$). More importantly, we observed that the magnetic response is greatly enhanced along one preferred direction spontaneously breaking the rotational symmetry. Instead of a simple 3-fold crystalline symmetry, the superconducting hysteresis loop shows dominating 2-fold and 4-fold symmetry. This observation confirms the breaking of the rotational symmetry and indicates the presence of nematic order in the superconducting ground state of Nb-doped Bi$_2$Se$_3$. Further, heat capacity measurements display an exponential decay in superconducting state and suggest that there is no line node in the superconducting gap. These observations provide strong evidence of odd-parity topological superconductivity.

Unconventional superconductors are characterized by superconducting order parameters that are non-invariant under crystal symmetry operations. When the order parameter is single-component, this non-invariance is manifested solely in the phase of the superconducting wavefunction, and can only be detected by phase-sensitive measurements. On the other hand, when the order parameter is multi-component, the magnitude of the superconducting gap can be different along symmetry-related crystallographic directions. The gap anisotropy directly leads to thermodynamic property of the superconducting state that spontaneously breaks the crystal rotational symmetry of the normal state. However, direct thermodynamic signature of rotational symmetry breaking due to superconductivity has not been found in any crystals.
Bismuth selenide (Bi 2 Se 3 ) makes the best material system to explore for unconventional superconductivity. The strong spin-orbit coupling in the triangular lattice has led to the topological insulating ground state in Bi 2 Se 3 [1,2]. Doping with metallic element such as copper (Cu) and strontium (Sr) made it superconducting [3][4][5][6][7][8][9][10][11]. We report here the first direct observation of rotational symmetry breaking in the superconducting property of Nb-doped Bi 2 Se 3 [12], a new member of superconducting doped topological insulators in addition to Cu-and Sr-doped Bi 2 Se 3 . Possible odd-parity pairing symmetries in doped Bi 2 Se 3 , favored by strong spin-orbit interactions, have been theoretically proposed and classified according to the representations of the D 3d point group [5]. Among them, only the odd-parity pairing in the two-dimensional E u representation gives rise to a nematic superconductor with broken rotational symmetry [13]. A recent nuclear magnetic resonance experiment on Cu x Bi 2 Se 3 reveals a twofold in-plane anisotropy in the spin susceptibility of the superconducting state [14], providing spectroscopic evidence for the E u pairing. The pairing symmetries of Nb-and Sr-doped Bi 2 Se 3 remain unknown.
We applied torque magnetometry to map the complete angular dependence of the in-plane magnetic anisotropy in Nb-doped Bi 2 Se 3 . The in-plane magnetization displays the field dependent hysteresis characteristic of a type-II superconductor. The observed hysteresis shows a large twofold anisotropy, which reveals the broken rotational symmetry in Nb x Bi 2 Se 3 . Our work establishes torque magnetometry as a new and powerful method for discovering nematic superconductivity [15,16].
We used torque magnetometry to measure the superconducting hysteresis loop and magnetization of Nbdoped Bi 2 Se 3 . Magnetic torque is given by Here V is the volume of the sample, H is the external magnetic field, and M is the magnetization of the sample, given by the derivative of the free energy with respect to the external field H. Torque magnetometry is thus a thermodynamic probe that measures the free energy in a sample. The torque is measured by mounting the sample standing on its edge in order to keep the external field in the ab-plane (see Fig. 1(A)). We then rotated the cantilever. This measures the in-plane anisotropy of the sample's magnetic properties in the superconducting state. Fig. 1(B) shows the crystal structure of Nb-doped Bi 2 Se 3 looking down the hexagonal axis. As shown in the figure, the external field is in the hexagonal plane. The azimuthal angle, φ, is the angle between the external magnetic field and the x-axis defined along the cantilever arm. Based on the X-ray diffraction pattern of arXiv:1603.04040v1 [cond-mat.supr-con] 13 Mar 2016 this particular sample, we find that φ = 0 • , 60 • , and 120 • corresponds to the in-plane mirror axis of the crystal, as shown in Fig. 1(B).
The samples of Nb-doped Bi 2 Se 3 used in our experiment have superconducting volume close to 100%, as shown by the volume magnetic susceptibility which approaches -1 in the zero-field cooled run (see Fig. 1(C)). This is much higher than that of Cu-doped Bi 2 Se 3 [8,9].
The measured torque shows a strong superconducting signal. Figure 1(D) shows some examples of the magnetic torque from the sample. The torque τ is plotted as a function of external magnetic field at temperate T = 0.3 K. We swept the field up and down from -1 T to 1 T to measure the entire superconducting hysteresis loop. Arrows along the curve show the field sweep direction. The τ − H loop is a signature of the strong flux pinning characteristic of type-II superconductors. The pinned flux lines form a vortex solid, and the flux density inside the superconductor always resists the change of the applied magnetic field. A simple analysis based on the Bean model shows that the hysteresis of the magnetization gives a direct measurement of the superconducting critical current density in the mixed state of type-II superconductors (see supplemental materials).
We note that the superconducting hysteresis loops at the selected angles have a clear angular dependence in φ. The magnitude of the hysteresis loop reaches a maximum at around 60 • and is nearly zero at 30 • and 90 • . The variation of the hysteresis loop size is the first indication of anomalous in-plane symmetry in Nb-doped Bi 2 Se 3 .
To further illustrate the angular dependence of the superconducting hysteresis loop, we mapped the magnitude of the hysteresis loop as a function of φ. To find the magnitude of the hysteresis loop, we took the difference of the sample's magnetic torque from the H-increasing sweep (τ + ) and H-decreasing sweep (τ − ). We denote this magnitude as ∆τ = τ + − τ − . Figure 2(A) shows the absolute value of the hysteresis magnitude in a polar plot where ∆τ is plotted against the azimuthal angle, φ. ∆τ was taken at different values of H. The sample was measured at T = 0.3 K over an angular range of 200 • . We note that reversing the sign of the magnetic field is equivalent to rotating the cantilever 180 • . We thus used the negative field sweep data to complete the 360 • angular dependence. The six-fold symmetry is clearly demonstrated by the observation that ∆τ goes to zero every 60 • . This confirms the H-symmetric nature of the magnetic torque from the sample. Figure 2 The most important observation is the angular symmetry of the hysteresis loop near zero field. Figure 3 displays the angular dependence of the effective magnetization hysteresis, ∆M = M + − M − = ∆τ µ0H at µ 0 H = 0.03 T. The fitting function, ∆M = A 2φ sin(2φ + 30 • ) + A 4φ sin(4φ − 30 • )), shows the symmetry of the hystere-sis loops. The existence of the 2-fold symmetric sin 2φ contribution, as well as the 4-fold symmetric sin 4φ term, suggests the rotational symmetry breaking, i.e., nematic order, in superconducting Nb-doped Bi 2 Se 3 .
Finally, our heat capacity measurement on Nb-doped Bi 2 Se 3 shows a fully gapped bulk superconductivity ( Fig.  4(A)). For the same sample, we measured the heat capacity C at selected T between 0.4 K and 20 K. Fig. 4(A) shows C T in the superconducting state at µ 0 H = 0 T as well as at 0.75 T, right above the closing of the hysteresis loop at base T . We note that above 4 K, the heat capacity C at 0 T is the same as that at 0.75 T, within measurement errors. Therefore, we use the 0.75 T heat capacity curve as the normal state heat capacity C n . From this we determined the phonon contribution following the same practice as the early work on Cu-doped Bi 2 Se 3 [8]: where the electronic heat capacity C n el = γ n T is for the normal state. Subtracting the phonon heat capacity, C ph , we infer the superconducting state electronic heat capacity C el at 0 T, which is plotted as C el T vs. T in Fig. 4(B). The heat capacity at 0 T shows an exponential decay as T drops to base temperature. This is a signature of a fully gapped, nodeless superconducting state.
We note that C el T trace approaches a finite value γ res near the base temperature. This suggests a partial nonsuperconducting volume in the crystal of about γres γn ∼ <20%. This value is consistent with the superconducting volume we determined from the Meissner effect.
The exponential decay, rather than a typical powerlaw dependence, observed in C el − T is indicative of a nodeless superconducting gap. Other heat capacity papers on unconventional superconductors fairly recognize that a high order power-law can be hard to distinguish from an exponential decay at low temperatures [10]. For line nodes, C el T should fall linearly with T , and for point nodes, C el T should go as T 2 [17]. Line nodes, as have been seen in other unconventional superconductors [18][19][20], are clearly ruled out by our data. The absence of line nodes is consistent with odd-parity superconductivity in doped Bi 2 Se 3 [13,21].
Discussion Our study of the magnetic torque demonstrates a symmetric 6-fold vanishing of the superconducting hysteresis. Magnetic torque is sensitive to the magnetic anisotropy of the superconducting signal, and the hysteretic M − H loops arise from the flux pinning of the vortex solids in the superconducting state. Therefore, the superconducting diamagnetic signal, which comes from the vortex solid, prefers to align with the 3 mirror planes of the triangular lattice. The diamagnetic signal is maximum when the magnetic field aligns with these preferred directions. The magnetic torque τ = µ 0 M × H vanishes between these preferred directions when the M vector is exactly collinear (parallel or antiparallel) of the applied field H.
The hysteresis loop is greatly enhanced along one direction. This suggests that the nematic order is a spontaneous symmetry breaking in the superconducting state. As pointed in ref. [13], the nematic order verifies the two-component nature of the superconducting order parameter. Thus an odd-parity superconducting order is very likely to exist in the ground state, which creates promise for topological superconductivity in Nb-doped Bi 2 Se 3 .
We note that early torque studies on high T c cuprates, untwinned YBa 2 Cu 3 O 7 [22] and Tl 2 Ba 2 CuO 6+δ [23], also show in-plane vanishing of the superconducting hysteresis along their crystal axes. A similar measurement on NbN and NbSe 2 , which are classical superconductors with swave symmetry [23], showed no in-plane vanishing of the hysteretic torque curves.
We would like to also point out a potential difference between our observations in Nb-doped Bi 2 Se 3 and the similar early experimental observation [14] and theoretical explanation [13] of nematic order in the spin susceptibility of Cu-doped Bi 2 Se 3 . In the Cu-doped material, quantum oscillations [7,9] and photoemission [6,11] reveal that there is only one bulk Fermi pocket. In contrast, the electronic state in Nb-doped Bi 2 Se 3 shows at least two distinct Fermi surfaces [24]. Our work calls for further theoretical and experiential exploration of the impact of multi-orbitals on the search for unconventional superconductors.
The unique sensitivity of our in-plane torque magnetometry leads to a promising new tool for probing the superconducting pairing symmetry of other unconventional superconductors. It can elucidate or confirm potential p-wave superconducting symmetry in materials such as Sr 2 RuO 4 [25] and UPt 3 [26,27]. In contrast to those materials, the exponential decay in the heat capacity in Nb-doped Bi 2 Se 3 suggests a superconducting gap without line nodes. It would be very interesting to investigate whether a nodal superconducting gap would enhance or diminish the nematic order in the superconducting state.

MATERIALS AND METHODS
We preformed torque magnetometry measurements with our home built cantilever setup by glueing the Nbdoped Bi 2 Se 3 single crystal to the end of a thin beryllium copper cantilever. We then placed the cantilever in an external magnetic field H. We measured the torque on the cantilever by tracking the capacitance between the metallic cantilever and a fixed gold film underneath using an AH2700A capacitance bridge with a 7 kHz drive frequency. We calibrated the spring constant of the cantilever by tracking the angular dependence of capacitance caused by the sample weight at zero magnetic field. The tilt angle φ is defined as the angle between the direction of the magnetic field and the positive x-axis, which is marked in Fig. 1(A) as the direction of the cantilever arm.
The sample heat capacity is measured in a Quantum Design Physical Properties Measurement Systems (PPMS) using the relaxation method.
The National High Magnetic Field Laboratory provided the magnet and He3 fridge. During the torque magnetometry measurement, samples were soaked in pumped liquid helium 3, and the magnetic field was swept at 0.25 T/min.
We   (color online) (A) Sample heat capacity C is displayed as the ratio of C and temperature T plotted against T . The zero field curve is compared with the µ0H = 0.75 T curve. The 0.75 T curve provides us a method to define the phonon contribution C ph . (B) The electronic part of heat capacity C el is shown as C el T vs. T . A clear kink is observed at the superconducting transition temperature Tc. Near T ∼ 0, the curve approaches a finite value in C el /T and gives a measurement of the non-superconducting volume fraction around 20% in this sample. As T increases from the base temperature, C el /T follows the exponential curve(dashed pink line), as expected from a fully gapped superconductor. Numerical calculation of C el in BCS superconductors are shown with the only parameter α ≡ k B Tc ∆ , where ∆ is the superconducting gap. For the overall trace, α = 1.76 trace gives the best fit of the heat capacity trace below Tc.

Superconducting Hysteresis
An external magnetic field kills the superconducting state in a type I superconductor at the critical field, H c . However, in a type II superconductor, there is a mixed state between the lower critical field, H c1 , and the upper critical field, H c2 . For H c1 < H < H c2 , magnetic flux penetrates the superconductor creating a lattice of vortices. Due to defects in the sample, these vortices are pinned in place. In order for the vortices to move, the Lorentz force from a current near the vortices would need to overcome this pinning force. Thus, the pinned magnetic flux has an irreversible response to a changing external magnetic field. This gives rise to hysteresis in the magnetic response of the superconductor.
The Bean model [1,2] successfully explained this hysteretic feature in type II superconductors. In this model, we assume that the current density in the superconductor can only take the values of 0 and J c , where J c is the critical current density. Due to the Ampere's law, the spacial profile of J c determines b(x), the magnetic flux density of unit volume at each location x in the superconductor. Integrating b(x) gives the total magnetic field density B inside the superconductor. Figure 5 shows the magnetic flux density inside the superconductor (represented by the grey shaded region) for (A) field increasing from H = 0 to H 0 and (B) field decreasing to H 0 after an applied external field greater than H c2 . Due to the flux-pinning, the internal magnetic flux density b(x)/µ 0 cannot respond to the change in the sweep direction of the external magnetic field. x. The spacial profile of J c is constant, as shown in the right panel of Fig  5(A).
The situation of field-sweeping-down is different. As the applied field is swept from a large field to the same H 0 , the b(x)/µ 0 profile response lags, as show in Fig.  5(B).
The magnetization of the sample is given by the difference between the average magnetic flux density within the sample, B, and the applied field, H, outside: where w is the typical width of the domain size in the superconductor, or the sample size if the whole sample is in a single domain. In the case of Panel A, Eq. 2 would be simply the shaded area −wJ c which gives M + , the magnetization at field sweeping up. Furthermore, the difference between Panel A and B demonstrates the hysteresis in the magnetization of type II superconductors. Going through the same analysis as above, we find the magnetization of field sweeping down M − = wJ c As a result, the hysteresis loop size ∆M (H) ≡ M + − M − = 2wJ c . Therefore, the measurement of the hysteresis loop is a direct probe of the critical current density in a type II superconductor.

Evolution of the nematic order under magnetic fields
We performed Fourier analysis on ∆M − φ to determine the strength of the nematic order as a function of magnetic field. Figure 6(A) shows the Fast Fourier Transform (FFT) of ∆M at 0.05 T. The first peak, A 2φ , is the amplitude of the nematic order sin(2φ). The second peak, A 4φ , represents the 4-fold term. Because increasing the magnetic field H greatly shrinks the magnetization loop ∆M , both A 2φ and A 4φ dramatically deceases. However, they decrease in different rates. Fig.6(B) shows the ratio of A 2φ over A 4φ . Both A 2φ and A 4φ are leading terms near zero field. When H increases close to the upper critical field, A 2φ dramat-ically vanishes. The vanishing of the nematic order as the superconductor approaches the normal state confirms that it is an intrinsic feature of the superconducting state in Nb-doped Bi 2 Se 3 .
We would like to note that for a broad field range that absolute amplitude of A 2φ and A 4φ are equal. At this condition, the fitting function turns to be f (φ) = 2A 2φ sin(φ − 30 • ) cos 3φ. This fitting reveals a possible origin of the observed rotational symmetry breaking. The effective superconducting magnetic moment follows the product of sin(φ − 30 • ) and cos 3φ, rather than the sum of two ordering functions. Therefore, there is a strong coupling between the 3-fold crystalline symmetry and a nematic ordering in the triangular superconductor Nbdoped Bi 2 Se 3 . The phases between these two sinusoidal functions are locked in the measurement. This observation shows that these two symmetric orderings couple strongly with each other. This suggests that while nematic order is a spontaneous symmetry breaking in the superconducting state, the order seems to be stabilized by the 3-fold crystalline symmetry. The coupling makes sure that the nematic ordering direction is locked to one of the mirror planes of the triangular lattice. As pointed in ref. [13], the nematic order verifies the two-component nature of the superconducting order parameter. Thus an odd-parity superconducting order is very likely to exist in the ground state, which creates promise for topological superconductivity in Nb-doped Bi 2 Se 3 . is from a different cool down in a dilution refrigerator. In both cases, the preferred axis is locked on a crystalline mirror plane. However, in the two cases, the preferred axis is along roughly the same axis. This second cooling data confirms the observation of the spontaneous symmetry breaking in Nb-doped Bi 2 Se 3 .

Rotational Symmetry Breaking in other Nb-doped Bi2Se3 samples
We repeated the search for rotational symmetry breaking in the hysteretic property of Nb-doped Bi 2 Se 3 with a second piece of superconducting Nb-doped Bi 2 Se 3 . This sample is designated sample E. Sample E was cooled down to 20 mK in a dilution refrigerator. We swept the field up and down from -1 T to 1 T to measure the entire superconducting hysteresis loop. As in the main text, we applied torque magnetometry to map the complete angular dependence of its in-plane magnetic anisotropy. M + − M − versus angle φ of sample E. There is a constant background magnetization labeled A 0 . The effective magnetization loop ∆M follows A 0 + A 2φ sin(2φ − α) + A 4φ sin(4φ − β). This angular-independent offset likely arises from the torsional twist of the cantilever setup, although further experiments are needed to determine the exact origin. For further analysis, we subtract away the A 0 term to get the angular dependence of the in-plane effective magnetization. Fig.8(B) shows a polar plot of the effective magnetization versus angle φ of sample E. The superconducting hysteresis loop closes at regular intervals corresponding to the axes normal to the mirror planes of the crystal structure. The rotational symmetry is again broken. This is consistent with the sample from the main text. The consistency of the spontaneous symmetry breaking between different superconducting Nb-doped Bi 2 Se 3 samples suggests this is a intrinsic feature of the system and not the result of geometric anisotropy in any given sample. Fig.8(C) shows a FFT showing the relative strength of the nematic term, sin(2φ), and the crystal symmetry term, sin(4φ). The first peak, A 2φ , is the amplitude of the nematic order sin(2φ). The second peak, A 4φ , represents the 4-fold symmetric term.
The ratio

A 2φ
A 4φ is a measure of the strength of the nematic order over the background crystal symmetry. As seen in Fig.8(D), A 2φ is the leading term near zero field. When H increases close to the upper critical field, A 2φ dramatically vanishes. The vanishing of the nematic order as the superconductor approaches the normal state follows the same trend as the main text sample shown in Fig.6. Normal State Torque signals Nb-doped Bi2Se3 samples Fig.9 shows the effective magnetic susceptibility χ ef f = d(τ /H) dH of Nb-doped Bi 2 Se 3 as a function of angle φ in H = 1.8 T and 1.9 T external magnetic field. The external field in this figure is much larger than Hc2, thus the sample is not in the superconducting state. Therefore we are able to see in the in-plane anisotropy of the magnetic susceptibility in the normal state. The angular dependence of χ ef f follows the characteristic sin 2φ dependence of the paramagnetic normal state. For a paramagnetic material, the magnetic torque τ = µ 0 M × H follows where ∆χ = χ z − χ x is the anisotropy of the magnetic susceptibility of the sample. The observation in Fig. 9 shows that the effective susceptibility follows the sin 2φ dependence and the maximum and minimum are aligned with 135 degree and 45 degree, respectively. This observation is consistent with the normal paramagnetic state. Normal state magnetic susceptibility anisotropy of Nb-doped Bi2Se3 in normal state (color online) Magnetic susceptibility anisotropy of Nb-doped Bi2Se3 as a function of angle φ in external magnetic field above Hc2. The magnetic susceptibility dM/dH follows a sin(2φ) dependence.