Statistical properties of large data sets with linear latent features

Analytical understanding of how low-dimensional latent features reveal themselves in large-dimensional data is still lacking. We study this by defining a probabilistic linear latent features model with additive noise and by analytically and numerically computing the statistical distributions of pairwise correlations and eigenvalues of the data correlation matrix. This allows us to resolve the latent feature structure across a wide range of data regimes set by the number of recorded variables, observations, latent features, and the signal-to-noise ratio. We find a characteristic imprint of latent features in the distribution of correlations and eigenvalues and provide an analytic estimate for the boundary between signal and noise, even in the absence of a spectral gap.

Figure 1

Distribution of pairwise correlations in the pure signal limit $\mathrm{SNR}\to \infty$, for $m=2,4$, and 30 latent features. Orange: analytical form (a symmetric $\mathrm{Beta}$ distribution). Gray: simulated data. Each simulation has $N=80$ variables and $T=2048$ observations and constitutes 1000 independent model realizations.

Figure 2

Eigenvalue density and bounds as a function of $\mathrm{SNR}$ in the classical statistics and the intensive limit. (a) Classical statistics limit ($q=0.01$, $q_{V^T}=0.9$) at three levels of $\mathrm{SNR}$. Top plot: zero noise analytic density, Eq. (20), (yellow); large noise limit given by the Marčenko-Pastur distribution (black); and intermediate noise semianalytic density (blue) with the numerical simulation (gray) for comparison. Vertical dashed blue line is the approximate boundary between noise (left) and signal (right). Bottom: eigenvalue bounds ${\lambda }_{\pm }^{\mathrm{SNR}}$ as a function of the $\mathrm{SNR}$. True bounds obtained as numerical solution of Eq. (15) (orange) and approximate bounds given by the analytic expression in Eq. (22) (green). The approximate noise region is striped. Horizontal dotted line indicates the $\mathrm{SNR}$ value of the blue density curve, and the dashed line connecting the plots indicates the signal-noise boundary. (b) Intensive limit ($q=0.01$, $q_{V^T}=0.09$) with plots analogous to (a). Top: zero noise density, Eq. (23) (yellow); for intermediate noise (blue), the left bump corresponds to noise, and the right bump corresponds to the latent signal. Bottom: for $\mathrm{SNR}⪆{\mathrm{SNR}}_{\mathrm{split}}$ (horizontal solid black line), the density splits into two bumps. Simulations constitute 360 independent realisations with $N=300$ and ${\sigma }_U^2={\sigma }_V^2=1$.

Figure 3

Comparison between the variances computed from simulated data of the latent feature model for $m=4,5,6,7,8$ (dashed gray lines) and the numerically evaluated limit expression for ${\sigma }_{UV}^2$ in Eq. (A13) as a function of the limit value $Z$ for even $m$ values. We have chosen ${\sigma }_U^2={\sigma }_V^2=1$.

Figure 4

Comparison of simulated data (gray) and the analytical distribution (orange). In the limit of large $m$, the distribution approaches a Gaussian form (blue). Simulated data constitute a single realization of the model with ${\sigma }_U^2={\sigma }_V^2=1$.

Figure 5

Distribution of pairwise correlations in the regimes of finite and small signal-to-noise ratio with $m=2,4$ and 30 latent features. Analytic form (orange) and simulated data (gray). (a) $\mathrm{SNR}=20$ and (b) $\mathrm{SNR}={10}^{-5}$ (large noise limit). Each simulation is run with $N=80$ variables and $T=2048$ observations and constitutes 1000 independent model realizations.

Figure 6

Difference between the eigenvalue density of the NECM spectrum with and without the signal-noise cross terms quantified by $d^{\prime }$. (a) Classical limit (orange), intensive limit (green), and neither of the two limits (blue). (b) Magnified view of $d^{\prime }$ in the limits of interest: classical limit (blue, orange, and green) and intensive limit (red and purple). Eigenvalue densities are computed from 120 realizations of the random matrix model.