Bias-variance decomposition of overparameterized regression with random linear features

In classical statistics, the bias-variance trade-off describes how varying a model's complexity (e.g., number of fit parameters) affects its ability to make accurate predictions. According to this trade-off, optimal performance is achieved when a model is expressive enough to capture trends in the data, yet not so complex that it overfits idiosyncratic features of the training data. Recently, it has become clear that this classic understanding of the bias variance must be fundamentally revisited in light of the incredible predictive performance of overparameterized models—models that avoid overfitting even when the number of fit parameters is large enough to perfectly fit the training data. Here, we present results for one of the simplest examples of an overparameterized model: regression with random linear features (i.e., a two-layer neural network with a linear activation function). Using the zero-temperature cavity method, we derive analytic expressions for the training error, test error, bias, and variance. We show that the linear random features model exhibits three phase transitions: two different transitions to an interpolation regime where the training error is zero, along with an additional transition between regimes with large bias and minimal bias. Using random matrix theory, we show how each transition arises due to small nonzero eigenvalues in the Hessian matrix. Finally, we compare and contrast the phase diagram of the random linear features model to the random nonlinear features model and ordinary regression, highlighting the additional phase transitions that result from the use of linear basis functions.

Figure 1

Random linear features model (two-layer linear neural network). Analytic solutions plotted as a function of ${\alpha }_p=N_p/M$ with fixed ${\alpha }_f=N_f/M$ for (a) less input features than training data points (${\alpha }_f=1/2$) and (b) more input features than training data points (${\alpha }_f=4$). Shown are the ensemble-averaged training error (blue squares), test error (black circles), squared bias (green triangles), and variance (red diamonds). Analytic solutions are indicated as dashed lines with numerical results shown as points. In (a), a black dashed vertical line marks the boundary between the large bias and minimal bias underparameterized regimes at ${\alpha }_p={\alpha }_f$, while in (b), a similar line marks the boundary between the under- and overparameterized regimes at ${\alpha }_p=1$. Analytic solutions as a function of ${\alpha }_p$ and ${\alpha }_f$ are also shown for the the ensemble-averaged (c) training error, (d) test error, (e) squared bias, and (f) variance. Results are shown for a linear teacher model $y\left(\vec{\mathbf{x}}\right)=\vec{\mathbf{x}}·\vec{\mathit{\beta }}+\varepsilon$, a signal-to-noise ratio of ${\sigma }_{\beta }^2{\sigma }_X^2/{\sigma }_{\varepsilon }^2=10$, and have been scaled by the variance of the training set labels ${\sigma }_y^2={\sigma }_{\beta }^2{\sigma }_X^2+{\varepsilon }^2$. In each panel, black dashed lines indicate boundaries between different regimes of the solutions depending on which is the smallest of the quantities $M, N_f$, or $N_p$. See Appendix for additional numerical details.

Figure 2

Susceptibilities and eigenvalue spectra. (a), (b) Analytic solutions for the susceptibility $\nu$ which measures the sensitivity of the fit parameters to small perturbations in the gradient. In the small $\lambda$ limit, $\nu \approx {\lambda }^{-1}{\nu }_{-1}+{\nu }_0$. (a) The coefficient ${\nu }_{-1}$ counts the fraction of unutilized fit parameters, or the fraction of parameters beyond that needed to attain minimal training error. (b) The coefficient ${\nu }_0$ diverges at each phase transition when $Z^{T}Z$ has a small eigenvalue. (c) Analytic solution for the minimum nonzero eigenvalue ${\sigma }_{min}^2$ of the Hessian matrix $Z^{T}Z$. (i)–(ix) Examples of the full eigenvalue spectrum are shown for each of the corresponding points in (c). See Appendix for additional simulation details.

Figure 3

Comparison of the random linear features model to linear regression (no basis functions) and the random nonlinear features model. (a) The training error, test error, bias, and variance as a function of ${\alpha }_p$ (or equivalently, ${\alpha }_f$) for linear regression (no basis functions) and (b) the same quantities for the random linear features model for the special case ${\alpha }_f={\alpha }_p$. The training error, test error, bias, and variance as a function of ${\alpha }_p$ and ${\alpha }_f$ for (c)–(f) the random nonlinear features model with ReLU activation, $\phi \left(h\right)=max(0,h)$, and (g)–(j) the random linear features model. In all panels, black dashed lines indicate phase transitions. Results for linear regression and the random nonlinear features model are reproduced from Ref. [35].

Figure 4

Comparison of analytic and numerical results for the random linear features model: Training error and bias-variance decomposition. Analytic solutions (top row) and numerical results (bottom row) are shown as a function of ${\alpha }_p=N_p/M$ and ${\alpha }_f=N_f/M$. Plotted are the ensemble-averaged (a) training error (b), test error (c), squared bias, and (d) variance. In each panel, black dashed lines show boundaries between different regimes of solutions, depending on which is smallest of the quantities $M, N_f$, or $N_p$. The vertical and horizontal lines bound the interpolation, or overparameterized, regime, located at ${\alpha }_p>1$ and ${\alpha }_f>1$, while the diagonal line marks the boundary between the large bias and minimal bias underparameterized regimes for a linear teacher model. All solutions have been scaled by the variance of the training set labels ${\sigma }_y^2={\sigma }_{\beta }^2{\sigma }_X^2+{\sigma }_{\varepsilon }^2$.