Properties of the Geometry of Solutions and Capacity of Multilayer Neural Networks with Rectified Linear Unit Activations

Carlo Baldassi, Enrico M. Malatesta, and Riccardo Zecchina
Phys. Rev. Lett. 123, 170602 – Published 23 October 2019
PDFHTMLExport Citation

Abstract

Rectified linear units (ReLUs) have become the main model for the neural units in current deep learning systems. This choice was originally suggested as a way to compensate for the so-called vanishing gradient problem which can undercut stochastic gradient descent learning in networks composed of multiple layers. Here we provide analytical results on the effects of ReLUs on the capacity and on the geometrical landscape of the solution space in two-layer neural networks with either binary or real-valued weights. We study the problem of storing an extensive number of random patterns and find that, quite unexpectedly, the capacity of the network remains finite as the number of neurons in the hidden layer increases, at odds with the case of threshold units in which the capacity diverges. Possibly more important, a large deviation approach allows us to find that the geometrical landscape of the solution space has a peculiar structure: While the majority of solutions are close in distance but still isolated, there exist rare regions of solutions which are much more dense than the similar ones in the case of threshold units. These solutions are robust to perturbations of the weights and can tolerate large perturbations of the inputs. The analytical results are corroborated by numerical findings.

  • Figure
  • Figure
  • Figure
  • Received 17 July 2019

DOI:https://doi.org/10.1103/PhysRevLett.123.170602

© 2019 American Physical Society

Physics Subject Headings (PhySH)

Statistical Physics & ThermodynamicsNetworks

Authors & Affiliations

Carlo Baldassi, Enrico M. Malatesta*, and Riccardo Zecchina

  • Artificial Intelligence Lab, Institute for Data Science and Analytics, Bocconi University, Milano 20135, Italy

  • *enrico.malatesta@unibocconi.it

Article Text (Subscription Required)

Click to Expand

Supplemental Material (Subscription Required)

Click to Expand

References (Subscription Required)

Click to Expand
Issue

Vol. 123, Iss. 17 — 25 October 2019

Reuse & Permissions
Access Options
CHORUS

Article Available via CHORUS

Download Accepted Manuscript
Author publication services for translation and copyediting assistance advertisement

Authorization Required


×
×

Images

×

Sign up to receive regular email alerts from Physical Review Letters

Log In

Cancel
×

Search


Article Lookup

Paste a citation or DOI

Enter a citation
×