Expectation propagation on the diluted Bayesian classifier

Alfredo Braunstein, Thomas Gueudré, Andrea Pagnani, and Mirko Pieropan
Phys. Rev. E 103, 043301 – Published 2 April 2021

Abstract

Efficient feature selection from high-dimensional datasets is a very important challenge in many data-driven fields of science and engineering. We introduce a statistical mechanics inspired strategy that addresses the problem of sparse feature selection in the context of binary classification by leveraging a computational scheme known as expectation propagation (EP). The algorithm is used in order to train a continuous-weights perceptron learning a classification rule from a set of (possibly partly mislabeled) examples provided by a teacher perceptron with diluted continuous weights. We test the method in the Bayes optimal setting under a variety of conditions and compare it to other state-of-the-art algorithms based on message passing and on expectation maximization approximate inference schemes. Overall, our simulations show that EP is a robust and competitive algorithm in terms of variable selection properties, estimation accuracy, and computational complexity, especially when the student perceptron is trained from correlated patterns that prevent other iterative methods from converging. Furthermore, our numerical tests demonstrate that the algorithm is capable of learning online the unknown values of prior parameters, such as the dilution level of the weights of the teacher perceptron and the fraction of mislabeled examples, quite accurately. This is achieved by means of a simple maximum likelihood strategy that consists in minimizing the free energy associated with the EP algorithm.

  • Figure
  • Figure
  • Figure
  • Figure
  • Figure
  • Figure
  • Received 29 September 2020
  • Accepted 15 February 2021

DOI:https://doi.org/10.1103/PhysRevE.103.043301

©2021 American Physical Society

Physics Subject Headings (PhySH)

Statistical Physics & Thermodynamics

Authors & Affiliations

Alfredo Braunstein1,2,3, Thomas Gueudré1,*, Andrea Pagnani1,2,3, and Mirko Pieropan1,†

  • 1Department of Applied Science and Technologies, Politecnico di Torino, Corso Duca Degli Abruzzi 24, Torino, Italy
  • 2Italian Institute for Genomic Medicine, IRCCS Candiolo, SP-142, I-10060 Candiolo, Italy
  • 3Istituto Nazionale di Fisica Nucleare Sezione di Torino, Via Pietro Giuria 1, I-10125 Torino, Italy

  • *Currently at Amazon Alexa, Torino.
  • Corresponding author: mirko.pieropan@polito.it

Article Text (Subscription Required)

Click to Expand

References (Subscription Required)

Click to Expand
Issue

Vol. 103, Iss. 4 — April 2021

Reuse & Permissions
Access Options
Author publication services for translation and copyediting assistance advertisement

Authorization Required


×
×

Images

×

Sign up to receive regular email alerts from Physical Review E

Log In

Cancel
×

Search


Article Lookup

Paste a citation or DOI

Enter a citation
×