Abstract
We study the mutual information between parameter and data for a family of supervised and unsupervised learning tasks. The parameter is a possibly, but not necessarily, high-dimensional vector. We derive exact bounds and asymptotic behaviors for the mutual information as a function of the data size and of some properties of the probability of the data given the parameter. We compare these exact results with the predictions of replica calculations. We briefly discuss the universal properties of the mutual information as a function of data size.
- Received 3 August 1998
DOI:https://doi.org/10.1103/PhysRevE.59.3344
©1999 American Physical Society