Training a perceptron in a discrete weight space

Learning in a perceptron having a discrete weight space, where each weight can take $2L+1$ different values, is examined analytically and numerically. The learning algorithm is based on the training of the continuous perceptron and prediction following the clipped weights. The learning is described by a new set of order parameters, composed of the overlaps between the teacher and the continuous/clipped students. Different scenarios are examined, among them on-line learning with discrete and continuous transfer functions. The generalization error of the clipped weights decays asymptotically as $\exp \left(-K{\alpha }^2\right)$ in the case of on-line learning with binary activation functions and $\exp \left(-e^{|\lambda |\alpha }\right)$ in the case of on-line learning with continuous one, where $\alpha$ is the number of examples divided by N, the size of the input vector and K is a positive constant. For finite N and L, perfect agreement between the discrete student and the teacher is obtained for $\alpha \propto L\sqrt{\ln \mathrm{}\left(\mathrm{NL}\right)\mathrm{}}.$ A crossover to the generalization error $\propto 1/\alpha ,$ characterizing continuous weights with binary output, is obtained for synaptic depth $L>O\left(\sqrt{N}\right).\mathrm{}$