Generalization of stochastic-resonance-based threshold networks with Tikhonov regularization

Injecting artificial noise into a feedforward threshold neural network allows it to become trainable by gradient-based methods and also enlarges the parameter space as well as the range of synaptic weights. This configuration constitutes a stochastic-resonance-based threshold neural network, where the noise level can adaptively converge to a nonzero optimal value for finding a local minimum of the loss criterion. We prove theoretically that the injected noise plays the role of a generalized Tikhonov regularizer for training the designed threshold network. Experiments on regression and classification problems demonstrate that the generalization of the stochastic-resonance-based threshold network is improved by the injection of noise. The feasibility of injecting noise into the threshold neural network opens up the potential for adaptive stochastic resonance in machine learning.

Figure 1

(a) Outputs of the designed threshold network and the sigmoid network. For comparison, the noisy observations and the target function $sin\left(2x\right)$ are also illustrated. (b) Learning curves of the loss versus the epoch number on training and validation data sets.

Figure 2

(a) Output of the trained threshold neural network as the approximation (patched surface) to the two-dimensional function in Eq. (7). The validation data ($★$) are also plotted. (b) Learning curves of the loss ${\mathcal{L}}_{\xi }$ on training and validation sets.

Figure 3

Learning curves of the noise level ${\sigma }_{\xi }$ in the stochastic-resonance-based threshold network for approximating unidimensional and two-dimensional functions. The other parameters are the same as in Figs. 1 and 2.

Figure 4

Decision boundary given by the designed threshold network trained by 200 pairs of data points for vertical distance $d=-7$ between regions A and B. Each region is with radius $r=10$ and width $w=6$.