Fluctuation-dissipation-type theorem in stochastic linear learning

The fluctuation-dissipation theorem (FDT) is a simple yet powerful consequence of the first-order differential equation governing the dynamics of systems subject simultaneously to dissipative and stochastic forces. The linear learning dynamics, in which the input vector maps to the output vector by a linear matrix whose elements are the subject of learning, has a stochastic version closely mimicking the Langevin dynamics when a full-batch gradient descent scheme is replaced by that of a stochastic gradient descent. We derive a generalized FDT for the stochastic linear learning dynamics and verify its validity among the well-known machine learning data sets such as MNIST, CIFAR-10, and EMNIST.

Figure 1

(a)–(c) Euclidean norm ratio $\left|\right|{\overline{\mathit{\Sigma }}}_{xx}^{\left(n\right)}\left|\right|/\left|\right|{\mathit{\Sigma }}_{xx}\left|\right|$ over several minibatches labeled by $n$ after the equilibrium is reached. (b) The Euclidean norm of the unequal time correlation matrix $\left|\right|\mathbf{D}\left(m\right)\left|\right|/\left|\right|\mathbf{D}\left(0\right)\left|\right|$ [Eq. (3.4)]. The minibatch size used in each figure is $N_m=5$, 500, and 100 for MNIST, CIFAR-10, and EMNIST, respectively.

Figure 2

Fluctuation analysis for (a) MNIST, (b) CIFAR-10, and (c) EMNIST data sets. Top: Plots of $\mathbf{D}$ obtained from each data set. Middle: Plots of ${\mathit{\Sigma }}_{xx}{\mathit{\Sigma }}_{WW}+{\mathit{\Sigma }}_{WW}{\mathit{\Sigma }}_{xx}$. Bottom: Normalized Fourier components for $\mathbf{D}$ (red) and ${\mathit{\Sigma }}_{xx}{\mathit{\Sigma }}_{WW}+{\mathit{\Sigma }}_{WW}{\mathit{\Sigma }}_{xx}$ (blue) plotted along $\mathbf{k}=(k_x,0)$ with $k_0=2\pi /a$.