Statistical mechanics of lossy compression using multilayer perceptrons

Statistical mechanics is applied to lossy compression using multilayer perceptrons for unbiased Boolean messages. We utilize a treelike committee machine (committee tree) and treelike parity machine (parity tree) whose transfer functions are monotonic. For compression using a committee tree, a lower bound of achievable distortion becomes small as the number of hidden units $K$ increases. However, it cannot reach the Shannon bound even where $K\to \infty$. For a compression using a parity tree with $K\ge 2$ hidden units, the rate distortion function, which is known as the theoretical limit for compression, is derived where the code length becomes infinity.

Figure 1

Rate distortion encoder and decoder.

Figure 2

The architecture of treelike multilayer perceptrons with $N$ input units and $K$ hidden units.

Figure 3

The rate distortion property of lossy compression using a committee tree. The limit of achievable code rate $R$ expected for $N\to \infty$ plotted versus the distortion $D$ for $K=1$ (dotted line), $K=3$ (short dashed line), and $K\to \infty$ (long dashed line). Solid line denotes rate-distortion function $R\left(D\right)$ for binary sequences by Shannon.

Figure 4

The rate distortion property of lossy compression using a parity tree. The limits of achievable code rate $R$ expected for $N\to \infty$ is plotted versus the distortion $D$ for $K=1$ (dashed line) and $K\ge 2$ (solid line). The solid line also denotes the rate-distortion function, which is identical to the limit of achievable distortion for $K\ge 2$. The dash-dotted line denotes the AT line for $K=2$. For $K\ge 3$, the RS solution does not exhibit AT instability throughout the achievable region.

Figure 5

Relationship between codeword and bit of reproduced message in lossy compression using parity tree with $K$ hidden units. $\mathrm{Symbol}+\mathrm{denotes}$ bit of the reproduced message is 1 and $-$ denotes that it is $-1$. Set ${\mathcal{S}}^N$ is divided by $K$ hyperplanes, whose normal vectors are orthogonal each other. For $K\ge 2$, vectors with same distortion as codeword $\mathit{s}$ are distributed throughout ${\mathcal{S}}^N$. (a) A nonmonotonic perceptron, $q=0$, (b) a $K=1$ parity tree, $q>0$, (c) a $K=2$ parity tree, $q=0$, and (d) a $K=3$ parity tree, $q=0$.