Statistical mechanics of lossy compression for nonmonotonic multilayer perceptrons

A lossy data compression scheme for uniformly biased Boolean messages is investigated via statistical mechanics techniques. We utilize a treelike committee machine (committee tree) and a treelike parity machine (parity tree) whose transfer functions are nonmonotonic. The scheme performance at the infinite code length limit is analyzed using the replica method. Both committee and parity treelike networks are shown to saturate the Shannon bound. The Almeida-Thouless stability of the replica symmetric solution is analyzed, and the tuning of the nonmonotonic transfer function is also discussed.

Figure 1

Rate distortion encoder and decoder.

Figure 2

General architecture of a treelike multilayer perceptron with $N$ input units and $K$ hidden units.

Figure 3

(Color online) AT line and rate-distortion function for the parity tree with three hidden units. The rate-distortion performance of the parity tree is given by the rate-distortion function. The original message is biased with bias $p=0.2$. The rate-distortion function is always above the AT line and thus the RS solution is always stable.

Figure 4

(Color online) AT line and rate-distortion function for the committee tree with three nonmonotonic hidden units. The rate-distortion performance of the committee tree is given by the rate-distortion function. The original message is unbiased $(p=0.5)$. The rate-distortion function is always above the AT line, and thus the RS solution is always stable.

Figure 5

(Color online) AT line and rate-distortion function for the committee tree with three nonmonotonic hidden units. The rate-distortion performance of the committee tree is given by the rate-distortion function. The original message is biased $(p=0.2)$. The rate-distortion function is always above the AT line, and thus the RS solution is always stable.

Figure 6

(Color online) AT line and rate-distortion function for the committee tree with a nonmonotonic output unit. The rate-distortion performance of the committee tree is given by the rate-distortion function. The original message is unbiased $(p=0.5)$. The AT line is calculated using $K=50$. The rate-distortion function is always above the AT line, and thus the RS solution is always stable.

Figure 7

(Color online) AT line and rate-distortion function for the committee tree with a nonmonotonic output unit. The rate-distortion performance of the committee tree is given by the rate-distortion function. The original message is biased $(p=0.2)$. The AT line is calculated using $K=50$. The rate distortion function is always above the AT line except for a very narrow region where $D\approx p$.