Boltzmann machines as two-dimensional tensor networks

Restricted Boltzmann machines (RBMs) and deep Boltzmann machines (DBMs) are important models in machine learning, and recently found numerous applications in quantum many-body physics. We show that there are fundamental connections between them and tensor networks. In particular, we demonstrate that any RBM and DBM can be exactly represented as a two-dimensional tensor network. This representation gives characterizations of the expressive power of RBMs and DBMs using entanglement structures of the tensor networks, and also provides an efficient tensor network contraction algorithm for the computing partition function of RBMs and DBMs. Using numerical experiments, we show that the proposed algorithm is more accurate than the state-of-the-art machine learning methods in estimating the partition function of RBMs and DBMs, and have potential applications in training DBMs for general machine learning tasks.

Figure 1

Mapping an RBM to a two-dimensional tensor network. (a) RBM with four visible neurons (red circles) and three hidden neurons (blue circles). (b) A tensor network representation of RBM; the red nodes and blue nodes are copy tensors corresponding to visible variables and hidden variables, respectively, and the brown squares denote Boltzmann matrices between visible nodes and hidden nodes. (c) The three-dimensional diagram after converting each copy tensor to an MPS. (d) The final two-dimensional tensor network representation of the RBM.

Figure 2

(a) and (b) are schematic diagrams of converting a copy tensor to an MPS. (a) corresponds to a hidden variable, while (b) corresponds to a visible variable which has an open index for representing a variable in $\tilde{\mathcal{P}}$. (c) Converting the three-dimensional structure to a single tensor.

Figure 3

Mapping a DBM with two hidden layers to a two-dimensional tensor network. The symbols have the same meaning as in Fig. 1.

Figure 4

(a) An RBM with four visible neurons and three hidden neurons. (b) A DBM with two hidden layers (blue and green). (c) A DBM with three hidden layers. (d), (e), and (f) are the two-dimensional tensor network representations of (a), (b), and (c), respectively.

Figure 5

Comparison of the running time and the error on the log partition function per visible variable given by our tensor network contraction algorithm (2D TN) and of the AIS algorithm on RBMs and DBMs with random couplings. Each data point is averaged over 100 instances, and the error bar represents one standard derivation. For AIS, we use $100000$ intermediate distributions and 1000 runs.

Figure 6

The relative error of $lnZ$ per visible variable calculated by AIS and two-dimensional tensor network contraction methods for an RBM trained on the Adult dataset.

Figure 7

Illustration of the mapping from an RBM with external fields to a two-dimensional tensor network. (a) The graphical representation of an RBM with four visible neurons (red circles) and three hidden neurons (blue circles). (b) The tensor network representation of the RBM with external fields. Red and blue nodes are copy tensors corresponding to visible and hidden variables, respectively; dark-brown nodes represent their external field vectors; brown squares denote Boltzmann matrices between visible and hidden nodes. (c) The three-dimensional diagram after converting each copy tensor to an MPS. (d) The final two-dimensional tensor network representation of the RBM with external fields.

Figure 8

(a) and (b) are schematic diagrams of converting a copy tensor with an external field vector to an MPS, (a) for hidden variables and (b) for visible variables. (c) Converting the three-dimensional structure to a single tensor.

Figure 9

Illustration of mapping a DBM with three hidden layers to a two-dimensional tensor network. (a) The graphical representation of a DBM with three hidden layers. Red circles are visible variables. Blue, green, and purple circles are hidden variables. (b) The tensor representation of a DBM. All nodes are copy tensors, and brown squares denote Boltzmann matrices between variables of two adjacent layers. Converting each copy tensor to an MPS, we obtain (c), where the color of the MPS corresponds to the color of the tensor in (b) one-to-one. Compress the tensor network of (c) in the vertical direction, and then arrive at figure (d), a two-dimensional tensor network representation. Blue tensors indicate that they are from the first hidden layer, green tensors denote that they are from the second hidden layer, and purple tensors denote that they are from the third hidden layer.

Figure 10

(a) A DBM with one visible layer (red circles) and four hidden layers (blue, green, purple, and orange circles). (b) A DBM with five hidden layers. (c) A DBM with six hidden layers. (d), (e), and (f) are two-dimensional tensor network representations of the DBM in (a), (b), and (c), respectively.

Figure 11

Illustration of the contraction process from a two-dimensional tensor network (mapped from a DBM with four hidden layers) to a scalar representing the partition function of the DBM, using the BMPS method.

Figure 12

Detailed process of merging, canonicalization, and truncation. Merging consists of (1) and (2); canonicalization refers to (3)–(6); truncation means (7)–(12).