Generation of ice states through deep reinforcement learning

We present a deep reinforcement learning framework where a machine agent is trained to search for a policy to generate a ground state for the square ice model by exploring the physical environment. After training, the agent is capable of proposing a sequence of local moves to achieve the goal. Analysis of the trained policy and the state value function indicates that the ice rule and loop-closing condition are learned without prior knowledge. We test the trained policy as a sampler in the Markov chain Monte Carlo and benchmark against the baseline loop algorithm. This framework can be generalized to other models with topological constraints where generation of constraint-preserving states is difficult.

Figure 1

(a) Local spin configurations that satisfy the two-in–two-out ice rule at each vertex. (b) Flipping a spin (unfilled arrow) in an ice state creates a pair of defects, resulting in a high-energy state. Propagation of a single defect (red circle) by flipping green (bright) spins creates no additional defects. When the loop is closed, the pair of defects (red circle and blue square) annihilates and a new ice configuration is generated.

Figure 2

Ice configurations are sparsely populated in the full configuration space. Each red dot represents an ice configuration. The distance between the red dots measures the difference between ice configurations. The black (dashed) circle indicates the configurations reachable by a single-spin flip. The probability $P$ for each ice configuration is a $\delta$ function; therefore, a single-spin flip alone cannot reach another ice configuration. By constructing a path (curved arrows) of single-spin flip moves, it is possible to reach a new ice configuration.

Figure 3

A global move in the Markov chain (a) from state $s_0$ to $s_T$ can be decomposed into a sequence of local moves. By extending the Markov chain to a Markov decision process, the policy $\pi \left(a\right|s)$ serves as a transition operator, making local decisions to generate a global movement in state space.

Figure 4

Architecture of agent-environment interface. From an initial ice state, the agent keeps executing actions on the environment and receiving corresponding rewards until the agent decides to make an update and finishes the episode. In the final configuration, only the spins altered by the agent are depicted.

Figure 5

Architecture of the multichannel neural network for the agent. For the NN policy, global observation is turned off.

Figure 6

(a) Local environment for an agent located at spin $\sigma$. There are six neighboring spins that the agent can choose to flip next. The blue spins ${\sigma }_4$ and ${\sigma }_5$ correspond to the possible choices in the loop algorithm. (b) Two types of actions that the agent can take: Flipping a neighboring spin or applying an update to the spin configuration.

Figure 7

Agent training process for $L=16$. The shaded area presents the loop length distribution, bounded by maximum and minimum lengths. The dashed line represents the acceptance ratio of the agent's proposed move. At the early stage of learning, the agent only creates $2\times 2$ smallest loops to obtain the goal rewards. In this phase, agent learns appropriate timing to execute $a_{\text{update}}$. Based on the learned small-loop strategy, the agent realizes that ice rule allows for larger loops to occur if there are no more defects are generated in the process. After $7\times {10}^5$ training steps, the acceptance ratio of the agent-proposed moves reaches close to 100%.

Figure 8

Decision-making process of the agent illustrated by the policy $\pi \left(a\right|s)$. Most of the time, the agent behaves in a similar way to a conventional long-loop algorithm which makes stochastic selection from two outgoing spins (with probability 0.5, denoted by red dashed lines). When the trajectory forms a closed loop, the agent executes the update operation with full confidence (orange striped bar).

Figure 9

State-value function $V^{\pi }\left(s\right)$ shows the expected future reward in the process of a loop formation. Shaded area shows the distribution of the value, bounded by the maximum and minimum value. A local minimum occurs when endpoints (circle and square) are far apart (A), and a local maximum occurs when two endpoints move closer to each other (B). When a closed loop emerges, the value function shows a jump and large reward is expected (C).

Figure 10

Probability that a site is visited starting from the same initial site and ice configuration for (a) loop algorithm, (b) NN policy, and (c) CNN policy. The NN policy shows significant memory effects while the CNN policy shows similar behavior as the loop algorithm.

Figure 11

Equivalence mapping between the (a) vertex spins on a square lattice and (b) Ising variables on a checkerboard lattice by a ${45}^{\circ }$ rotation. This specific configuration corresponds to the ice state with a Néel order, where a regular vortex pattern (black circular arrows) is shown.

Figure 12

The structure factor for $L=64$ computed using the configurations generated from (a) the conventional loop algorithm, (b) the NN policy, and (c) the CNN policy. The gray square indicates the magnetic first Brillouin zone. The NN and CNN policies are trained in a $L=32$ setting and a moving horizon embedding described in the Appendix pp1 is used to generate states for $L=64$. (d) Line cuts of $S\left(\mathbf{q}\right)$ along high-symmetry directions in the first Brillouin zone as indicated in the orange (dashed) lines in (a). Both the trained policies show bias toward the Néel order.

Figure 13

The sliding horizon embedding scheme. The trained agent (blue circle) takes input from an $L_t\times L_t$ window (dark inner square) embedded in a $L\times L$ lattice. Dashed square corresponds to the initial position of the observation window.