Quantifying the stochasticity of policy parameters in reinforcement learning problems

The stochastic dynamics of reinforcement learning is studied using a master equation formalism. We consider two different problems—$Q$ learning for a two-agent game and the multiarmed bandit problem with policy gradient as the learning method. The master equation is constructed by introducing a probability distribution over continuous policy parameters or over both continuous policy parameters and discrete state variables (a more advanced case). We use a version of the moment closure approximation to solve for the stochastic dynamics of the models. Our method gives accurate estimates for the mean and the (co)variance of policy variables. For the case of the two-agent game, we find that the variance terms are finite at steady state and derive a system of algebraic equations for computing them directly.

Figure 1

Reinforcement learning for a two-agent game. (a) Schematic of the learning algorithm. (b) Example stochastic dynamics of $Q$ values. Parameters used in the study were $\alpha =0.1$, $T=1.0$, $A=\left[\right(2,1),(0,1\left)\right]$, and $B=\left[\right(1,0),(2,2\left)\right]$.

Figure 2

Comparison of simulation results with those of the iterative analytical method. (a) Dynamics of $Q$ values. Transparent trajectories represent independent learning simulations. Dotted lines represent estimates from the iterative analytical approach. (b) Dynamics of covariance terms (4 of the 10 independent elements are shown for clarity). Parameters used are identical to those in the caption of Fig. 1. $Q$ values were initialized at $q_{\mathrm{init},l}^a=q_{\mathrm{init},n}^b=0$.

Figure 3

Comparison of steady-state covariance elements estimated using simulation and analytical methods [Eq. (16)]. Inset: Linear scaling of covariance terms with the learning rate $\alpha$ (obtained from simulation).

Figure 4

Estimating the stochastic learning dynamics in the two-armed bandit problem. (a) Policy parameter $q$ and (b) action 1 probability $x\left(q\right)$ as a function of the learning iteration. Transparent trajectories represent example stochastic realizations of the learning process. (c) Variance of the policy parameter and (c) variance of the action 1 probability estimated via simulation (solid lines) vs our iterative analytical approach (dashed lines). Parameters and initial conditions used: $r_1=1$, $r_2=-1$, $s_1=s_2=1$, and $q\left(0\right)=0$.

Figure 5

Stochastic learning dynamics in the $K$-armed bandit model with $K=50$. (a) Mean policy parameters $E\left[q_i\right]$ as a function of the learning iteration. (b) Dynamics of the variance terms $V_{ii}$ (orange) and covariance terms $V_{ij}$ (pink). Dashed lines are obtained from our iterative analytical approach, while the solid lines are generated from numerical simulations. Due to the large number of different summary statistics, only a subset of the terms are shown. Parameters used in the study: $r_i=1-2(i-1)/(K-1)$, $s_i=0.1$, $\alpha =0.01$.

Figure 6

Comparison of summary statistics obtained from simulation vs analytics in the state-dependent two-agent game. (a) Dynamics of $Q$ values. (b) Dynamics of covariance element (a subset is shown for clarity). Same parameters as those in Fig. 1 were used.