Path integral control and state-dependent feedback

In this paper we address the problem of computing state-dependent feedback controls for path integral control problems. To this end we generalize the path integral control formula and utilize this to construct parametrized state-dependent feedback controllers. In addition, we show a relation between control and importance sampling: Better control, in terms of control cost, yields more efficient importance sampling, in terms of effective sample size. The optimal control provides a zero-variance estimate.

Figure 1

Estimate of $Var\left({\alpha }^{\epsilon }\right)$, where ${\alpha }^{\epsilon }:=e^{-S^{u^{\epsilon }}\left(t_0\right)}/\psi (t_0,x_0)$ with upper and lower bounds from Theorem 2 with respect to the control problem in Example 1. Here we considered a range of suboptimal importance controls $u^{\epsilon }(t,x)=u^{*}(t,x)+\sqrt{\epsilon }$. The estimate of the variance is based on ${10}^4$ paths that were generated with $dt=0.001$.

Figure 2

The approximate controls calculated with ${10}^4$ sample paths in two importance sampling iterations using a time discretization of $dt=0.001$ for numerical integration. The histogram was created with ${10}^4$ draws from $X^{u^{*}}\left(t\right)$ at $t=1/2$.