又邑忠烈右屹淤.PDF

Reinforcement Learning In Continuous Time and Space Kenji Doya ATR Human Information Pro cessing Research Lab oratories 2-2 Hikaridai, Seika, Soraku, Kyoto 619-0288, Japan Neural Computation, 12(1), 219-245 (2000). Abstract This pap er presents a reinforcement learning framework for continuous- time dynamical systems without a priori discretization of time, state, and action. Based on the Hamilton-Jacobi-Bellman (HJB) equation for innite- horizon, discounted reward problems, we derive algorithms for estimating value functions and for improving p olicies with the use of function approx- imators. The pro cess of value function estimation is formulated as the minimization of a continuous-time form of the temp oral dierence (TD) error. Up date metho ds based on backward Euler approximation and ex- p onential eligibility traces are derived and their corresp ondences with the conventional residual gradient, TD(0), and TD(  ) algorithms are shown. For p olicy improvement, two metho ds, namely, a continuous actor-critic metho d and a value-gradient based greedy p olicy, are formulated. As a sp ecial case of the latter, a nonlinear feedback control law using the value gradient and the mo del of the input gain is derived. The \advantage up- dating, a mo del-free algorithm derived previously, is also formulated in the HJB based framework. The p erformance of the prop osed algorithms is rst tested in a non- linear control task of swinging up a p endulum with limited torque. It is sho