Fitted strongQstrong-iteration in continuous action-space MDPs.pdfVIP

Fitted Q-iteration in continuous action-space MDPs ´ ´ Andras Antos Remi Munos Computer and Automation Research Inst. SequeL project-team, INRIA Lille of the Hungarian Academy of Sciences 59650 Villeneuve d’Ascq, France Kende u. 13-17, Budapest 1111, Hungary remi.munos@inria.fr antos@sztaki.hu ´ Csaba Szepesvari Department of Computing Science University of Alberta Edmonton T6G 2E8, Canada szepesva@cs.ualberta.ca Abstract We consider continuous state, continuous action batch reinforcement learning where the goal is to learn a good policy from a sufficiently rich trajectory gen- erated by some policy. We study a variant of fitted Q-iteration, where the greedy action selection is replaced by searching for a policy in a restricted set of can- didate policies by maximizing the average action values. We provide a rigorous analysis of this algorithm, proving what we believe is the first finite-time bound for value-function based algorithms for continuous state and action problems. 1 Preliminaries We will build on the results from [1, 2, 3] and for this reason we use the same notation as these papers. The unattributed results cited in this section can be found in the book [4]. A discounted MDP is defined by a quintuple , where is the (possible infinite) state space, is the set of actions, is the transition probability kernel with defining the next-state distribution upon taking action from state , g