Clear Water Bay, Kowloon.pdf

Multi-model Approach to Non-stationary Reinforcement Learning Samuel P. M. Choi, Dit Yan Yeung, Nevin L. Zhang Department of Computer Science Hong Kong University of Science and Technology Clear Water Bay, Kowloon Hong Kongfpmchoi,dyyeung,lzhangg@cs.ust.hk ABSTRACT This paper proposes a novel alogrithm for a class of non- stationary reinforcement learning problems in which the environmental changes are rare and finite. Through dis- carding corrupted models and combining similar ones, the proposed algorithm maintains a collection of frequently encountered environment models and enables an effective adaptation when a similar environment recurs. The algo- rithm has empirically compared with the finite window ap- proach, a widely-used method for non-stationary RL prob- lems. Results have shown that our algorithm consistently outperforms the finite window approach in various empiri- cal setups. KEY WORDS Reinforcement Learning, Non-stationary Environment 1 Introduction Learning to perform sequential decision tasks in a com- plex environment is non-trivial, especially when the envi- ronment is not known in advance. Reinforcement learn- ing (RL) [4, 9] is a computational approach to such a task through learning from interaction. Thus far, most existing RL researches are focussed on stationary Markovian envi- ronments; i.e., the underlying dynamics of the environment depend solely on the current state and are independent of time. Non-stationary environments, on the contrary, refer to the stochastic environments in which the underlying pa- rameters may vary over time. Non-stationary problems are very common in the real world. Consider a robot rover which explores in an un- visited planet. When roaming around, the rover may en- counter various types of weathers and terrains (e.g. uphill, downhill, and craters). In order to navigate in a desired manner, the rover may need different sequence of control actions for each environment. Hence, it would be inef- fective to treat all