- 106 views
- 147 downloads
An Empirical Study of Experience Replay for Control in Continuous State Domains
-
- Author / Creator
- Li, Xin
-
In this thesis, we investigate the empirical performance of several experience replay techniques. Efficient experience replay plays an important role in model-free reinforcement learning by improving sample efficiency through reusing past experience. However, replay-based methods were largely forgotten before being repopularized by the Deep Q-Learning Network (DQN) architecture and have since become a standard component in offline training. In this work, we revisit understudied classic replay strategies, backward replay and on-policy replay, a heuristic for replaying trajectories in the temporally backward order following on-policy action sequences, proposed in the original experience replay literature. We re-evaluate them in several classic reinforcement learning control problems under linear and nonlinear function approximations. We observe that (1) on-policy replay outperforms off-policy replay in two continuous state 2D maze domains under an exploratory policy, and (2) contrary to the previous claim in the original replay literature, replay settings exist where on-policy replay underperforms off-policy replay. We hypothesize that the practical benefit of on-policy replay is problem dependent and sensitive to the state distribution of the replay buffer. In addition, we propose a simple time-stepping replay strategy called ``jumpy replay'' that takes advantage of state generalization to speed up value propagation, which presents a comparable or better performance with a vanilla backward replay baseline across 5 replay settings.
-
- Subjects / Keywords
-
- Graduation date
- Fall 2022
-
- Type of Item
- Thesis
-
- Degree
- Master of Science
-
- License
- This thesis is made available by the University of Alberta Library with permission of the copyright owner solely for non-commercial purposes. This thesis, or any portion thereof, may not otherwise be copied or reproduced without the written consent of the copyright owner, except to the extent permitted by Canadian copyright law.