Investigating the Interplay of Prioritized Replay and Generalization

• Author / Creator

  Mohammad Panahi, Parham

• Experience replay, the reuse of past data to improve sample efficiency, is ubiquitous in reinforcement learning. Though a variety of smart sampling schemes have been introduced to improve performance, uniform sampling by far remains the most common approach. One exception is Prioritized Experience Replay (PER), where sampling is done proportionally to TD errors, inspired by the success of prioritized sweeping in dynamic programming. The original work on PER showed improvements in Atari, but follow-up results were mixed. In this thesis, we investigate several variations on PER, to attempt to understand where and when PER may be useful. Our findings in prediction tasks reveal that while PER can improve value propagation in tabular settings, behavior is significantly different when combined with neural networks. Certain mitigations---like delaying target network updates to control generalization and using estimates of expected TD errors in PER to avoid chasing stochasticity---can avoid large spikes in error with PER and neural networks but generally do not outperform uniform replay. In control tasks, none of the prioritized variants consistently outperform uniform replay with neural networks; however, recomputing priorities and sampling without replacement improves the performance of PER over uniform replay in tabular settings. This thesis sheds light on the interaction between prioritization, bootstrapping, and neural networks and proposes several improvements for PER in tabular settings and noisy domains.

• Subjects / Keywords

  • Reinforcement Learning
  • Experience Replay

• Graduation date

  Fall 2024

• Type of Item

  Thesis

• Degree

  Master of Science

• DOI

  https://doi.org/10.7939/r3-hj51-3v03

• License

  This thesis is made available by the University of Alberta Library with permission of the copyright owner solely for non-commercial purposes. This thesis, or any portion thereof, may not otherwise be copied or reproduced without the written consent of the copyright owner, except to the extent permitted by Canadian copyright law.