Improving the reliability of reinforcement learning algorithms through biconjugate Bellman errors

• Author / Creator

  Patterson, Andrew

• In this thesis, we seek to improve the reliability of reinforcement learning algorithms for nonlinear function approximation. Semi-gradient temporal difference (TD) update rules form the basis of most state-of-the-art value function learning systems despite clear counterexamples proving their potential instability. Gradient TD updates have provable stability under broad conditions, yet significantly underperform semi-gradient approaches on several problems of interest. In this thesis, we present a simple modification to an existing gradient TD method, prove that this method—called TDRC—remains stable, and show empirically that TDRC performs comparatively with semi-gradient approaches. Taking advantage of the connection between Fenchel duality and orthogonal projections, we justify the use of nonlinear value function approximation using gradient TD updates and show that these methods continue to inherit improved reliability over semi-gradient approaches in the nonlinear function approximation setting. We then extend this method to value-based control with neural networks and empirically validate its performance compared to semi-gradient methods. Finally, we propose two novel statistically robust losses—the mean Huber Projected Bellman error and the mean absolute Projected Bellman Error—and derive a family of off-policy gradient TD algorithms to optimize these losses for both prediction and control.

• Subjects / Keywords

  • Machine Learning
  • Reinforcement Learning

• Graduation date

  Spring 2024

• Type of Item

  Thesis

• Degree

  Doctor of Philosophy

• DOI

  https://doi.org/10.7939/r3-m94n-pm33

• License

  This thesis is made available by the University of Alberta Libraries with permission of the copyright owner solely for non-commercial purposes. This thesis, or any portion thereof, may not otherwise be copied or reproduced without the written consent of the copyright owner, except to the extent permitted by Canadian copyright law.