Analysis of an Alternate Policy Gradient Estimator for Softmax Policies

• Author / Creator

  Garg, Shivam

• Policy gradient (PG) estimators are ineffective in dealing with softmax policies that are sub-optimally saturated, which refers to the situation when the policy concentrates its probability mass on sub-optimal actions. Sub-optimal policy saturation may arise from a bad policy initialization or a sudden change, i.e. a non-stationarity, in the environment that occurs after the policy has already converged. Unfortunately, current softmax PG estimators require a large number of updates to overcome policy saturation, which causes low sample efficiency and poor adaptability to new situations. To mitigate this problem, we propose a novel policy gradient estimator for softmax policies that utilizes the bias in the critic estimate and the noise present in the reward signal to escape the saturated regions of the policy parameter space. Our experiments, conducted on bandits and various reinforcement learning environments, and the resulting analysis show that this new estimator is more robust to policy saturation, and can be readily adapted to work with different PG algorithms and function approximation schemes.

• Subjects / Keywords

  • Reinforcement Learning
  • Policy gradient methods
  • Softmax policies
  • Policy saturation
  • Alternate policy gradient estimator
  • Softmax gravity well
  • Policy gradient methods for non-stationary environments

• Graduation date

  Spring 2022

• Type of Item

  Thesis

• Degree

  Master of Science

• DOI

  https://doi.org/10.7939/r3-1w5k-fb72

• License

  This thesis is made available by the University of Alberta Libraries with permission of the copyright owner solely for non-commercial purposes. This thesis, or any portion thereof, may not otherwise be copied or reproduced without the written consent of the copyright owner, except to the extent permitted by Canadian copyright law.