Greedy Actor-Critic: A New Conditional Cross-Entropy Method for Policy Improvement

• Author / Creator

  Neumann,Samuel

• Actor-Critics are a popular class of algorithms for control. Their ability to learn complex behaviours in continuous-action environments make them directly applicable to many real-world scenarios. These algorithms are composed of two parts - a critic and an actor. The critic learns to critique actions taken by the actor. The actor, a policy, uses this criticism to learn to attain higher rewards. Generally, this policy is improved by matching it to the Boltzmann distribution over action values. In this thesis, we introduce an alternative policy update, based on the cross-entropy method (CEM). This Conditional CEM (CCEM) applies the CEM to maximize an action-value critic conditioned on state. The algorithm works by initializing the actor policy as a wide distribution and iteratively concentrating on highly valued actions by using a maximum likelihood update toward the top percentile of an empirical action distribution. This empirical action distribution is generated by an additional, entropy regularized proposal policy that also concentrates on maximally valued actions, although more slowly. Under ideal conditions, the CCEM guarantees policy improvement and tracks the expected solution of the CEM across states. Finally, we introduce a new actor-critic algorithm called Greedy Actor-Critic that uses the CCEM for policy improvement. We empirically show that Greedy Actor-Critic can perform better than Soft Actor-Critic on a suite of classic control environments and is somewhat less sensitive to hyperparameters than SAC is on this environmental suite.

• Subjects / Keywords

  • reinforcement learning
  • actor-critic
  • greedy actor-critic
  • percentile policy
  • policy optimization
  • policy search
  • policy gradient
  • conditional cross-entropy optimization
  • CCEM
  • cross-entropy optimization

• Graduation date

  Fall 2022

• Type of Item

  Thesis

• Degree

  Master of Science

• DOI

  https://doi.org/10.7939/r3-s4hr-bf82

• License

  This thesis is made available by the University of Alberta Library with permission of the copyright owner solely for non-commercial purposes. This thesis, or any portion thereof, may not otherwise be copied or reproduced without the written consent of the copyright owner, except to the extent permitted by Canadian copyright law.