Continuous Multilevel Actions in Reinforcement Learning

• Author / Creator

  Mitchell, Daniel

• Multilevel action selection is a reinforcement learning technique in which an action is broken into two parts, the type and the parameters. When using multilevel action selection in reinforcement learning, one must break the action space into multiple subsets. These subsets are typically disjoint and their union is equal to the original action space. When doing action selection, the subset, representing the action type, is chosen separately from the exact value of the action itself, the parameter values. The majority of research into multilevel action selection focuses on applying it to problems with conceptually distinct action types, such as robot soccer, where an agent can run, turn, tackle, or shoot. However, this is not the only application. In this thesis I focus on a different application of multilevel action selection, where I break down a simple one-dimensional action space into action types in order to focus on specific areas. The goal is to improve learning time by focusing on one area of the action space and disregarding all actions outside of that area, reducing the number of actions to search through. Once an agent has enough experience with actions from a type leading to poor return, it can generalize its experience to the entire action type and instead favour the other types which are more rewarding. I attempt to solve the mountain car and cart pole problems, which I chose for their simple action spaces with a conceptual difference between forwards and backwards thrust. I find that in these problems, multilevel action selection can improve the performance, measured by total return, of a reinforcement learning algorithm.

• Subjects / Keywords

  • Reinforcement Learning
  • Multilevel Action Selection

• Graduation date

  Fall 2023

• Type of Item

  Thesis

• Degree

  Master of Science

• DOI

  https://doi.org/10.7939/r3-h1ya-6d52

• License

  This thesis is made available by the University of Alberta Libraries with permission of the copyright owner solely for non-commercial purposes. This thesis, or any portion thereof, may not otherwise be copied or reproduced without the written consent of the copyright owner, except to the extent permitted by Canadian copyright law.