Recurrent Linear Transformers for Reinforcement Learning

• Author / Creator

  Pramanik, Subhojeet

• The transformer architecture is effective in processing sequential data, both because of its ability to leverage parallelism, and because of its self-attention mechanism capable of capturing long-range dependencies. However, the self-attention mechanism is slow for streaming data, that is when the input sequence is received one at a time. This limits its application in sequential decision-making problems such as online reinforcement learning. The self-attention mechanism requires past activations, that is the history, to be provided as context. As such, the inference cost, the cost of applying self-attention to a single element in a sequence, depends on the length of the input context. Increasing the context length of self-attention directly increases the inference cost. In this thesis, we present recurrent alternatives to the transformer self-attention that offer a context-independent inference cost, while also leveraging long-range dependencies. Our approaches are called the Recurrent Linear Transformer (ReLiT) and Approximate Recurrent Linear Transformer (AReLiT). We evaluate them on T-Maze, a partially observable reinforcement learning task that requires long-term memory, demonstrating their effectiveness when compared to existing RNN and transformer baselines. Additionally, we provide results in Memory Maze, a 3D pixel-based environment, and we empirically demonstrate the computational efficiency of our approach compared to standard transformer architectures.

• Subjects / Keywords

  • reinforcement learning
  • transformers

• Graduation date

  Fall 2023

• Type of Item

  Thesis

• Degree

  Master of Science

• DOI

  https://doi.org/10.7939/r3-fx4n-7p10

• License

  This thesis is made available by the University of Alberta Libraries with permission of the copyright owner solely for non-commercial purposes. This thesis, or any portion thereof, may not otherwise be copied or reproduced without the written consent of the copyright owner, except to the extent permitted by Canadian copyright law.