mlsanityData to 7 Oct 2026

Recurrent Reinforcement Learning with Memoroids

arXiv 2402.09900 ↗First posted 15 Feb 2024cs.LG cs.AIcited by 5

Not on the map. Why?

Abstract (arXiv)

Memory models such as Recurrent Neural Networks (RNNs) and Transformers address Partially Observable Markov Decision Processes (POMDPs) by mapping trajectories to latent Markov states. Neither model scales particularly well to long sequences, especially compared to an emerging class of memory models called Linear Recurrent Models. We discover that the recurrent update of these models resembles a monoid, leading us to reformulate existing models using a novel monoid-based framework that we call memoroids. We revisit the traditional approach to batching in recurrent reinforcement learning, highlighting theoretical and empirical deficiencies. We leverage memoroids to propose a batching method that improves sample efficiency, increases the return, and simplifies the implementation of recurrent loss functions in reinforcement learning.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 25 of its 39 references that are in the map. First posted on arXiv 15 Feb 2024.

Builds on

25 of 39 references.

References · 25

  1. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  2. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  3. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  4. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  5. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  6. 2023Mastering Diverse Domains through World Modelscited by 1,193
  7. 2015Prioritized Experience Replaycited by 1,070
  8. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  9. 2022Simplified State Space Layers for Sequence Modelingcited by 776
  10. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 769
  11. 2021Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layerscited by 607
  12. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
13 more
  1. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 471
  2. 2015Deep Recurrent Q-Learning for Partially Observable MDPscited by 457
  3. 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 397
  4. 2023Human-Timescale Adaptation in an Open-Ended Task Spacecited by 136
  5. 2017Parallelizing Linear Recurrent Neural Nets Over Sequence Lengthcited by 126
  6. 2023Structured State Space Models for In-Context Reinforcement Learningcited by 109
  7. 2022Discovered Policy Optimisationcited by 103
  8. 2023When Do Transformers Shine in RL? Decoupling Memory from Credit Assignmentcited by 62
  9. 2023POPGym: Benchmarking Partially Observable Reinforcement Learningcited by 59
  10. 2022skrl: Modular and Flexible Library for Reinforcement Learningcited by 35
  11. 2023Exploring the Promise and Limits of Real-Time Recurrent Learningcited by 19
  12. 2023Reinforcement Learning with Fast and Forgetful Memorycited by 8
  13. 2018Theoretically Efficient Parallel Graph Algorithms Can Be Fast and Scalablecited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 1.

5 papers.

  1. 2025Investigating Memory in Model-Free RL with POPGym Arcadecited by 1
  2. 2025Recurrent State Encoders for Efficient Neural Combinatorial Optimizationcited by 1
  3. 2025Long-Horizon Model-Based Offline Reinforcement Learning Without Explicit Conservatismcited by 1
  4. 2025Recurrent Off-Policy Deep Reinforcement Learning Doesn't Have to be Slowcited by 0
  5. 2026Cross-Domain Policy Optimization via Bellman Consistency and Hybrid Criticscited by 0