mlsanityData to 7 Oct 2026

Unraveling the Complexity of Memory in RL Agents: an Approach for Classification and Evaluation

arXiv 2412.06531 ↗First posted 9 Dec 2024cs.LG cs.AIcited by 5

Not on the map. Why?

Abstract (arXiv)

The incorporation of memory into agents is essential for numerous tasks within the domain of Reinforcement Learning (RL). In particular, memory is paramount for tasks that require the use of past information, adaptation to novel environments, and improved sample efficiency. However, the term "memory" encompasses a wide range of concepts, which, coupled with the lack of a unified methodology for validating an agent's memory, leads to erroneous judgments about agents' memory capabilities and prevents objective comparison with other memory-enhanced agents. This paper aims to streamline the concept of memory in RL by providing practical precise definitions of agent memory types, such as long-term vs. short-term memory and declarative vs. procedural memory, inspired by cognitive science. Using these definitions, we categorize different classes of agent memory, propose a robust experimental methodology for evaluating the memory capabilities of RL agents, and standardize evaluations. Furthermore, we empirically demonstrate the importance of adhering to the proposed methodology when evaluating different types of agent memory by conducting experiments with different RL agents and what its violation leads to.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 44 of its 68 references that are in the map. First posted on arXiv 9 Dec 2024.

Builds on

44 of 68 references.

References · 44

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  3. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  4. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  5. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  6. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  7. 2022Simplified State Space Layers for Sequence Modelingcited by 776
  8. 2018Recurrent World Models Facilitate Policy Evolutioncited by 761
  9. 2014Neural Turing Machinescited by 630
  10. 2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 490
  11. 2015Deep Recurrent Q-Learning for Partially Observable MDPscited by 457
  12. 2023Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Taskscited by 320
32 more
  1. 2019Stabilizing Transformers for Reinforcement Learningcited by 262
  2. 2019VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learningcited by 210
  3. 2017Neural Episodic Controlcited by 148
  4. 2023Transformer-based World Models Are Happy With 100k Interactionscited by 137
  5. 2023Human-Timescale Adaptation in an Open-Ended Task Spacecited by 136
  6. 2019Behaviour Suite for Reinforcement Learningcited by 129
  7. 2021Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDPscited by 115
  8. 2023Structured State Space Models for In-Context Reinforcement Learningcited by 109
  9. 2017Neural Map: Structured Memory for Deep Reinforcement Learningcited by 88
  10. 2023When Do Transformers Shine in RL? Decoupling Memory from Credit Assignmentcited by 62
  11. 2016Control of Memory, Active Perception, and Action in Minecraftcited by 61
  12. 2020Counterfactual Credit Assignment in Model-Free Reinforcement Learningcited by 60
  13. 2018Learning Longer-term Dependencies in RNNs with Auxiliary Lossescited by 58
  14. 2022Evaluating Long-Term Memory in 3D Mazescited by 41
  15. 2022Transformers are Meta-Reinforcement Learnerscited by 39
  16. 2021Towards mental time travel: a hierarchical memory for reinforcement learning agentscited by 38
  17. 2024Mastering Memory Tasks with World Modelscited by 38
  18. 2019Generalization of Reinforcement Learners with Working and Episodic Memorycited by 35
  19. 2022Deep Transformer Q-Networks for Partially Observable Reinforcement Learningcited by 35
  20. 2015Deep Attention Recurrent Q-Networkcited by 27
  21. 2023Recurrent Action Transformer with Memorycited by 17
  22. 2023Memory Gym: Towards Endless Tasks to Benchmark Memory Capabilities of Agentscited by 16
  23. 2022Value Memory Graph: A Graph-Structured World Model for Offline Reinforcement Learningcited by 11
  24. 2022Explain My Surprise: Learning Efficient Long-Term Memory by Predicting Uncertain Outcomescited by 11
  25. 2024KalMamba: Towards Efficient Probabilistic State Space Models for RL under Uncertaintycited by 9
  26. 2023Reinforcement Learning with Fast and Forgetful Memorycited by 8
  27. 2024Stable Hadamard Memory: Revitalizing Memory-Augmented Agents for Reinforcement Learningcited by 8
  28. 2017Recurrent Deterministic Policy Gradient Method for Bipedal Locomotion on Rough Terrain Challengecited by 7
  29. 2019Interval timing in deep reinforcement learning agentscited by 7
  30. 2024Hierarchical Transformers are Efficient Meta-Reinforcement Learnerscited by 6
  31. 2024Learning Memory Mechanisms for Decision Making through Demonstrationscited by 6
  32. 2021Graph Convolutional Memory using Topological Priorscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 2.

5 papers.

  1. 2025Memory, Benchmark & Robots: A Benchmark for Solving Complex Tasks with Reinforcement Learningcited by 34
  2. 2025ELMUR: External Layer Memory with Update/Rewrite for Long-Horizon RL Problemscited by 6
  3. 2025Re:Frame -- Retrieving Experience From Associative Memorycited by 1
  4. 2026Memory Retention Is Not Enough to Master Memory Tasks in Reinforcement Learningcited by 1
  5. 2026From Static Policies to Adaptive Priors in Offline Reinforcement Learningcited by 0