mlsanityData to 7 Oct 2026

Memory Gym: Towards Endless Tasks to Benchmark Memory Capabilities of Agents

arXiv 2309.17207 ↗First posted 29 Sep 2023cs.LGcited by 16

Not on the map. Why?

Abstract (arXiv)

Memory Gym presents a suite of 2D partially observable environments, namely Mortar Mayhem, Mystery Path, and Searing Spotlights, designed to benchmark memory capabilities in decision-making agents. These environments, originally with finite tasks, are expanded into innovative, endless formats, mirroring the escalating challenges of cumulative memory games such as "I packed my bag". This progression in task design shifts the focus from merely assessing sample efficiency to also probing the levels of memory effectiveness in dynamic, prolonged scenarios. To address the gap in available memory-based Deep Reinforcement Learning baselines, we introduce an implementation within the open-source CleanRL library that integrates Transformer-XL (TrXL) with Proximal Policy Optimization. This approach utilizes TrXL as a form of episodic memory, employing a sliding window technique. Our comparative study between the Gated Recurrent Unit (GRU) and TrXL reveals varied performances across our finite and endless tasks. TrXL, on the finite environments, demonstrates superior effectiveness over GRU, but only when utilizing an auxiliary loss to reconstruct observations. Notably, GRU makes a remarkable resurgence in all endless tasks, consistently outperforming TrXL by significant margins. Website and Source Code: https://marcometer.github.io/jmlr_2024.github.io/

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 32 of its 51 references that are in the map. First posted on arXiv 29 Sep 2023.

Builds on

32 of 51 references.

References · 32

  1. 2017Attention Is All You Needcited by 59,932
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  4. 2019Optuna: A Next-generation Hyperparameter Optimization Frameworkcited by 2,455
  5. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  6. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  7. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  8. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  9. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  10. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  11. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  12. 2018Learning Dexterous In-Hand Manipulationcited by 643
20 more
  1. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 478
  2. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 469
  3. 2015Deep Recurrent Q-Learning for Partially Observable MDPscited by 457
  4. 2020Curriculum Learning for Reinforcement Learning Domains: A Framework and Surveycited by 350
  5. 2019Emergent Tool Use From Multi-Agent Autocurriculacited by 308
  6. 2023Structured State Space Models for In-Context Reinforcement Learningcited by 109
  7. 2020Enhanced POET: Open-Ended Reinforcement Learning through Unbounded Invention of Learning Challenges and their Solutionscited by 103
  8. 2019V-MPO: On-Policy Maximum a Posteriori Policy Optimization for Discrete and Continuous Controlcited by 85
  9. 2015Memory-based control with recurrent neural networkscited by 77
  10. 2023POPGym: Benchmarking Partially Observable Reinforcement Learningcited by 59
  11. 2020Grounded Language Learning Fast and Slowcited by 54
  12. 2021Memory-based Deep Reinforcement Learning for POMDPscited by 53
  13. 2018ViZDoom Competitions: Playing Doom from Pixelscited by 50
  14. 2022Evaluating Long-Term Memory in 3D Mazescited by 41
  15. 2022History Compression via Language Models in Reinforcement Learningcited by 39
  16. 2021Towards mental time travel: a hierarchical memory for reinforcement learning agentscited by 38
  17. 2019Generalization of Reinforcement Learners with Working and Episodic Memorycited by 35
  18. 2021Efficient Transformers in Reinforcement Learning using Actor-Learner Distillationcited by 35
  19. 2021Reinforcement Learning with Latent Flowcited by 18
  20. 2018Psychlab: A Psychology Laboratory for Deep Reinforcement Learning Agentscited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 8; 2026: 7.

16 papers.

Sort:

By area:

  1. 2026MemoryArena: Benchmarking Agent Memory in Interdependent Multi-Session Agentic Taskscited by 72
  2. 2025Graphs Meet AI Agents: Taxonomy, Progress, and Future Opportunitiescited by 23
  3. 2025Rethinking Progression of Memory State in Robotic Manipulation: An Object-Centric Perspectivecited by 20
  4. 2024Unraveling the Complexity of Memory in RL Agents: an Approach for Classification and Evaluationcited by 5
  5. 2025RLBenchNet: The Right Network for the Right Reinforcement Learning Taskcited by 2
  6. 2025Orchestrator: Active Inference for Multi-Agent Systems in Long-Horizon Taskscited by 2
  7. 2025Learning Robust Penetration Testing Policies under Partial Observability: A systematic evaluationcited by 2
  8. 2025Investigating Memory in Model-Free RL with POPGym Arcadecited by 1
  9. 2025Exploring the Stratified Space Structure of an RL Game with the Volume Growth Transformcited by 1
  10. 2026Memory Retention Is Not Enough to Master Memory Tasks in Reinforcement Learningcited by 1
  11. 2025Learning When to Switch: Adaptive Policy Selection via Reinforcement Learningcited by 0
  12. 2026StateLinFormer: Stateful Training Enhancing Long-term Memory in Navigationcited by 0
4 more
  1. 2026Stratifying Reinforcement Learning with Signal Temporal Logiccited by 0
  2. 2026Reward Structure Shapes the Interaction Between Episodic Exploration and Neural Memory in Reinforcement Learningcited by 0
  3. 2026Minimal Recurrent Behavioral Memory for Imitation under Partial Observabilitycited by 0
  4. 2026ALER: Adaptive Learnable Experience Rewriting for Reinforcement Learningcited by 0