mlsanityData to 14 Oct 2026

Investigating Memory in Model-Free RL with POPGym Arcade

arXiv 2503.01450 ↗First posted 3 Mar 2025cs.LG cs.AI cs.ROcited by 3

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

How should we analyze memory in deep RL? We introduce tools for analyzing policies under partial observability and revealing how agents use memory to make decisions. To utilize these tools, we present POPGym Arcade, a collection of Atari-inspired, hardware-accelerated environments sharing a single observation and action space. Each environment provides fully and partially observable variants, enabling counterfactual studies on observability. We find that controlled studies are necessary for fair comparisons and identify a pathology where value functions smear credit over irrelevant history. Using this pathology, we demonstrate how out-of-distribution scenarios can contaminate memory, perturbing the policy far into the future. Our code is available at https://github.com/bolt-research/popgym-arcade.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 41 of its 63 references that are in the map. First posted on arXiv 3 Mar 2025.

Builds on

41 of 63 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Stable Hadamard Memory: Revitalizing Memory-Augmented Agents for Reinforcement Learning14 Oct
  2. 2024Real-Time Recurrent Learning using Trace Units in Reinforcement Learning2 Sep
  3. 2024Simplifying Deep Temporal Difference Learning5 Jul
  4. 2024Position: Benchmarking is Limited in Reinforcement Learning Research23 Jun
  5. 2024Recurrent Reinforcement Learning with Memoroids15 Feb

Other references · 36

  1. 2017Attention Is All You Needcited by 60,361
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  4. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,465
  5. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,367
  6. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  7. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  8. 2018Exploration by Random Network Distillationcited by 993
  9. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 532
  10. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 512
  11. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 502
  12. 2015Deep Recurrent Q-Learning for Partially Observable MDPscited by 465
24 more
  1. 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 408
  2. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 396
  3. 2024Learning to (Learn at Test Time): RNNs with Expressive Hidden Statescited by 355
  4. 2019Stabilizing Transformers for Reinforcement Learningcited by 270
  5. 2024The Illusion of State in State-Space Modelscited by 187
  6. 2016ViZDoom: A Doom-based AI Research Platform for Visual Reinforcement Learningcited by 179
  7. 2021Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDPscited by 122
  8. 2023Structured State Space Models for In-Context Reinforcement Learningcited by 111
  9. 2022Discovered Policy Optimisationcited by 104
  10. 2021The Distracting Control Suite -- A Challenging Benchmark for Reinforcement Learning from Pixelscited by 100
  11. 2022When Is Partially Observable Reinforcement Learning Not Scary?cited by 100
  12. 2024Craftax: A Lightning-Fast Benchmark for Open-Ended Reinforcement Learningcited by 88
  13. 2021On the difficulty of learning chaotic dynamics with RNNscited by 67
  14. 2023When Do Transformers Shine in RL? Decoupling Memory from Credit Assignmentcited by 63
  15. 2022EnvPool: A Highly Parallel Reinforcement Learning Environment Execution Enginecited by 62
  16. 2023Jumanji: a Diverse Suite of Scalable Reinforcement Learning Environments in JAXcited by 58
  17. 2023Pgx: Hardware-Accelerated Parallel Game Simulators for Reinforcement Learningcited by 51
  18. 2024Kinetix: Investigating the Training of General Agents through Open-Ended Physics-Based Control Taskscited by 45
  19. 2021Podracer architectures for scalable Reinforcement Learningcited by 38
  20. 2019Generalization of Reinforcement Learners with Working and Episodic Memorycited by 36
  21. 2024NAVIX: Scaling MiniGrid Environments with JAXcited by 18
  22. 2023Memory Gym: Towards Endless Tasks to Benchmark Memory Capabilities of Agentscited by 16
  23. 2023Reinforcement Learning with Fast and Forgetful Memorycited by 9
  24. 2019MDP Playground: An Analysis and Debug Testbed for Reinforcement Learningcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026In-Context Reinforcement Learning under Non-Stationarity: A Surveycited by 1
  2. 2026ALER: Adaptive Learnable Experience Rewriting for Reinforcement Learningcited by 0
  3. 2026Does Learning Recurrent Networks Matter in Partially Observable RL?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Synthetic POMDPs to Challenge Memory-Augmented RL: Memory Demand Structure Modeling
  2. 2025Benchmarking Partial Observability in Reinforcement Learning with a Suite of Memory-Improvable Domains
  3. 2024Unraveling the Complexity of Memory in RL Agents: an Approach for Classification and Evaluation
  4. 2026Memory Retention Is Not Enough to Master Memory Tasks in Reinforcement Learning
  5. 2024Mastering Memory Tasks with World Models
  6. 2026Forager: a lightweight testbed for continual learning with partial observability in RL
  7. 2026MemoWM: How World Models Change What Agents Need to Remember
  8. 2026MemSyco-Bench: Benchmarking Sycophancy in Agent Memory