mlsanityData to 14 Oct 2026

Synthetic POMDPs to Challenge Memory-Augmented RL: Memory Demand Structure Modeling

arXiv 2508.04282 ↗First posted 6 Aug 2025cs.AIcited by 1

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Recent benchmarks for memory-augmented reinforcement learning (RL) have introduced partially observable Markov decision process (POMDP) environments in which agents must use historical observations to make decisions. However, these benchmarks often lack fine-grained control over the challenges posed to memory models. Synthetic environments offer a solution, enabling precise manipulation of environment dynamics for rigorous and interpretable evaluation of memory-augmented RL. This paper advances the design of such customizable POMDPs with three key contributions: (1) a theoretical framework for analyzing POMDPs based on Memory Demand Structure (MDS) and related concepts; (2) a methodology using linear dynamics, state aggregation, and reward redistribution to construct POMDPs with predefined MDS; and (3) a suite of lightweight, scalable POMDP environments with tunable difficulty, grounded in our theoretical insights. Overall, our work clarifies core challenges in partially observable RL, offers principled guidelines for POMDP design, and aids in selecting and developing suitable memory architectures for RL tasks.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 17 of its 36 references that are in the map. First posted on arXiv 6 Aug 2025.

Builds on

17 of 36 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025Benchmarking Partial Observability in Reinforcement Learning with a Suite of Memory-Improvable Domains31 Jul

Other references · 16

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  3. 2016OpenAI Gymcited by 1,736
  4. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 483
  5. 2021Benchmarking the Spectrum of Agent Capabilitiescited by 200
  6. 2020The NetHack Learning Environmentcited by 198
  7. 2018RUDDER: Return Decomposition for Delayed Rewardscited by 173
  8. 2019Behaviour Suite for Reinforcement Learningcited by 132
  9. 2021On the Expressivity of Markov Rewardcited by 71
  10. 2019Variational Recurrent Models for Solving Partially Observable Control Taskscited by 47
  11. 2019Reinforcement Learning with Non-Markovian Rewardscited by 45
  12. 2021Acting in Delayed Environments with Non-Stationary Markov Policiescited by 31
4 more
  1. 2021Off-Policy Reinforcement Learning with Delayed Rewardscited by 25
  2. 2020Learning Non-Markovian Reward Models in MDPscited by 11
  3. 2024Detecting Hidden Triggers: Mapping Non-Markov Reward Functions to Markovcited by 5
  4. 2025Constructing Non-Markovian Decision Process via History Aggregatorcited by 1

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025ELMUR: External Layer Memory with Update/Rewrite for Long-Horizon RL Problemscited by 6

Similar, not cited

Close in wording, no citation either way

  1. 2025Bi-directional Recurrence Improves Transformer in Partially Observable Markov Decision Processes
  2. 2024Rethinking Transformers in Solving POMDPs
  3. 2024Stable Hadamard Memory: Revitalizing Memory-Augmented Agents for Reinforcement Learning
  4. 2025Robust Finite-Memory Policy Gradients for Hidden-Model POMDPs
  5. 2026Synthetic Monitoring Environments for Reinforcement Learning
  6. 2023Safe Policy Improvement for POMDPs via Finite-State Controllers
  7. 2025Scalable Policy-Based RL Algorithms for POMDPs
  8. 2025Learning POMDPs with Linear Function Approximation and Finite Memory