mlsanityData to 14 Oct 2026

A Convolution and Attention Based Encoder for Reinforcement Learning under Partial Observability

arXiv 2505.23857 ↗First posted 29 May 2025cs.LG cs.AIcited by 0

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Partially Observable Markov Decision Processes (POMDPs) remain a core challenge in reinforcement learning due to incomplete state information. We address this by reformulating POMDPs as fully observable processes with fixed-length observation histories as augmented states. To efficiently encode these histories, we propose a lightweight temporal encoder based on depthwise separable convolution and self-attention, avoiding the overhead of recurrent and Transformer-based models. Integrated into an actor-critic framework, our method achieves superior performance on continuous control benchmarks under partial observability. More broadly, this work shows that lightweight temporal encoding can improve the scalability of AI systems under uncertainty. It advances the development of agents capable of reasoning robustly in real-world environments where information is incomplete or delayed.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 15 of its 33 references that are in the map. First posted on arXiv 29 May 2025.

Builds on

15 of 33 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Efficient Recurrent Off-Policy RL Requires a Context-Encoder-Specific Learning Rate24 May

Other references · 14

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  4. 2016Xception: Deep Learning with Depthwise Separable Convolutionscited by 2,332
  5. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  6. 2015Deep Recurrent Q-Learning for Partially Observable MDPscited by 465
  7. 2022Online Decision Transformercited by 196
  8. 2021Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDPscited by 122
  9. 2021Memory-based Deep Reinforcement Learning for POMDPscited by 54
  10. 2020MAGIC: Learning Macro-Actions for Online POMDP Planningcited by 17
  11. 2020Multiagent Rollout and Policy Iteration for POMDP with Application to Multi-Robot Repair Problemscited by 13
  12. 2022Perceiving the World: Question-guided Reinforcement Learning for Text-based Gamescited by 10
2 more
  1. 2017Recurrent Deterministic Policy Gradient Method for Bipedal Locomotion on Rough Terrain Challengecited by 7
  2. 2023ODE-based Recurrent Model-free Reinforcement Learning for POMDPscited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Uncertainty Representations in State-Space Layers for Deep Reinforcement Learning under Partial Observability
  2. 2025Learning to Focus: Prioritizing Informative Histories with Structured Attention Mechanisms in Partially Observable Reinforcement Learning
  3. 2024Learning Interpretable Policies in Hindsight-Observable POMDPs through Partially Supervised Reinforcement Learning
  4. 2025Benchmarking Partial Observability in Reinforcement Learning with a Suite of Memory-Improvable Domains
  5. 2024An Examination of Offline-Trained Encoders in Vision-Based Deep Reinforcement Learning for Autonomous Driving
  6. 2021Temporal Predictive Coding For Model-Based Planning In Latent Space
  7. 2024Bridging State and History Representations: Understanding Self-Predictive RL
  8. 2025MEMBOT: Memory-Based Robot in Intermittent POMDP