mlsanityData to 14 Oct 2026

Deep Recurrent Q-Learning for Partially Observable MDPs

arXiv 1507.06527 ↗First posted 23 Jul 2015cs.LGcited by 465

Not on the map. Why?

Cited by 99 in Deep reinforcement learning methods, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Deep Reinforcement Learning has yielded proficient controllers for complex tasks. However, these controllers have limited memory and rely on being able to perceive the complete game screen at each decision point. To address these shortcomings, this article investigates the effects of adding recurrency to a Deep Q-Network (DQN) by replacing the first post-convolutional fully-connected layer with a recurrent LSTM. The resulting textitDeep Recurrent Q-Network (DRQN), although capable of seeing only a single frame at each timestep, successfully integrates information through time and replicates DQN's performance on standard Atari games and partially observed equivalents featuring flickering game screens. Additionally, when trained with partial observations and evaluated with incrementally more complete observations, DRQN's performance scales as a function of observability. Conversely, when trained with full observations and evaluated with partial observations, DRQN's performance degrades less than DQN's. Thus, given the same length of history, recurrency is a viable alternative to stacking a history of frames in the DQN's input layer and while recurrency confers no systematic advantage when learning to play the game, the recurrent net can better adapt at evaluation time if the quality of observations changes.

Key facts. Cited by 465 papers in the map (data through 14 Oct 2026). First posted on arXiv 23 Jul 2015.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2019: 1; 2020: 1; 2021: 88; 2022: 86; 2023: 83; 2024: 55; 2025: 79; 2026: 72.

465 papers.

Sort:

By area:

  1. 2023LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learningcited by 1,410
  2. 2021Robust Reinforcement Learning on State Observations with Learned Optimal Adversarycited by 135
  3. 2021Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDPscited by 122
  4. 2021Stabilizing Deep Q-Learning with ConvNets and Vision Transformers under Data Augmentationcited by 105
  5. 2023PIRLNav: Pretraining with Imitation and RL Finetuning for ObjectNavcited by 93
  6. 2021S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learningcited by 85
  7. 2021Federated Reinforcement Learning: Techniques, Applications, and Open Challengescited by 81
  8. 2020Fighting Copycat Agents in Behavioral Cloning from Observation Historiescited by 71
  9. 2022A Survey on Model-based Reinforcement Learningcited by 66
  10. 2021Cooperative Exploration for Multi-Agent Deep Reinforcement Learningcited by 57
  11. 2022Safe Reinforcement Learning via Shielding under Partial Observabilitycited by 56
  12. 2023Enhancing Deep Reinforcement Learning: A Tutorial on Generative Diffusion Models in Network Optimizationcited by 56
38 more
  1. 2021Memory-based Deep Reinforcement Learning for POMDPscited by 54
  2. 2024Bridging State and History Representations: Understanding Self-Predictive RLcited by 54
  3. 2023A Survey on Transformers in Reinforcement Learningcited by 50
  4. 2024An Introduction to Centralized Training for Decentralized Execution in Cooperative Multi-Agent Reinforcement Learningcited by 50
  5. 2021Adversarially Guided Actor-Criticcited by 47
  6. 2025HAMLET: Switch your Vision-Language-Action Model into a History-Aware Policycited by 46
  7. 2021ELLA: Exploration through Learned Language Abstractioncited by 42
  8. 2021Deep Multiagent Reinforcement Learning: Challenges and Directionscited by 42
  9. 2023AMAGO: Scalable In-Context Reinforcement Learning for Adaptive Agentscited by 41
  10. 2022Provable Reinforcement Learning with a Short-Term Memorycited by 40
  11. 2021UPDeT: Universal Multi-agent Reinforcement Learning via Policy Decoupling with Transformerscited by 39
  12. 2022History Compression via Language Models in Reinforcement Learningcited by 39
  13. 2022Deep Transformer Q-Networks for Partially Observable Reinforcement Learningcited by 36
  14. 2022Scalable Multi-Agent Reinforcement Learning through Intelligent Information Aggregationcited by 36
  15. 2022On Transforming Reinforcement Learning by Transformer: The Development Trajectorycited by 36
  16. 2025Memory, Benchmark & Robots: A Benchmark for Solving Complex Tasks with Reinforcement Learningcited by 36
  17. 2022MASER: Multi-Agent Reinforcement Learning with Subgoals Generated from Experience Replay Buffercited by 34
  18. 2023Multi-agent Reinforcement Learning: A Comprehensive Surveycited by 33
  19. 2021Action-Sufficient State Representation Learning for Control with Structural Constraintscited by 31
  20. 2025ReflAct: World-Grounded Decision Making in LLM Agents via Goal-State Reflectioncited by 31
  21. 2023OCAtari: Object-Centric Atari 2600 Reinforcement Learning Environmentscited by 30
  22. 2021Deep Reinforcement Learning Versus Evolution Strategies: A Comparative Surveycited by 28
  23. 2025A Comprehensive Survey on Multi-Agent Cooperative Decision-Making: Scenarios, Approaches, Challenges and Perspectivescited by 27
  24. 2022Planning in Observable POMDPs in Quasipolynomial Timecited by 25
  25. 2022A Comprehensive Survey of Data Augmentation in Visual Reinforcement Learningcited by 25
  26. 2025AVA-VLA: Improving Vision-Language-Action models with Active Visual Attentioncited by 25
  27. 2019Learning Belief Representations for Imitation Learning in POMDPscited by 24
  28. 2022Quantum deep recurrent reinforcement learningcited by 24
  29. 2021Time Discretization-Invariant Safe Action Repetition for Policy Gradient Methodscited by 23
  30. 2022EAGER: Asking and Answering Questions for Automatic Reward Shaping in Language-guided RLcited by 23
  31. 2025Continuous Thought Machinescited by 22
  32. 2021Structured World Belief for Reinforcement Learning in POMDPcited by 21
  33. 2021Recurrent Off-policy Baselines for Memory-based Continuous Controlcited by 21
  34. 2022Towards Comprehensive Testing on the Robustness of Cooperative Multi-agent Reinforcement Learningcited by 21
  35. 2022Reinforcement learning on graphs: A surveycited by 20
  36. 2021Sublinear Regret for Learning POMDPscited by 19
  37. 2021HAVEN: Hierarchical Cooperative Multi-Agent Reinforcement Learning with Dual Coordination Mechanismcited by 19
  38. 2022Generalization in Cooperative Multi-Agent Systemscited by 19

50 most cited of 465 All 465 (JSON)