mlsanityData to 7 Oct 2026

Memory-based Deep Reinforcement Learning for POMDPs

arXiv 2102.12344 ↗First posted 24 Feb 2021cs.LG cs.AI cs.ROcited by 53

Not on the map. Why?

Abstract (arXiv)

A promising characteristic of Deep Reinforcement Learning (DRL) is its capability to learn optimal policy in an end-to-end manner without relying on feature engineering. However, most approaches assume a fully observable state space, i.e. fully observable Markov Decision Processes (MDPs). In real-world robotics, this assumption is unpractical, because of issues such as sensor sensitivity limitations and sensor noise, and the lack of knowledge about whether the observation design is complete or not. These scenarios lead to Partially Observable MDPs (POMDPs). In this paper, we propose Long-Short-Term-Memory-based Twin Delayed Deep Deterministic Policy Gradient (LSTM-TD3) by introducing a memory component to TD3, and compare its performance with other DRL algorithms in both MDPs and POMDPs. Our results demonstrate the significant advantages of the memory component in addressing POMDPs, including the ability to handle missing and noisy observation data.

Key facts. Cited by 53 papers in the map (data through 7 Oct 2026). Builds on 26 of its 51 references that are in the map. First posted on arXiv 24 Feb 2021.

Builds on

26 of 51 references.

References · 26

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2015Continuous control with deep reinforcement learningcited by 3,279
  5. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  6. 2015Trust Region Policy Optimizationcited by 2,503
  7. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  8. 2016OpenAI Gymcited by 1,684
  9. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  10. 2015Deep Recurrent Q-Learning for Partially Observable MDPscited by 457
  11. 2016Benchmarking Deep Reinforcement Learning for Continuous Controlcited by 361
  12. 2016Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updatescited by 342
14 more
  1. 2017Socially Aware Motion Planning with Deep Reinforcement Learningcited by 128
  2. 2016Playing FPS Games with Deep Reinforcement Learningcited by 95
  3. 2017Virtual-to-real Deep Reinforcement Learning: Continuous Control of Mobile Robots for Mapless Navigationcited by 90
  4. 2015Memory-based control with recurrent neural networkscited by 77
  5. 2018On Improving Deep Reinforcement Learning for POMDPscited by 49
  6. 2015Towards Vision-Based Deep Reinforcement Learning for Robotic Motion Controlcited by 33
  7. 2018Reinforcement Learning of Active Vision for Manipulating Objects under Occlusionscited by 25
  8. 2019Exploring applications of deep reinforcement learning for real-world autonomous driving systemscited by 18
  9. 2020The Effect of Multi-step Methods on Overestimation in Deep Reinforcement Learningcited by 14
  10. 2014Robotic manipulation of multiple objects as a POMDPcited by 9
  11. 2017Robot gains Social Intelligence through Multimodal Deep Reinforcement Learningcited by 7
  12. 2017Recurrent Deterministic Policy Gradient Method for Bipedal Locomotion on Rough Terrain Challengecited by 7
  13. 2018Distributed Deep Reinforcement Learning: Learn how to play Atari games in 21 minutescited by 6
  14. 2019Q-learning for POMDP: An application to learning locomotion gaitscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 9; 2023: 18; 2024: 7; 2025: 11; 2026: 4.

53 papers.

Sort:

By area:

  1. 2021Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDPscited by 115
  2. 2023When Do Transformers Shine in RL? Decoupling Memory from Credit Assignmentcited by 62
  3. 2025Memory, Benchmark & Robots: A Benchmark for Solving Complex Tasks with Reinforcement Learningcited by 34
  4. 2022Learning in Observable POMDPs, without Computationally Intractable Oraclescited by 30
  5. 2021Recurrent Off-policy Baselines for Memory-based Continuous Controlcited by 19
  6. 2022Leveraging Fully Observable Policies for Learning under Partial Observabilitycited by 17
  7. 2024Equivariant Reinforcement Learning under Partial Observabilitycited by 17
  8. 2023Memory Gym: Towards Endless Tasks to Benchmark Memory Capabilities of Agentscited by 16
  9. 2022Latent State Marginalization as a Low-cost Approach for Improving Explorationcited by 13
  10. 2023POMDP inference and robust solution via deep reinforcement learning: An application to railway optimal maintenancecited by 13
  11. 2023Provable Representation with Efficient Planning for Partial Observable Reinforcement Learningcited by 10
  12. 2024Real-Time Recurrent Learning using Trace Units in Reinforcement Learningcited by 10
38 more
  1. 2022Generalization, Mayhems and Limits in Recurrent Proximal Policy Optimizationcited by 9
  2. 2022A Machine with Short-Term, Episodic, and Semantic Memory Systemscited by 9
  3. 2023SMARLA: A Safety Monitoring Approach for Deep Reinforcement Learning Agentscited by 9
  4. 2023Bridging the Sim-to-Real Gap from the Information Bottleneck Perspectivecited by 8
  5. 2022Deep Learning for Wireless Networked Systems: a joint Estimation-Control-Scheduling Approachcited by 7
  6. 2023A Platform-Agnostic Deep Reinforcement Learning Framework for Effective Sim2Real Transfer towards Autonomous Drivingcited by 7
  7. 2023Seq2Seq Imitation Learning for Tactile Feedback-based Manipulationcited by 6
  8. 2023ODE-based Recurrent Model-free Reinforcement Learning for POMDPscited by 6
  9. 2025Zero-Shot Reinforcement Learning Under Partial Observabilitycited by 6
  10. 2023PID-Inspired Inductive Biases for Deep Reinforcement Learning in Partially Observable Control Taskscited by 4
  11. 2023Weathering Ongoing Uncertainty: Learning and Planning in a Time-Varying Partially Observable Environmentcited by 4
  12. 2026MINT: Minimal Information Neuro-Symbolic Tree for Objective-Driven Knowledge-Gap Reasoning and Active Elicitationcited by 4
  13. 2026SoK: The Pitfalls of Deep Reinforcement Learning for Cybersecuritycited by 4
  14. 2022Spatial-temporal recurrent reinforcement learning for autonomous shipscited by 3
  15. 2024An Empirical Study on the Power of Future Prediction in Partially Observable Environmentscited by 3
  16. 2025Guided Policy Optimization under Partial Observabilitycited by 3
  17. 20232-Level Reinforcement Learning for Ships on Inland Waterways: Path Planning and Followingcited by 2
  18. 2024Self-organized free-flight arrival for urban air mobilitycited by 2
  19. 2025Sequential Monte Carlo for Policy Optimization in Continuous POMDPscited by 2
  20. 2023Model-free Motion Planning of Autonomous Agents for Complex Tasks in Partially Observable Environmentscited by 1
  21. 2023Robust Recurrent Reinforcement Learning under Evolving Hidden Disturbances with Application to Rover Wheel Slipcited by 1
  22. 2023An investigation of belief-free DRL and MCTS for inspection and maintenance planningcited by 1
  23. 2024Chaos-based reinforcement learning with TD3cited by 1
  24. 2025Depth-Constrained ASV Navigation with Deep RL and Limited Sensingcited by 1
  25. 2026Memory Retention Is Not Enough to Master Memory Tasks in Reinforcement Learningcited by 1
  26. 2021Blockwise Sequential Model Learning for Partially Observable Reinforcement Learningcited by 0
  27. 2021Missing Velocity in Dynamic Obstacle Avoidance based on Deep Reinforcement Learningcited by 0
  28. 2022A Deep Learning Approach To Estimation Using Measurements Received Over a Networkcited by 0
  29. 2022Multi-Step First: A Lightweight Deep Reinforcement Learning Strategy for Robust Continuous Control with Partial Observabilitycited by 0
  30. 2023Deep Reinforcement Learning for mmWave Initial Beam Alignmentcited by 0
  31. 2023Observer-Feedback-Feedforward Controller Structures in Reinforcement Learningcited by 0
  32. 2023Two-step dynamic obstacle avoidancecited by 0
  33. 2024An Improved Strategy for Blood Glucose Control Using Multi-Step Deep Reinforcement Learningcited by 0
  34. 2024When to Localize? A Risk-Constrained Reinforcement Learning Approachcited by 0
  35. 2025A Convolution and Attention Based Encoder for Reinforcement Learning under Partial Observabilitycited by 0
  36. 2025Decoupled Hierarchical Reinforcement Learning with State Abstraction for Discrete Gridscited by 0
  37. 2025Optimizing Drivers' Discount Order Acceptance Strategies: A Policy-Improved Deep Deterministic Policy Gradient Frameworkcited by 0
  38. 2025Success in Humanoid Reinforcement Learning under Partial Observationcited by 0

50 most cited of 53 All 53 (JSON)