mlsanityData to 7 Oct 2026

Recurrent Off-policy Baselines for Memory-based Continuous Control

arXiv 2110.12628 ↗First posted 25 Oct 2021cs.LG cs.AI cs.ROcited by 19

Not on the map. Why?

Abstract (arXiv)

When the environment is partially observable (PO), a deep reinforcement learning (RL) agent must learn a suitable temporal representation of the entire history in addition to a strategy to control. This problem is not novel, and there have been model-free and model-based algorithms proposed for this problem. However, inspired by recent success in model-free image-based RL, we noticed the absence of a model-free baseline for history-based RL that (1) uses full history and (2) incorporates recent advances in off-policy continuous control. Therefore, we implement recurrent versions of DDPG, TD3, and SAC (RDPG, RTD3, and RSAC) in this work, evaluate them on short-term and long-term PO domains, and investigate key design choices. Our experiments show that RDPG and RTD3 can surprisingly fail on some domains and that RSAC is the most reliable, reaching near-optimal performance on nearly all domains. However, one task that requires systematic exploration still proved to be difficult, even for RSAC. These results show that model-free RL can learn good temporal representation using only reward signals; the primary difficulty seems to be computational cost and exploration. To facilitate future research, we have made our PyTorch implementation publicly available at https://github.com/zhihanyang2022/off-policy-continuous-control.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 23 of its 33 references that are in the map. First posted on arXiv 25 Oct 2021.

Builds on

23 of 33 references.

References · 23

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  4. 2015Continuous control with deep reinforcement learningcited by 3,279
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  6. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  7. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  8. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  9. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  10. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  11. 2015Deep Recurrent Q-Learning for Partially Observable MDPscited by 457
  12. 2015LSTM: A Search Space Odysseycited by 315
11 more
  1. 2019Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Modelcited by 228
  2. 2021Tianshou: a Highly Modularized Deep Reinforcement Learning Librarycited by 139
  3. 2018Deep Variational Reinforcement Learning for POMDPscited by 122
  4. 2016Playing FPS Games with Deep Reinforcement Learningcited by 95
  5. 2015Memory-based control with recurrent neural networkscited by 77
  6. 2021Memory-based Deep Reinforcement Learning for POMDPscited by 53
  7. 2019ChainerRL: A Deep Reinforcement Learning Librarycited by 51
  8. 2018On Improving Deep Reinforcement Learning for POMDPscited by 49
  9. 2019Variational Recurrent Models for Solving Partially Observable Control Taskscited by 46
  10. 2020Discriminative Particle Filter Reinforcement Learning for Complex Partial Observationscited by 27
  11. 2020Belief-Grounded Networks for Accelerated Robot Learning under Partial Observabilitycited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 7; 2024: 6; 2025: 4; 2026: 1.

19 papers.

Sort:

By area:

  1. 2023When Do Transformers Shine in RL? Decoupling Memory from Credit Assignmentcited by 62
  2. 2023POPGym: Benchmarking Partially Observable Reinforcement Learningcited by 59
  3. 2023AMAGO: Scalable In-Context Reinforcement Learning for Adaptive Agentscited by 36
  4. 2022Deep Transformer Q-Networks for Partially Observable Reinforcement Learningcited by 35
  5. 2024Equivariant Reinforcement Learning under Partial Observabilitycited by 17
  6. 2024AMAGO-2: Breaking the Multi-Task Barrier in Meta-Reinforcement Learning with Transformerscited by 14
  7. 2024Efficient Recurrent Off-Policy RL Requires a Context-Encoder-Specific Learning Ratecited by 10
  8. 2023Reinforcement Learning with Fast and Forgetful Memorycited by 8
  9. 2023Seq2Seq Imitation Learning for Tactile Feedback-based Manipulationcited by 6
  10. 2023PID-Inspired Inductive Biases for Deep Reinforcement Learning in Partially Observable Control Taskscited by 4
  11. 2024Intrinsic Rewards for Exploration without Harm from Observational Noise: A Simulation Study Based on the Free Energy Principlecited by 3
  12. 2024Residual Learning and Context Encoding for Adaptive Offline-to-Online Reinforcement Learningcited by 2
7 more
  1. 2025Sequential Monte Carlo for Policy Optimization in Continuous POMDPscited by 2
  2. 2023Robust Recurrent Reinforcement Learning under Evolving Hidden Disturbances with Application to Rover Wheel Slipcited by 1
  3. 2024MarineFormer: A Spatio-Temporal Attention Model for USV Navigation in Dynamic Marine Environmentscited by 1
  4. 2025Long-Horizon Model-Based Offline Reinforcement Learning Without Explicit Conservatismcited by 1
  5. 2026Reinforcement Learning for Control Systems with Time Delays: A Comprehensive Surveycited by 1
  6. 2025Distributed Detection of Adversarial Attacks in Multi-Agent Reinforcement Learning with Continuous Action Spacecited by 0
  7. 2025Curiosity-Driven Development of Action and Language in Robots Through Self-Explorationcited by 0