Data to 7 Oct 2026

Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDPs

arXiv 2110.05038 ↗First posted 11 Oct 2021cs.LG cs.AI cs.ROcited by 115

Not on the map. Why?

Abstract (arXiv)

Many problems in RL, such as meta-RL, robust RL, generalization in RL, and temporal credit assignment, can be cast as POMDPs. In theory, simply augmenting model-free RL with memory-based architectures, such as recurrent neural networks, provides a general approach to solving all types of POMDPs. However, prior work has found that such recurrent model-free RL methods tend to perform worse than more specialized algorithms that are designed for specific types of POMDPs. This paper revisits this claim. We find that careful architecture and hyperparameter decisions can often yield a recurrent model-free implementation that performs on par with (and occasionally substantially better than) more sophisticated recent techniques. We compare to 21 environments from 6 prior specialized methods and find that our implementation achieves greater sample efficiency and asymptotic performance than these methods on 18/21 environments. We also release a simple and efficient implementation of recurrent model-free RL for future work to use as a baseline for POMDPs.

Key facts. Cited by 115 papers in the map (data through 7 Oct 2026). Builds on 79 of its 137 references that are in the map. First posted on arXiv 11 Oct 2021.

Builds on

79 of 137 references.

References · 79

2014Adam: A Method for Stochastic Optimizationcited by 36,960
2017Proximal Policy Optimization Algorithmscited by 14,441
2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
2015Trust Region Policy Optimizationcited by 2,503
2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,341
2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
2018Soft Actor-Critic Algorithms and Applicationscited by 1,179

38 more

2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
2018Recurrent World Models Facilitate Policy Evolutioncited by 761
2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 490
2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 478
2015Deep Recurrent Q-Learning for Partially Observable MDPscited by 457
2017Robust Adversarial Reinforcement Learningcited by 439
2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 400
2019Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variablescited by 372
2016Learning to reinforcement learncited by 369
2016Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learningcited by 351
2018Quantifying Generalization in Reinforcement Learningcited by 345
2017Adversarial Attacks on Neural Network Policiescited by 314
2015Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Imagescited by 266
2019Stabilizing Transformers for Reinforcement Learningcited by 262
2020Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observationscited by 235
2019Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Modelcited by 228
2019VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learningcited by 210
2019Adversarial Policies: Attacking Deep Reinforcement Learningcited by 209
2016Learning to Navigate in Complex Environmentscited by 192
2018A Study on Overfitting in Deep Reinforcement Learningcited by 170
2019Action Robust Reinforcement Learning and Applications in Continuous Controlcited by 167
2017Tactics of Adversarial Attack on Deep Reinforcement Learning Agentscited by 163
2018RUDDER: Return Decomposition for Delayed Rewardscited by 162
2019Soft Actor-Critic for Discrete Action Settingscited by 161
2017Robust Deep Reinforcement Learning with Adversarial Attackscited by 152
2021Tianshou: a Highly Modularized Deep Reinforcement Learning Librarycited by 139
2016EPOpt: Learning Robust Neural Network Policies Using Model Ensemblescited by 136
2021Robust Reinforcement Learning on State Observations with Learned Optimal Adversarycited by 133
2018Assessing Generalization in Deep Reinforcement Learningcited by 132
2018Deep Variational Reinforcement Learning for POMDPscited by 122
2018Generalization and Regularization in DQNcited by 117
2019Network Randomization: A Simple Technique for Generalization in Deep Reinforcement Learningcited by 102
2019Generalization in Reinforcement Learning with Selective Noise Injection and Information Bottleneckcited by 101
2019Observational Overfitting in Reinforcement Learningcited by 90
2018A Dissection of Overfitting and Generalization in Continuous Reinforcement Learningcited by 86
2021Decoupling Value and Policy for Generalization in Reinforcement Learningcited by 81

50 most cited of 79 All 79 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 8; 2023: 26; 2024: 33; 2025: 22; 2026: 26.

115 papers.

Sort:

By area:

2022In-context Reinforcement Learning with Algorithm Distillationcited by 186
2023Structured State Space Models for In-Context Reinforcement Learningcited by 109
2023Large Language Models as Generalizable Policies for Embodied Taskscited by 86
2024Dense Reward for Free in Reinforcement Learning from Human Feedbackcited by 77
2023When Do Transformers Shine in RL? Decoupling Memory from Credit Assignmentcited by 62
2023POPGym: Benchmarking Partially Observable Reinforcement Learningcited by 59
2024Bridging State and History Representations: Understanding Self-Predictive RLcited by 52
2023AMAGO: Scalable In-Context Reinforcement Learning for Adaptive Agentscited by 36
2022Deep Transformer Q-Networks for Partially Observable Reinforcement Learningcited by 35
2025Memory, Benchmark & Robots: A Benchmark for Solving Complex Tasks with Reinforcement Learningcited by 34
2022Learning in Observable POMDPs, without Computationally Intractable Oraclescited by 30
2025AVA-VLA: Improving Vision-Language-Action models with Active Visual Attentioncited by 22

38 more

2023Reinforcement Learning in the Era of LLMs: What is Essential? What is needed? An RL Perspective on RLHF, Prompting, and Beyondcited by 20
2022Leveraging Fully Observable Policies for Learning under Partial Observabilitycited by 17
2024Equivariant Reinforcement Learning under Partial Observabilitycited by 17
2023TGRL: An Algorithm for Teacher Guided Reinforcement Learningcited by 16
2023Recurrent Hypernetworks are Surprisingly Strong in Meta-RLcited by 15
2024Diffusion Spectral Representation for Reinforcement Learningcited by 15
2024AMAGO-2: Breaking the Multi-Task Barrier in Meta-Reinforcement Learning with Transformerscited by 14
2024MAMBA: an Effective World Model Approach for Meta-Reinforcement Learningcited by 13
2025Normalizing Flows are Capable Models for Continuous Controlcited by 13
2024Do Transformer World Models Give Better Policy Gradients?cited by 10
2024Efficient Recurrent Off-Policy RL Requires a Context-Encoder-Specific Learning Ratecited by 10
2024Real-Time Recurrent Learning using Trace Units in Reinforcement Learningcited by 10
2025AssistanceZero: Scalably Solving Assistance Gamescited by 10
2023ContraBAR: Contrastive Bayes-Adaptive Deep RLcited by 9
2023Accountability in Offline Reinforcement Learning: Explaining Decisions with a Corpus of Examplescited by 9
2024Rethinking Transformers in Solving POMDPscited by 9
2025Partially Observable Reinforcement Learning with Memory Tracescited by 9
2023Reinforcement Learning with Fast and Forgetful Memorycited by 8
2024Recurrent Natural Policy Gradient for POMDPscited by 8
2024Artificial Generational Intelligence: Cultural Accumulation in Reinforcement Learningcited by 8
2024Stable Hadamard Memory: Revitalizing Memory-Augmented Agents for Reinforcement Learningcited by 8
2022Meta-Learning Parameterized Skillscited by 7
2023Graph schemas as abstractions for transfer learning, inference, and planningcited by 7
2023Improving Intrinsic Exploration by Creating Stationary Objectivescited by 7
2024Symmetry-aware Reinforcement Learning for Robotic Assembly under Partial Observability with a Soft Wristcited by 7
2024Mitigating Partial Observability in Sequential Decision Processes via the Lambda Discrepancycited by 7
2024Pessimistic Iterative Planning with RNNs for Robust POMDPscited by 7
2022Disentangling Abstraction from Statistical Pattern Matching in Human and Machine Learningcited by 6
2023Seq2Seq Imitation Learning for Tactile Feedback-based Manipulationcited by 6
2023ODE-based Recurrent Model-free Reinforcement Learning for POMDPscited by 6
2023Real-Time Recurrent Reinforcement Learningcited by 6
2025Zero-Shot Reinforcement Learning Under Partial Observabilitycited by 6
2025Benchmarking Partial Observability in Reinforcement Learning with a Suite of Memory-Improvable Domainscited by 6
2022Time-Varying Propensity Score to Bridge the Gap between the Past and Presentcited by 5
2023A Deep Recurrent-Reinforcement Learning Method for Intelligent AutoScaling of Serverless Functionscited by 5
2024Uncertainty Representations in State-Space Layers for Deep Reinforcement Learning under Partial Observabilitycited by 5
2024Unraveling the Complexity of Memory in RL Agents: an Approach for Classification and Evaluationcited by 5
2023Active hypothesis testing in unknown environments using recurrent neural networks and model free reinforcement learningcited by 4

50 most cited of 115 All 115 (JSON)