mlsanityData to 7 Oct 2026

Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning

arXiv 2002.08396 ↗First posted 19 Feb 2020cs.LG cs.RO stat.MLcited by 213

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Off-policy reinforcement learning algorithms promise to be applicable in settings where only a fixed data-set (batch) of environment interactions is available and no new experience can be acquired. This property makes these algorithms appealing for real world problems such as robot control. In practice, however, standard off-policy algorithms fail in the batch setting for continuous control. In this paper, we propose a simple solution to this problem. It admits the use of data generated by arbitrary behavior policies and uses a learned prior -- the advantage-weighted behavior model (ABM) -- to bias the RL policy towards actions that have previously been executed and are likely to be successful on the new task. Our method can be seen as an extension of recent work on batch-RL that enables stable learning from conflicting data-sources. We find improvements on competitive baselines in a variety of RL tasks -- including standard continuous control benchmarks and multi-task learning for simulated and real-world robots.

Key facts. Cited by 213 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 17 of its 33 references that are in the map. First posted on arXiv 19 Feb 2020.

Builds on

17 of 33 references.

Closest ancestors · 1

Same direction, newest first

  1. 2019An Optimistic Perspective on Offline Reinforcement Learning10 Jul

Other references · 16

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2015Trust Region Policy Optimizationcited by 2,503
  5. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  6. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  7. 2018DeepMind Control Suitecited by 935
  8. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  9. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  10. 2018Maximum a Posteriori Policy Optimisationcited by 318
  11. 2017Distral: Robust Multitask Reinforcement Learningcited by 231
  12. 2018Learning by Playing - Solving Sparse Reward Tasks from Scratchcited by 164
4 more
  1. 2015Learning Continuous Control Policies by Stochastic Value Gradientscited by 156
  2. 2016Sequence Tutor: Conservative Fine-Tuning of Sequence Generation Models with KL-controlcited by 100
  3. 2019Information asymmetry in KL-regularized RLcited by 53
  4. 2019Exploiting Hierarchy for Learning and Transfer in KL-regularized RLcited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 12; 2021: 52; 2022: 57; 2023: 40; 2024: 37; 2025: 7; 2026: 8.

213 papers.

Sort:

By area:

  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  2. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  3. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  4. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  5. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  6. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
  7. 2020MOPO: Model-based Offline Policy Optimizationcited by 630
  8. 2021COMBO: Conservative Offline Model-Based Policy Optimizationcited by 365
  9. 2020Is Pessimism Provably Efficient for Offline RL?cited by 362
  10. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimismcited by 275
  11. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 230
  12. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problemscited by 228
38 more
  1. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemblecited by 211
  2. 2023Efficient Diffusion Policies for Offline Reinforcement Learningcited by 151
  3. 2021Offline RL Without Off-Policy Evaluationcited by 150
  4. 2023AdaptDiffuser: Diffusion Models as Adaptive Self-evolving Plannerscited by 147
  5. 2023Supervised Pretraining Can Learn In-Context Reinforcement Learningcited by 137
  6. 2021Goal-Conditioned Reinforcement Learning with Imagined Subgoalscited by 129
  7. 2023Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functionscited by 129
  8. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learningcited by 125
  9. 2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learningcited by 115
  10. 2020Hyperparameter Selection for Offline Reinforcement Learningcited by 113
  11. 2021Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learningcited by 109
  12. 2022Q-learning Decision Transformer: Leveraging Dynamic Programming for Conditional Sequence Modelling in Offline RLcited by 103
  13. 2020Offline Reinforcement Learning from Images with Latent Space Modelscited by 94
  14. 2021Online and Offline Reinforcement Learning by Planning with a Learned Modelcited by 90
  15. 2021Benchmarks for Deep Off-Policy Evaluationcited by 89
  16. 2022When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning?cited by 86
  17. 2021Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learningcited by 78
  18. 2022Pre-Training for Robots: Offline RL Enables Learning New Tasks from a Handful of Trialscited by 78
  19. 2021Demonstration-Guided Reinforcement Learning with Learned Skillscited by 73
  20. 2020Offline Learning from Demonstrations and Unlabeled Experiencecited by 72
  21. 2022Offline Q-Learning on Diverse Multi-Task Data Both Scales And Generalizescited by 72
  22. 2022COptiDICE: Offline Constrained Reinforcement Learning via Stationary Distribution Correction Estimationcited by 69
  23. 2021Risk-Averse Offline Reinforcement Learningcited by 65
  24. 2021Conservative Data Sharing for Multi-Task Offline Reinforcement Learningcited by 64
  25. 2022Watch and Match: Supercharging Imitation with Regularized Optimal Transportcited by 64
  26. 2021NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learningcited by 63
  27. 2021No RL, No Simulation: Learning to Navigate without Navigatingcited by 63
  28. 2020RL Unplugged: A Suite of Benchmarks for Offline Reinforcement Learningcited by 57
  29. 2022Reinforcement Learning with Sparse Rewards using Guidance from Offline Demonstrationcited by 54
  30. 2022Retrieval-Augmented Reinforcement Learningcited by 53
  31. 2021Corruption-Robust Offline Reinforcement Learningcited by 51
  32. 2021Modeling Strong and Human-Like Gameplay with KL-Regularized Searchcited by 50
  33. 2022Mastering the Game of No-Press Diplomacy via Human-Regularized Reinforcement Learning and Planningcited by 47
  34. 2021Efficient Deep Reinforcement Learning with Imitative Expert Priors for Autonomous Drivingcited by 46
  35. 2023Behavior Proximal Policy Optimizationcited by 43
  36. 2020Overcoming Model Bias for Robust Offline Deep Reinforcement Learningcited by 42
  37. 2022DARA: Dynamics-Aware Reward Augmentation in Offline Reinforcement Learningcited by 40
  38. 2021Baby Intuitions Benchmark (BIB): Discerning the goals, preferences, and actions of otherscited by 39

50 most cited of 213 All 213 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2021A Workflow for Offline Model-Free Robotic Reinforcement Learning
  2. 2023Harnessing Mixed Offline Reinforcement Learning Datasets via Trajectory Weighting
  3. 2022Behavior Prior Representation learning for Offline Reinforcement Learning
  4. 2020MOReL : Model-Based Offline Reinforcement Learning
  5. 2022Offline Robot Reinforcement Learning with Uncertainty-Guided Human Expert Sampling
  6. 2022Bridging the Gap Between Offline and Online Reinforcement Learning Evaluation Methodologies
  7. 2022A Unified Framework for Alternating Offline Model Training and Policy Learning
  8. 2023Finetuning Offline World Models in the Real World