mlsanityData to 7 Oct 2026

Offline RL Policies Should be Trained to be Adaptive

arXiv 2207.02200 ↗First posted 5 Jul 2022cs.LG cs.AI stat.MLcited by 52

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline RL algorithms must account for the fact that the dataset they are provided may leave many facets of the environment unknown. The most common way to approach this challenge is to employ pessimistic or conservative methods, which avoid behaviors that are too dissimilar from those in the training dataset. However, relying exclusively on conservatism has drawbacks: performance is sensitive to the exact degree of conservatism, and conservative objectives can recover highly suboptimal policies. In this work, we propose that offline RL methods should instead be adaptive in the presence of uncertainty. We show that acting optimally in offline RL in a Bayesian sense involves solving an implicit POMDP. As a result, optimal policies for offline RL must be adaptive, depending not just on the current state but rather all the transitions seen so far during evaluation.We present a model-free algorithm for approximating this optimal adaptive policy, and demonstrate the efficacy of learning such adaptive policies in offline RL benchmarks.

Key facts. Cited by 52 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 25 of its 42 references that are in the map. First posted on arXiv 5 Jul 2022.

Builds on

25 of 42 references.

Closest ancestors · 9

Same direction, newest first

  1. 2022Why So Pessimistic? Estimating Uncertainties for Offline RL through Ensembles, and Why Their Independence Matters27 May
  2. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  3. 2021A Workflow for Offline Model-Free Robotic Reinforcement Learning22 Sep
  4. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
  5. 2020Is Pessimism Provably Efficient for Offline RL?30 Dec
  6. 2020EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL21 Jul
  7. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  8. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  9. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr

Other references · 16

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  4. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  5. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  6. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  7. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  8. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  9. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
  10. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 478
  11. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 278
  12. 2013(More) Efficient Reinforcement Learning via Posterior Samplingcited by 233
4 more
  1. 2019VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learningcited by 210
  2. 2020SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learningcited by 150
  3. 2021Why Generalization in RL is Difficult: Epistemic POMDPs and Implicit Partial Observabilitycited by 102
  4. 2019If MaxEnt RL is the Answer, What is the Question?cited by 27

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 5; 2023: 15; 2024: 12; 2025: 11; 2026: 9.

52 papers.

Sort:

By area:

  1. 2022Is Conditional Generative Modeling all you need for Decision-Making?cited by 504
  2. 2023Policy Expansion for Bridging Offline-to-Online Reinforcement Learningcited by 103
  3. 2023Anti-Exploration by Random Network Distillationcited by 49
  4. 2023Adaptive Policy Learning for Offline-to-Online Reinforcement Learningcited by 40
  5. 2022Semi-Supervised Offline Reinforcement Learning with Action-Free Trajectoriescited by 27
  6. 2023Train Once, Get a Family: State-Adaptive Balances for Offline-to-Online Reinforcement Learningcited by 26
  7. 2022Law Informs Code: A Legal Informatics Approach to Aligning Artificial Intelligence with Humanscited by 23
  8. 2022One Risk to Rule Them All: A Risk-Sensitive Perspective on Model-Based Offline Reinforcement Learningcited by 21
  9. 2022Confidence-Conditioned Value Functions for Offline Reinforcement Learningcited by 20
  10. 2023Learning to Discern: Imitating Heterogeneous Human Demonstrations with Preference and Representation Learningcited by 14
  11. 2024Bayesian Design Principles for Offline-to-Online Reinforcement Learningcited by 14
  12. 2023Unsupervised Behavior Extraction via Random Intent Priorscited by 13
38 more
  1. 2023A Survey of Demonstration Learningcited by 12
  2. 2023Instructed Diffuser with Temporal Condition Guidance for Offline Reinforcement Learningcited by 12
  3. 2023Efficient Planning with Latent Diffusioncited by 11
  4. 2024Bayes Adaptive Monte Carlo Tree Search for Offline Model-based Reinforcement Learningcited by 11
  5. 2023Design from Policies: Conservative Test-Time Adaptation for Offline Policy Optimizationcited by 8
  6. 2024Exclusively Penalized Q-learning for Offline Reinforcement Learningcited by 8
  7. 2024Uncertainty-based Offline Variational Bayesian Reinforcement Learning for Robustness under Diverse Data Corruptionscited by 8
  8. 2025Beyond Markovian: Reflective Exploration via Bayes-Adaptive RL for LLM Reasoningcited by 8
  9. 2025Posterior Behavioral Cloning: Pretraining BC Policies for Efficient RL Finetuningcited by 8
  10. 2024Solving Continual Offline RL through Selective Weights Activation on Aligned Spacescited by 6
  11. 2023Bridging Imitation and Online Reinforcement Learning: An Optimistic Talecited by 5
  12. 2023Katakomba: Tools and Benchmarks for Data-Driven NetHackcited by 5
  13. 2024An Offline Adaptation Framework for Constrained Multi-Objective Reinforcement Learningcited by 5
  14. 2023Budgeting Counterfactual for Offline RLcited by 4
  15. 2025Behavioral Exploration: Learning to Explore via In-Context Adaptationcited by 4
  16. 2024Towards Fast Safe Online Reinforcement Learning via Policy Finetuningcited by 3
  17. 2025Wavelet Fourier Diffuser: Frequency-Aware Diffusion Model for Reinforcement Learningcited by 3
  18. 2024Debiased Offline Representation Learning for Fast Online Adaptation in Non-stationary Dynamicscited by 2
  19. 2024Continual Diffuser (CoD): Mastering Continual Offline Reinforcement Learning with Experience Rehearsalcited by 2
  20. 2025Analytic Energy-Guided Policy Optimization for Offline Reinforcement Learningcited by 2
  21. 2025Goal-Oriented Skill Abstraction for Offline Multi-Task Reinforcement Learningcited by 2
  22. 2026Model-based Offline RL via Robust Value-Aware Model Learning with Implicitly Differentiable Adaptive Weightingcited by 2
  23. 2026Robust Regularized Policy Iteration under Transition Uncertaintycited by 2
  24. 2025State Combinatorial Generalization In Decision Making With Conditional Diffusion Modelscited by 1
  25. 2025Long-Horizon Model-Based Offline Reinforcement Learning Without Explicit Conservatismcited by 1
  26. 2026Epistemic Robust Offline Reinforcement Learningcited by 1
  27. 2026Offline Policy Optimization with Posterior Samplingcited by 1
  28. 2023Risk-Sensitive and Robust Model-Based Reinforcement Learning and Planningcited by 0
  29. 2023Automatic Trade-off Adaptation in Offline RLcited by 0
  30. 2024Augmenting Offline RL with Unlabeled Datacited by 0
  31. 2024Disentangled Representations for Causal Cognitioncited by 0
  32. 2024Streetwise Agents: Empowering Offline RL Policies to Outsmart Exogenous Stochastic Disturbances in RTCcited by 0
  33. 2025Decision Flow Policy Optimizationcited by 0
  34. 2025Automated Construction of Artificial Lattice Structures with Designer Electronic Statescited by 0
  35. 2025Detecting Model Drifts in Non-Stationary Environment Using Edit Operation Measurescited by 0
  36. 2026Lyapunov-Guided Self-Alignment: Test-Time Adaptation for Offline Safe Reinforcement Learningcited by 0
  37. 2026COOPO: Cyclic Offline-Online Policy Optimization Algorithmcited by 0
  38. 2026Regularized Offline Policy Optimization with Posterior Hybrid Bayesian Beliefcited by 0

50 most cited of 52 All 52 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Offline Policy Evaluation for Reinforcement Learning with Adaptively Collected Data
  2. 2022Optimal Conservative Offline RL with General Function Approximation via Augmented Lagrangian
  3. 2023Bridging Distributionally Robust Learning and Offline RL: An Approach to Mitigate Distribution Shift and Partial Data Coverage
  4. 2023Offline Data Enhanced On-Policy Policy Gradient with Provable Guarantees
  5. 2025On the Statistical Complexity for Offline and Low-Adaptive Reinforcement Learning with Structures
  6. 2025Evaluation-Time Policy Switching for Offline Reinforcement Learning
  7. 2025Fully Offline Reinforcement Learning
  8. 2026Conservative Query and Adaptive Regularization for Offline RL Under Uncertainty Estimation