mlsanityData to 14 Oct 2026

Hundreds Guide Millions: Adaptive Offline Reinforcement Learning with Expert Guidance

arXiv 2309.01448 ↗First posted 4 Sep 2023cs.LGcited by 13

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning (RL) optimizes the policy on a previously collected dataset without any interactions with the environment, yet usually suffers from the distributional shift problem. To mitigate this issue, a typical solution is to impose a policy constraint on a policy improvement objective. However, existing methods generally adopt a ``one-size-fits-all'' practice, i.e., keeping only a single improvement-constraint balance for all the samples in a mini-batch or even the entire offline dataset. In this work, we argue that different samples should be treated with different policy constraint intensities. Based on this idea, a novel plug-in approach named Guided Offline RL (GORL) is proposed. GORL employs a guiding network, along with only a few expert demonstrations, to adaptively determine the relative importance of the policy improvement and policy constraint for every sample. We theoretically prove that the guidance provided by our method is rational and near-optimal. Extensive experiments on various environments suggest that GORL can be easily installed on most offline RL algorithms with statistically significant performance improvements.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 48 of its 68 references that are in the map. First posted on arXiv 4 Sep 2023.

Builds on

48 of 68 references.

Closest ancestors · 8

Same direction, newest first

  1. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems2 Mar
  2. 2021RvS: What is Essential for Offline RL via Supervised Learning?20 Dec
  3. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  4. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  5. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  6. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  7. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  8. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 40

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  6. 2016OpenAI Gymcited by 1,736
  7. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  8. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  9. 2017Learning to Compare: Relation Network for Few-Shot Learningcited by 1,143
  10. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  11. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  12. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 743
28 more
  1. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 715
  2. 2018Learning to Reweight Examples for Robust Deep Learningcited by 675
  3. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  4. 2017MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labelscited by 653
  5. 2017End-to-end Driving via Conditional Imitation Learningcited by 485
  6. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 340
  7. 2016Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collectioncited by 316
  8. 2019Learning Latent Plans from Playcited by 316
  9. 2018Learning to Drive in a Daycited by 273
  10. 2017Meta Networkscited by 269
  11. 2017Deep Reinforcement Learning framework for Autonomous Drivingcited by 225
  12. 2015Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hourscited by 209
  13. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 188
  14. 2017Imitation from Observation: Learning to Imitate Behaviors from Raw Video via Context Translationcited by 167
  15. 2017Active Bias: Training More Accurate Neural Networks by Emphasizing High Variance Samplescited by 147
  16. 2018Learning Synergies between Pushing and Grasping with Self-supervised Deep Reinforcement Learningcited by 128
  17. 2019Training Agents using Upside-Down Reinforcement Learningcited by 107
  18. 2019Reward-Conditioned Policiescited by 102
  19. 2018Supervised Reinforcement Learning with Recurrent Neural Network for Dynamic Treatment Recommendationcited by 101
  20. 2020The Importance of Pessimism in Fixed-Dataset Policy Optimizationcited by 99
  21. 2019BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learningcited by 91
  22. 2017Deep Reinforcement Learning for Sepsis Treatmentcited by 71
  23. 2020Rewriting History with Inverse RL: Hindsight Inference for Policy Improvementcited by 60
  24. 2017A Reinforcement Learning Approach to Weaning of Mechanical Ventilation in Intensive Care Unitscited by 50
  25. 2021Efficient Deep Reinforcement Learning with Imitative Expert Priors for Autonomous Drivingcited by 47
  26. 2018Learning to Teach with Dynamic Loss Functionscited by 40
  27. 2017Fidelity-Weighted Learningcited by 19
  28. 2020Visual Perception Generalization for Vision-and-Language Navigation via Meta-Learningcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 2; 2025: 6; 2026: 1.

13 papers.

Sort:

By area:

  1. 2023Rank-DETR for High Quality Object Detectioncited by 50
  2. 2025ART: Anonymous Region Transformer for Variable Multi-Layer Transparent Image Generationcited by 37
  3. 2023Train Once, Get a Family: State-Adaptive Balances for Offline-to-Online Reinforcement Learningcited by 26
  4. 2024Efficient Diffusion Transformer with Step-wise Dynamic Attention Mediatorscited by 24
  5. 2023Understanding, Predicting and Better Resolving Q-Value Divergence in Offline-RLcited by 20
  6. 2024ACL-QL: Adaptive Conservative Level in Q-Learning for Offline Reinforcement Learningcited by 5
  7. 2023DOMAIN: MilDly COnservative Model-BAsed OfflINe Reinforcement Learningcited by 4
  8. 2025Linear Differential Vision Transformer: Learning Visual Contrasts via Pairwise Differentialscited by 4
  9. 2025Model-Based Offline Reinforcement Learning with Adversarial Data Augmentationcited by 2
  10. 2025Re:Frame -- Retrieving Experience From Associative Memorycited by 1
  11. 2025Adaptive Scaling of Policy Constraints for Offline Reinforcement Learningcited by 1
  12. 2025Mildly Conservative Regularized Evaluation for Offline Reinforcement Learningcited by 0
1 more
  1. 2026Beyond State-Wise Mirror Descent: Offline Policy Optimization with Parametric Policiescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Adaptive Policy Learning for Offline-to-Online Reinforcement Learning
  2. 2021NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning
  3. 2025A Clean Slate for Offline Reinforcement Learning
  4. 2020MOPO: Model-based Offline Policy Optimization
  5. 2025On the Statistical Complexity for Offline and Low-Adaptive Reinforcement Learning with Structures
  6. 2022Leveraging Offline Data in Online Reinforcement Learning
  7. 2020Expert-Supervised Reinforcement Learning for Offline Policy Learning and Evaluation
  8. 2023Improving and Benchmarking Offline Reinforcement Learning Algorithms