mlsanityData to 14 Oct 2026

Offline Reinforcement Learning with Behavioral Supervisor Tuning

arXiv 2404.16399 ↗First posted 25 Apr 2024cs.LG cs.AIcited by 3

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning (RL) algorithms are applied to learn performant, well-generalizing policies when provided with a static dataset of interactions. Many recent approaches to offline RL have seen substantial success, but with one key caveat: they demand substantial per-dataset hyperparameter tuning to achieve reported performance, which requires policy rollouts in the environment to evaluate; this can rapidly become cumbersome. Furthermore, substantial tuning requirements can hamper the adoption of these algorithms in practical domains. In this paper, we present TD3 with Behavioral Supervisor Tuning (TD3-BST), an algorithm that trains an uncertainty model and uses it to guide the policy to select actions within the dataset support. TD3-BST can learn more effective policies from offline datasets compared to previous methods and achieves the best performance across challenging benchmarks without requiring per-dataset tuning.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 34 of its 47 references that are in the map. First posted on arXiv 25 Apr 2024.

Builds on

34 of 47 references.

Closest ancestors · 18

Same direction, newest first

  1. 2023Uni-O4: Unifying Online and Offline Deep Reinforcement Learning with Multi-Step On-Policy Optimization6 Nov
  2. 2023Revisiting the Minimalist Approach to Offline Reinforcement Learning16 May
  3. 2023Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularization28 Mar
  4. 2023Anti-Exploration by Random Network Distillation31 Jan
  5. 2022Offline Reinforcement Learning with Closed-Form Policy Improvement Operators29 Nov
  6. 2022Offline RL With Realistic Datasets: Heteroskedasticity and Support Constraints2 Nov
  7. 2022CORL: Research-oriented Deep Offline Reinforcement Learning Library13 Oct
  8. 2022Value Memory Graph: A Graph-Structured World Model for Offline Reinforcement Learning9 Jun
  9. 2022Why So Pessimistic? Estimating Uncertainties for Offline RL through Ensembles, and Why Their Independence Matters27 May
  10. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  11. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  12. 2021Offline RL Without Off-Policy Evaluation16 Jun
6 more
  1. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  2. 2021Offline Reinforcement Learning as Anti-Exploration11 Jun
  3. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
  4. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  5. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  6. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 16

  1. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  2. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  3. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  4. 2018A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attackscited by 1,485
  5. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  6. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  7. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  8. 2014Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Imagescited by 776
  9. 2021Implicit Behavioral Cloningcited by 452
  10. 2019Reinforcement Learning in Healthcare: A Surveycited by 310
  11. 2018Learning to Navigate in Cities Without a Mapcited by 104
  12. 2021Model Selection for Offline Reinforcement Learning: Practical Considerations for Healthcare Settingscited by 68
4 more
  1. 2021Tactical Optimism and Pessimism for Deep Reinforcement Learningcited by 55
  2. 2022Offline Reinforcement Learning for Safer Blood Glucose Control in People with Type 1 Diabetescited by 23
  3. 2020When Ensembling Smaller Models is More Efficient than Single Large Modelscited by 20
  4. 2023Morse Neural Networks for Uncertainty Quantificationcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 1; 2026: 1.

3 papers.

  1. 2024Offline Model-Based Reinforcement Learning with Anti-Explorationcited by 1
  2. 2025Imagination-Limited Q-Learning for Offline Reinforcement Learningcited by 1
  3. 2026Conservative Query and Adaptive Regularization for Offline RL Under Uncertainty Estimationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025A Clean Slate for Offline Reinforcement Learning
  2. 2022Improving TD3-BC: Relaxed Policy Constraint for Offline Learning and Stable Online Fine-Tuning
  3. 2021Behavioral Priors and Dynamics Models: Improving Performance and Domain Transfer in Offline RL
  4. 2020Expert-Supervised Reinforcement Learning for Offline Policy Learning and Evaluation
  5. 2025Behavior-Adaptive Q-Learning: A Unifying Framework for Offline-to-Online RL
  6. 2021NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning
  7. 2023Harnessing Mixed Offline Reinforcement Learning Datasets via Trajectory Weighting
  8. 2023Iteratively Refined Behavior Regularization for Offline Reinforcement Learning