mlsanityData to 14 Oct 2026

Improving Offline RL by Blending Heuristics

arXiv 2306.00321 ↗First posted 1 Jun 2023cs.LGcited by 10

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

We propose Heuristic Blending (HUBL), a simple performance-improving technique for a broad class of offline RL algorithms based on value bootstrapping. HUBL modifies the Bellman operators used in these algorithms, partially replacing the bootstrapped values with heuristic ones that are estimated with Monte-Carlo returns. For trajectories with higher returns, HUBL relies more on the heuristic values and less on bootstrapping; otherwise, it leans more heavily on bootstrapping. HUBL is very easy to combine with many existing offline RL implementations by relabeling the offline datasets with adjusted rewards and discount factors. We derive a theory that explains HUBL's effect on offline RL as reducing offline RL's complexity and thus increasing its finite-sample performance. Furthermore, we empirically demonstrate that HUBL consistently improves the policy quality of four state-of-the-art bootstrapping-based offline RL algorithms (ATAC, CQL, TD3+BC, and IQL), by 9% on average over 27 datasets of the D4RL and Meta-World benchmarks.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 33 of its 50 references that are in the map. First posted on arXiv 1 Jun 2023.

Builds on

33 of 50 references.

Closest ancestors · 12

Same direction, newest first

  1. 2022CORL: Research-oriented Deep Offline Reinforcement Learning Library13 Oct
  2. 2022On the Role of Discount Factor in Offline Reinforcement Learning7 Jun
  3. 2022Adversarially Trained Actor Critic for Offline Reinforcement Learning5 Feb
  4. 2022Comparing Model-free and Model-based Algorithms for Offline Reinforcement Learning14 Jan
  5. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  6. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  7. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism22 Mar
  8. 2020Is Pessimism Provably Efficient for Offline RL?30 Dec
  9. 2020Overcoming Model Bias for Robust Offline Deep Reinforcement Learning12 Aug
  10. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  11. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  12. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr

Other references · 21

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  3. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,204
  4. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  5. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 639
  6. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 490
  7. 2017Count-Based Exploration with Neural Density Modelscited by 257
  8. 2015An Emphatic Approach to the Problem of Off-policy Temporal-Difference Learningcited by 103
  9. 2021Cautiously Optimistic Policy Optimization and Exploration with Linear Function Approximationcited by 53
  10. 2016Learning and Policy Search in Stochastic Dynamical Systems with Bayesian Neural Networkscited by 49
  11. 2021Heuristic-Guided Reinforcement Learningcited by 42
  12. 2016Unifying task specification in reinforcement learningcited by 41
9 more
  1. 2018An Off-policy Policy Gradient Theorem Using Emphatic Weightingscited by 34
  2. 2019Deep Value Model Predictive Controlcited by 20
  3. 2019Using a Logarithmic Mapping to Enable Lower Discount Factors in Reinforcement Learningcited by 16
  4. 2022Monte Carlo Augmented Actor-Critic for Sparse Reward Deep Reinforcement Learning from Suboptimal Demonstrationscited by 14
  5. 2021Emphatic Algorithms for Deep Reinforcement Learningcited by 12
  6. 2019Partially Linear Additive Gaussian Graphical Modelscited by 6
  7. 2020Temporal Poisson Square Root Graphical Modelscited by 6
  8. 2018Planning with a Receding Horizon for Manipulation in Clutter using a Learned Value Functioncited by 5
  9. 2023A Data-Driven State Aggregation Approach for Dynamic Discrete Choice Modelscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 6; 2025: 1; 2026: 3.

10 papers.

  1. 2024Doubly Mild Generalization for Offline Reinforcement Learningcited by 21
  2. 2025Offline RL with Smooth OOD Generalization in Convex Hull and its Neighborhoodcited by 3
  3. 2024Enhancing Q-Learning with Large Language Model Heuristicscited by 2
  4. 2024Integrating Domain Knowledge for handling Limited Data in Offline RLcited by 1
  5. 2026SPAR: Support-Preserving Action Rectificationcited by 1
  6. 2024Reinforcement Learning Paycheck Optimization for Multivariate Financial Goalscited by 0
  7. 2024Efficient Offline Reinforcement Learning: First Imitate, then Improvecited by 0
  8. 2024Hypercube Policy Regularization Framework for Offline Reinforcement Learningcited by 0
  9. 2026Convex-Hull-Neighborhood Smooth Dual Generalization: Controlling Local Correction Propagation in Offline RLcited by 0
  10. 2026RoMAN-Flow: Taming Autoregressive Normalizing Flows for Offline Reinforcement Learning in Robotic Manipulationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Offline RL Policies Should be Trained to be Adaptive
  2. 2025An Empirical Study on the Effectiveness of Incorporating Offline RL As Online RL Subroutines
  3. 2025Horizon Reduction Makes RL Scalable
  4. 2021Showing Your Offline Reinforcement Learning Work: Online Evaluation Budget Matters
  5. 2024Is Value Learning Really the Main Bottleneck in Offline RL?
  6. 2022Offline RL With Realistic Datasets: Heteroskedasticity and Support Constraints
  7. 2023Offline Data Enhanced On-Policy Policy Gradient with Provable Guarantees
  8. 2023Bridging Distributionally Robust Learning and Offline RL: An Approach to Mitigate Distribution Shift and Partial Data Coverage