mlsanityData to 14 Oct 2026

Offline RL with Smooth OOD Generalization in Convex Hull and its Neighborhood

arXiv 2506.08417 ↗First posted 10 Jun 2025cs.LG cs.AIcited by 3

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline Reinforcement Learning (RL) struggles with distributional shifts, leading to the Q-value overestimation for out-of-distribution (OOD) actions. Existing methods address this issue by imposing constraints; however, they often become overly conservative when evaluating OOD regions, which constrains the Q-function generalization. This over-constraint issue results in poor Q-value estimation and hinders policy improvement. In this paper, we introduce a novel approach to achieve better Q-value estimation by enhancing Q-function generalization in OOD regions within Convex Hull and its Neighborhood (CHN). Under the safety generalization guarantees of the CHN, we propose the Smooth Bellman Operator (SBO), which updates OOD Q-values by smoothing them with neighboring in-sample Q-values. We theoretically show that SBO approximates true Q-values for both in-sample and OOD actions within the CHN. Our practical algorithm, Smooth Q-function OOD Generalization (SQOG), empirically alleviates the over-constraint issue, achieving near-accurate Q-value estimation. On the D4RL benchmarks, SQOG outperforms existing state-of-the-art methods in both performance and computational efficiency.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 33 of its 53 references that are in the map. First posted on arXiv 10 Jun 2025.

Builds on

33 of 53 references.

Closest ancestors · 19

Same direction, newest first

  1. 2023Towards Robust Offline Reinforcement Learning under Diverse Data Corruption19 Oct
  2. 2023Accountability in Offline Reinforcement Learning: Explaining Decisions with a Corpus of Examples11 Oct
  3. 2023Beyond Uniform Sampling: Offline Reinforcement Learning with Imbalanced Datasets6 Oct
  4. 2023Policy Regularization with Dataset Constraint for Offline Reinforcement Learning11 Jun
  5. 2023Improving Offline RL by Blending Heuristics1 Jun
  6. 2023Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularization28 Mar
  7. 2023Extreme Q-Learning: MaxEnt RL without Entropy5 Jan
  8. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learning9 Jun
  9. 2022RORL: Robust Offline Reinforcement Learning via Conservative Smoothing6 Jun
  10. 2022Why So Pessimistic? Estimating Uncertainties for Offline RL through Ensembles, and Why Their Independence Matters27 May
  11. 2022When Data Geometry Meets Deep Function: Generalizing Offline Reinforcement Learning23 May
  12. 2021A Dataset Perspective on Offline Reinforcement Learning8 Nov
7 more
  1. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  2. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  3. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  4. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
  5. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  6. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  7. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 14

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  3. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  4. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  5. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  6. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 715
  7. 2019On the Inductive Bias of Neural Tangent Kernelscited by 177
  8. 2020Flows for simultaneous manifold learning and density estimationcited by 118
  9. 2016Why is Posterior Sampling Better than Optimism for Reinforcement Learning?cited by 117
  10. 2019BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learningcited by 91
  11. 2020Improving Generalization in Reinforcement Learning with Mixture Regularizationcited by 81
  12. 2021Manifold Topology Divergence: a Framework for Comparing Data Manifoldscited by 31
2 more
  1. 2024SEABO: A Simple Search-Based Method for Offline Imitation Learningcited by 14
  2. 2024Kernel Metric Learning for In-Sample Off-Policy Evaluation of Deterministic RL Policiescited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025Simplicial Embeddings Improve Sample Efficiency in Actor-Critic Agentscited by 12
  2. 2026AdamO: A Collapse-Suppressed Optimizer for Offline RLcited by 1
  3. 2026Convex-Hull-Neighborhood Smooth Dual Generalization: Controlling Local Correction Propagation in Offline RLcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Strategically Conservative Q-Learning
  2. 2022Bellman Residual Orthogonalization for Offline Reinforcement Learning
  3. 2024Doubly Mild Generalization for Offline Reinforcement Learning
  4. 2025Adaptive Neighborhood-Constrained Q Learning for Offline Reinforcement Learning
  5. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning
  6. 2026Residuals-based Offline Reinforcement Learning
  7. 2021Offline Reinforcement Learning: Fundamental Barriers for Value Function Approximation
  8. 2026Peng's Q(λ) for Conservative Value Estimation in Offline Reinforcement Learning