mlsanityData to 7 Oct 2026

Convex-Hull-Neighborhood Smooth Dual Generalization: Controlling Local Correction Propagation in Offline RL

arXiv 2608.03108 ↗First posted 4 Aug 2026cs.LG cs.CLcited by 0

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning (offline RL) can benefit from nearby out-of-distribution (OOD) actions, but estimation errors at these actions may be amplified by bootstrapping. Existing regularization and local-generalization methods control either the admissible OOD region or the influence of generalized targets, often through separate mechanisms. We propose Convex Hull Neighborhood Smooth Dual Generalization (CSDG), which expresses the Bellman backup as an in-sample value target plus a CHN-local correction. This formulation makes the generalized contribution explicit and separates it from the in-sample reference path. The correction is obtained by smoothing in-sample-oriented and OOD-oriented candidates sampled at different perturbation radii. A mixture coefficient lambda scales its contribution to each backup, while the recursive discount remains gamma. Under boundedness and fixed perturbation kernels, we derive an exact one-step correction identity, a time-varying iterate bound, and a fixed-point bound that depends only on the branch discrepancy at the fixed point. We further characterize the implicit policies induced by the idealized operators and give a conditional non-degradation criterion. The practical algorithm approximates these quantities using asymmetric bounded noise and expectile regression, without exact support classification or an additional pessimistic OOD penalty. Experiments on Gym-MuJoCo and AntMaze show strong aggregate performance and stable value estimation. Code is available at: https://github.com/YOUNG-fnxm/CSDG

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 45 of its 56 references that are in the map. First posted on arXiv 4 Aug 2026.

Builds on

45 of 56 references.

Closest ancestors · 31

Same direction, newest first

  1. 2026UNIQ: Conformal Calibration for Adaptive Conservatism in Offline Reinforcement Learning28 May
  2. 2026Flow Actor-Critic for Offline Reinforcement Learning20 Feb
  3. 2025Offline RL with Smooth OOD Generalization in Convex Hull and its Neighborhood10 Jun
  4. 2024Doubly Mild Generalization for Offline Reinforcement Learning12 Nov
  5. 2024Hokoff: Real Game Dataset from Honor of Kings and its Offline Reinforcement Learning Benchmarks20 Aug
  6. 2023Iteratively Refined Behavior Regularization for Offline Reinforcement Learning9 Jun
  7. 2023Improving Offline RL by Blending Heuristics1 Jun
  8. 2023Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularization28 Mar
  9. 2023The In-Sample Softmax for Offline Reinforcement Learning28 Feb
  10. 2023Extreme Q-Learning: MaxEnt RL without Entropy5 Jan
  11. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learning9 Jun
  12. 2022RORL: Robust Offline Reinforcement Learning via Conservative Smoothing6 Jun
19 more
  1. 2022When Data Geometry Meets Deep Function: Generalizing Offline Reinforcement Learning23 May
  2. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning23 Feb
  3. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  4. 2021Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learning19 Aug
  5. 2021Conservative Offline Distributional Reinforcement Learning12 Jul
  6. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  7. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning17 May
  8. 2021Continuous Doubly Constrained Batch Reinforcement Learning18 Feb
  9. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  10. 2020Offline Reinforcement Learning from Images with Latent Space Models21 Dec
  11. 2020PLAS: Latent Action Space for Offline Reinforcement Learning14 Nov
  12. 2020OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning26 Oct
  13. 2020EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL21 Jul
  14. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  15. 2020MOPO: Model-based Offline Policy Optimization27 May
  16. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  17. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  18. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  19. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 14

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  3. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  4. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  5. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  6. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  7. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  8. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  9. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  10. 2020Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimizationcited by 120
  11. 2015An Emphatic Approach to the Problem of Off-policy Temporal-Difference Learningcited by 103
  12. 2019BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learningcited by 90
2 more
  1. 2019Off-Policy Deep Reinforcement Learning by Bootstrapping the Covariate Shiftcited by 47
  2. 2021UMBRELLA: Uncertainty-Aware Model-Based Offline Reinforcement Learning Leveraging Planningcited by 23

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Improving Generalization and Robustness in Offline Reinforcement Learning via Boundary-Aware Data Augmentation
  2. 2022Domain Generalization for Robust Model-Based Offline Reinforcement Learning
  3. 2022Regularizing a Model-based Policy Stationary Distribution to Stabilize Offline Reinforcement Learning
  4. 2022Offline Reinforcement Learning with Realizability and Single-policy Concentrability
  5. 2023Offline Primal-Dual Reinforcement Learning for Linear MDPs
  6. 2024On Sample-Efficient Offline Reinforcement Learning: Data Diversity, Posterior Sampling, and Beyond
  7. 2022Offline Policy Optimization in RL with Variance Regularizaton
  8. 2023A Connection between One-Step Regularization and Critic Regularization in Reinforcement Learning