mlsanityData to 14 Oct 2026

When Data Geometry Meets Deep Function: Generalizing Offline Reinforcement Learning

arXiv 2205.11027 ↗First posted 23 May 2022cs.LG cs.AI cs.ROcited by 31

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

In offline reinforcement learning (RL), one detrimental issue to policy learning is the error accumulation of deep Q function in out-of-distribution (OOD) areas. Unfortunately, existing offline RL methods are often over-conservative, inevitably hurting generalization performance outside data distribution. In our study, one interesting observation is that deep Q functions approximate well inside the convex hull of training data. Inspired by this, we propose a new method, DOGE (Distance-sensitive Offline RL with better GEneralization). DOGE marries dataset geometry with deep function approximators in offline RL, and enables exploitation in generalizable OOD areas rather than strictly constraining policy within data distribution. Specifically, DOGE trains a state-conditioned distance function that can be readily plugged into standard actor-critic methods as a policy constraint. Simple yet elegant, our algorithm enjoys better generalization compared to state-of-the-art methods on D4RL benchmarks. Theoretical analysis demonstrates the superiority of our approach to existing methods that are solely based on data distribution or support constraints.

Key facts. Cited by 31 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 46 of its 62 references that are in the map. First posted on arXiv 23 May 2022.

Builds on

46 of 62 references.

Closest ancestors · 21

Same direction, newest first

  1. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning23 Feb
  2. 2021Offline Reinforcement Learning with Soft Behavior Regularization14 Oct
  3. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  4. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  5. 2021Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learning19 Aug
  6. 2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coverage13 Jul
  7. 2021Offline RL Without Off-Policy Evaluation16 Jun
  8. 2021Bellman-consistent Pessimism for Offline Reinforcement Learning13 Jun
  9. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  10. 2021Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learning9 Jun
  11. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning17 May
  12. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
9 more
  1. 2021Offline Reinforcement Learning with Pseudometric Learning2 Mar
  2. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  3. 2020Is Pessimism Provably Efficient for Offline RL?30 Dec
  4. 2020Critic Regularized Regression26 Jun
  5. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  6. 2020MOPO: Model-based Offline Policy Optimization27 May
  7. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  8. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  9. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 25

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  4. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  6. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  7. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  8. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  9. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 773
  10. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  11. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 573
  12. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 566
13 more
  1. 2021Implicit Behavioral Cloningcited by 452
  2. 2018Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layerscited by 340
  3. 2019Batch Policy Learning under Constraintscited by 244
  4. 2020How Neural Networks Extrapolate: From Feedforward to Graph Neural Networkscited by 222
  5. 2019A Theoretical Analysis of Deep Q-Learningcited by 215
  6. 2019On the Inductive Bias of Neural Tangent Kernelscited by 177
  7. 2018Deep Reinforcement Learning and the Deadly Triadcited by 174
  8. 2020Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learningcited by 134
  9. 2018Assessing Generalization in Deep Reinforcement Learningcited by 133
  10. 2020Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimizationcited by 120
  11. 2021Constraints Penalized Q-learning for Safe Offline Reinforcement Learningcited by 91
  12. 2021DeepThermal: Combustion Optimization for Thermal Power Generating Units Using Offline Reinforcement Learningcited by 48
  13. 2019A Finite-Time Analysis of Q-Learning with Neural Network Function Approximationcited by 47

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 9; 2024: 6; 2025: 10; 2026: 6.

31 papers.

Sort:

By area:

  1. 2023Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularizationcited by 116
  2. 2024Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Modelcited by 64
  3. 2023Offline Multi-Agent Reinforcement Learning with Implicit Global-to-Local Value Regularizationcited by 43
  4. 2024Diffusion-DICE: In-Sample Diffusion Guidance for Offline Reinforcement Learningcited by 42
  5. 2023PROTO: Iterative Policy Regularized Offline-to-Online Reinforcement Learningcited by 29
  6. 2024ODICE: Revealing the Mystery of Distribution Correction Estimation via Orthogonal-gradient Updatecited by 21
  7. 2023Seizing Serendipity: Exploiting the Value of Past Success in Off-Policy Actor-Criticcited by 20
  8. 2023Look Beneath the Surface: Exploiting Fundamental Symmetry for Sample-Efficient Offline RLcited by 18
  9. 2023Fighting Uncertainty with Gradients: Offline Reinforcement Learning via Diffusion Score Matchingcited by 17
  10. 2023H2O+: An Improved Framework for Hybrid Offline-and-Online RL with Dynamics Gapscited by 17
  11. 2023Mind the Gap: Offline Policy Optimization for Imperfect Rewardscited by 15
  12. 2025Skill Expansion and Composition in Parameter Spacecited by 14
19 more
  1. 2025Towards Robust Zero-Shot Reinforcement Learningcited by 7
  2. 2023Offline Reinforcement Learning with Imbalanced Datasetscited by 6
  3. 2025Learning Conservative Neural Control Barrier Functions from Offline Datacited by 5
  4. 2025V-OCBF: Learning Safety Filters from Offline Data via Value-Guided Offline Control Barrier Functionscited by 5
  5. 2025Data Center Cooling System Optimization Using Offline Reinforcement Learningcited by 4
  6. 2025An Optimal Discriminator Weighted Imitation Perspective for Reinforcement Learningcited by 4
  7. 2026Safe Flow Q-Learning: Offline Safe Reinforcement Learning with Reachability-Based Flow Policiescited by 4
  8. 2024FOSP: Fine-tuning Offline Safe Policy through World Modelscited by 3
  9. 2025Offline RL with Smooth OOD Generalization in Convex Hull and its Neighborhoodcited by 3
  10. 2025Adaptive Neighborhood-Constrained Q Learning for Offline Reinforcement Learningcited by 2
  11. 2026Epigraph-Guided Flow Matching for Safe and Performant Offline Reinforcement Learningcited by 2
  12. 2024Strategically Conservative Q-Learningcited by 1
  13. 2025Imagination-Limited Q-Learning for Offline Reinforcement Learningcited by 1
  14. 2024UDQL: Bridging The Gap between MSE Loss and The Optimal Value Function in Offline Reinforcement Learningcited by 0
  15. 2025Quantile Q-Learning: Revisiting Offline Extreme Q-Learning with Quantile Regressioncited by 0
  16. 2026Horizon Adaptive Offline Policy Learning via Value Stitchingcited by 0
  17. 2026Diffusion Policy with Behavioral Advantage Correction for Offline Reinforcement Learningcited by 0
  18. 2026Revisiting TD Target Aggregation under Uncertainty in Q-Learningcited by 0
  19. 2026Convex-Hull-Neighborhood Smooth Dual Generalization: Controlling Local Correction Propagation in Offline RLcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021Offline Reinforcement Learning: Fundamental Barriers for Value Function Approximation
  2. 2023Neural Network Approximation for Pessimistic Offline Reinforcement Learning
  3. 2021The Least Restriction for Offline Reinforcement Learning
  4. 2023Offline Reinforcement Learning with On-Policy Q-Function Regularization
  5. 2023Offline Reinforcement Learning with Additional Covering Distributions
  6. 2022Optimal Conservative Offline RL with General Function Approximation via Augmented Lagrangian
  7. 2026Improving Generalization and Robustness in Offline Reinforcement Learning via Boundary-Aware Data Augmentation
  8. 2026From Robotics to Sepsis Treatment: Offline RL via Geometric Pessimism