mlsanityData to 7 Oct 2026

Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularization

arXiv 2303.15810 ↗First posted 28 Mar 2023cs.LG cs.AIcited by 114

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Most offline reinforcement learning (RL) methods suffer from the trade-off between improving the policy to surpass the behavior policy and constraining the policy to limit the deviation from the behavior policy as computing Q-values using out-of-distribution (OOD) actions will suffer from errors due to distributional shift. The recently proposed textitIn-sample Learning paradigm (i.e., IQL), which improves the policy by quantile regression using only data samples, shows great promise because it learns an optimal policy without querying the value function of any unseen actions. However, it remains unclear how this type of method handles the distributional shift in learning the value function. In this work, we make a key finding that the in-sample learning paradigm arises under the textitImplicit Value Regularization (IVR) framework. This gives a deeper understanding of why the in-sample learning paradigm works, i.e., it applies implicit value regularization to the policy. Based on the IVR framework, we further propose two practical algorithms, Sparse Q-learning (SQL) and Exponential Q-learning (EQL), which adopt the same value regularization used in existing works, but in a complete in-sample manner. Compared with IQL, we find that our algorithms introduce sparsity in learning the value function, making them more robust in noisy data regimes. We also verify the effectiveness of SQL and EQL on D4RL benchmark datasets and show the benefits of in-sample learning by comparing them with CQL in small data regimes.

Key facts. Cited by 114 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 42 of its 50 references that are in the map. First posted on arXiv 28 Mar 2023.

Builds on

42 of 50 references.

Closest ancestors · 17

Same direction, newest first

  1. 2023Mind the Gap: Offline Policy Optimization for Imperfect Rewards3 Feb
  2. 2023Extreme Q-Learning: MaxEnt RL without Entropy5 Jan
  3. 2022When Data Geometry Meets Deep Function: Generalizing Offline Reinforcement Learning23 May
  4. 2022When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning?12 Apr
  5. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning23 Feb
  6. 2021d3rlpy: An Offline Deep Reinforcement Learning Library6 Nov
  7. 2021Offline Reinforcement Learning with Soft Behavior Regularization14 Oct
  8. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  9. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  10. 2021Offline RL Without Off-Policy Evaluation16 Jun
  11. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  12. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning17 May
5 more
  1. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
  2. 2020Critic Regularized Regression26 Jun
  3. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  4. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  5. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 25

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  4. 2021Convex optimizationcited by 2,479
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  6. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  7. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  8. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  9. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  10. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  11. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  12. 2019A Theory of Regularized Markov Decision Processescited by 279
13 more
  1. 2021MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scalecited by 226
  2. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 187
  3. 2017Bridging the Gap Between Value and Policy Based Reinforcement Learningcited by 178
  4. 2017A unified view of entropy-regularized Markov decision processescited by 174
  5. 2018Understanding the impact of entropy on policy optimizationcited by 171
  6. 2019BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learningcited by 90
  7. 2021Constraints Penalized Q-learning for Safe Offline Reinforcement Learningcited by 90
  8. 2022Discriminator-Weighted Offline Imitation Learning from Suboptimal Demonstrationscited by 89
  9. 2021Model Selection for Offline Reinforcement Learning: Practical Considerations for Healthcare Settingscited by 68
  10. 2022A Policy-Guided Imitation Approach for Offline Reinforcement Learningcited by 58
  11. 2021DeepThermal: Combustion Optimization for Thermal Power Generating Units Using Offline Reinforcement Learningcited by 47
  12. 2018Path Consistency Learning in Tsallis Entropy Regularized MDPscited by 22
  13. 2019Tsallis Reinforcement Learning: A Unified Framework for Maximum Entropy Reinforcement Learningcited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 20; 2024: 30; 2025: 42; 2026: 22.

114 papers.

Sort:

By area:

  1. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policiescited by 307
  2. 2025Flow Q-Learningcited by 162
  3. 2023HIQL: Offline Goal-Conditioned RL with Latent States as Actionscited by 133
  4. 2024Is Value Learning Really the Main Bottleneck in Offline RL?cited by 82
  5. 2023Inverse Preference Learning: Preference-based RL without a Reward Functioncited by 68
  6. 2024Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Modelcited by 62
  7. 2025Horizon Reduction Makes RL Scalablecited by 55
  8. 2025Diffusion Guidance Is a Controllable Policy Improvement Operatorcited by 51
  9. 2023Offline Multi-Agent Reinforcement Learning with Implicit Global-to-Local Value Regularizationcited by 42
  10. 2024Diffusion-DICE: In-Sample Diffusion Guidance for Offline Reinforcement Learningcited by 42
  11. 2023Boosting Continuous Control with Consistency Policycited by 36
  12. 2024Diffusion Actor-Critic: Formulating Constrained Policy Iteration as Diffusion Noise Regression for Offline Reinforcement Learningcited by 33
38 more
  1. 2023PROTO: Iterative Policy Regularized Offline-to-Online Reinforcement Learningcited by 28
  2. 2025Scaling Offline RL via Efficient and Expressive Shortcut Modelscited by 27
  3. 2023Critic-Guided Decision Transformer for Offline Reinforcement Learningcited by 25
  4. 2023Towards Robust Offline Reinforcement Learning under Diverse Data Corruptioncited by 24
  5. 2023Uni-O4: Unifying Online and Offline Deep Reinforcement Learning with Multi-Step On-Policy Optimizationcited by 23
  6. 2025Generative Auto-Bidding with Value-Guided Explorationscited by 23
  7. 2023SMORE: Score Models for Offline Goal-Conditioned Reinforcement Learningcited by 21
  8. 2024ODICE: Revealing the Mystery of Distribution Correction Estimation via Orthogonal-gradient Updatecited by 21
  9. 2024Doubly Mild Generalization for Offline Reinforcement Learningcited by 21
  10. 2023Seizing Serendipity: Exploiting the Value of Past Success in Off-Policy Actor-Criticcited by 20
  11. 2025floq: Training Critics via Flow-Matching for Scaling Compute in Value-Based RLcited by 20
  12. 2024AlignIQL: Policy Alignment in Implicit Q-Learning through Constrained Optimizationcited by 19
  13. 2023Look Beneath the Surface: Exploiting Fundamental Symmetry for Sample-Efficient Offline RLcited by 18
  14. 2025Behavior-Regularized Diffusion Policy Optimization for Offline Reinforcement Learningcited by 18
  15. 2024Offline Reinforcement Learning with OOD State Correction and OOD Action Suppressioncited by 17
  16. 2023H2O+: An Improved Framework for Hybrid Offline-and-Online RL with Dynamics Gapscited by 15
  17. 2025Skill Expansion and Composition in Parameter Spacecited by 13
  18. 2025Prior-Guided Diffusion Planning for Offline Reinforcement Learningcited by 13
  19. 2025Transitive RL: Value Learning via Divide and Conquercited by 12
  20. 2025Scalable Offline Model-Based RL with Action Chunkscited by 12
  21. 2024Conservative and Risk-Aware Offline Multi-Agent Reinforcement Learningcited by 10
  22. 2024A2PO: Towards Effective Offline Reinforcement Learning from an Advantage-aware Perspectivecited by 10
  23. 2024Hummer: Towards Limited Competitive Preference Datasetcited by 10
  24. 2024Offline-Boosted Actor-Critic: Adaptively Blending Optimal Historical Behaviors in Deep Off-Policy RLcited by 10
  25. 2024Optimistic Critic Reconstruction and Constrained Fine-Tuning for General Offline-to-Online RLcited by 9
  26. 2025DEAS: DEtached value learning with Action Sequence for Scalable Offline RLcited by 9
  27. 2023A Simple Unified Uncertainty-Guided Framework for Offline-to-Online Reinforcement Learningcited by 8
  28. 2024ComaDICE: Offline Cooperative Multi-Agent Reinforcement Learning with Stationary Distribution Shift Regularizationcited by 8
  29. 2023Robust Offline Reinforcement learning with Heavy-Tailed Rewardscited by 7
  30. 2024Adaptive Q-Aid for Conditional Supervised Learning in Offline Reinforcement Learningcited by 7
  31. 2024Tackling Data Corruption in Offline Reinforcement Learning via Sequence Modelingcited by 7
  32. 2025Penalizing Infeasible Actions and Reward Scaling in Reinforcement Learning with Offline Datacited by 7
  33. 2025Towards Robust Zero-Shot Reinforcement Learningcited by 7
  34. 2023Offline Reinforcement Learning with Imbalanced Datasetscited by 6
  35. 2024Value Improved Actor Critic Algorithmscited by 6
  36. 2025Efficient Whole Slide Pathology VQA via Token Compressioncited by 6
  37. 2025Expressive Value Learning for Scalable Offline Reinforcement Learningcited by 6
  38. 2025Dichotomous Diffusion Policy Optimizationcited by 6

50 most cited of 114 All 114 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022Offline Policy Optimization in RL with Variance Regularizaton
  2. 2022Offline Reinforcement Learning with Adaptive Behavior Regularization
  3. 2024On Sample-Efficient Offline Reinforcement Learning: Data Diversity, Posterior Sampling, and Beyond
  4. 2023Iteratively Refined Behavior Regularization for Offline Reinforcement Learning
  5. 2023Offline Reinforcement Learning with On-Policy Q-Function Regularization
  6. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning
  7. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learning
  8. 2024Out-of-Distribution Adaptation in Offline RL: Counterfactual Reasoning via Causal Normalizing Flows