mlsanityData to 7 Oct 2026

Doubly Mild Generalization for Offline Reinforcement Learning

arXiv 2411.07934 ↗First posted 12 Nov 2024cs.LG cs.AIcited by 21

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline Reinforcement Learning (RL) suffers from the extrapolation error and value overestimation. From a generalization perspective, this issue can be attributed to the over-generalization of value functions or policies towards out-of-distribution (OOD) actions. Significant efforts have been devoted to mitigating such generalization, and recent in-sample learning approaches have further succeeded in entirely eschewing it. Nevertheless, we show that mild generalization beyond the dataset can be trusted and leveraged to improve performance under certain conditions. To appropriately exploit generalization in offline RL, we propose Doubly Mild Generalization (DMG), comprising (i) mild action generalization and (ii) mild generalization propagation. The former refers to selecting actions in a close neighborhood of the dataset to maximize the Q values. Even so, the potential erroneous generalization can still be propagated, accumulated, and exacerbated by bootstrapping. In light of this, the latter concept is introduced to mitigate the generalization propagation without impeding the propagation of RL learning signals. Theoretically, DMG guarantees better performance than the in-sample optimal policy in the oracle generalization scenario. Even under worst-case generalization, DMG can still control value overestimation at a certain level and lower bound the performance. Empirically, DMG achieves state-of-the-art performance across Gym-MuJoCo locomotion tasks and challenging AntMaze tasks. Moreover, benefiting from its flexibility in both generalization aspects, DMG enjoys a seamless transition from offline to online learning and attains strong online fine-tuning performance.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 66 of its 93 references that are in the map. First posted on arXiv 12 Nov 2024.

Builds on

66 of 93 references.

Closest ancestors · 34

Same direction, newest first

  1. 2024Offline Reinforcement Learning with OOD State Correction and OOD Action Suppression25 Oct
  2. 2024Hokoff: Real Game Dataset from Honor of Kings and its Offline Reinforcement Learning Benchmarks20 Aug
  3. 2024Offline Multi-task Transfer RL with Representational Penalization19 Feb
  4. 2023Supported Trust Region Optimization for Offline Reinforcement Learning15 Nov
  5. 2023Counterfactual Conservative Q Learning for Offline Multi-agent Reinforcement Learning22 Sep
  6. 2023Policy Regularization with Dataset Constraint for Offline Reinforcement Learning11 Jun
  7. 2023Improving Offline RL by Blending Heuristics1 Jun
  8. 2023Revisiting the Minimalist Approach to Offline Reinforcement Learning16 May
  9. 2023Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularization28 Mar
  10. 2023Extreme Q-Learning: MaxEnt RL without Entropy5 Jan
  11. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learning9 Jun
  12. 2022RORL: Robust Offline Reinforcement Learning via Conservative Smoothing6 Jun
22 more
  1. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning23 Feb
  2. 2022Supported Policy Optimization for Offline Reinforcement Learning13 Feb
  3. 2022Adversarially Trained Actor Critic for Offline Reinforcement Learning5 Feb
  4. 2021Offline Reinforcement Learning with Value-based Episodic Memory19 Oct
  5. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  6. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  7. 2021Conservative Offline Distributional Reinforcement Learning12 Jul
  8. 2021Offline RL Without Off-Policy Evaluation16 Jun
  9. 2021Bellman-consistent Pessimism for Offline Reinforcement Learning13 Jun
  10. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  11. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
  12. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  13. 2020PLAS: Latent Action Space for Offline Reinforcement Learning14 Nov
  14. 2020EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL21 Jul
  15. 2020Critic Regularized Regression26 Jun
  16. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  17. 2020MOPO: Model-based Offline Policy Optimization27 May
  18. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  19. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  20. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  21. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning19 Feb
  22. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 32

  1. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2021Convex optimizationcited by 2,479
  4. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  5. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  6. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  7. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,519
  8. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  9. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  10. 2017Constrained Policy Optimizationcited by 928
  11. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  12. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
20 more
  1. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  2. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  3. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learningcited by 543
  4. 2019Model-Based Reinforcement Learning for Ataricited by 495
  5. 2020Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 475
  6. 2018Regularisation of Neural Networks by Enforcing Lipschitz Continuitycited by 311
  7. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  8. 2022A Review of Safe Reinforcement Learning: Methods, Theory and Applicationscited by 169
  9. 2020Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimizationcited by 120
  10. 2015An Emphatic Approach to the Problem of Off-policy Temporal-Difference Learningcited by 103
  11. 2023A Tutorial on Meta-Reinforcement Learningcited by 102
  12. 2019BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learningcited by 90
  13. 2023Empirical Design in Reinforcement Learningcited by 54
  14. 2021Heuristic-Guided Reinforcement Learningcited by 41
  15. 2018An Off-policy Policy Gradient Theorem Using Emphatic Weightingscited by 34
  16. 2021Model-based Meta Reinforcement Learning using Graph Structured Surrogate Modelscited by 16
  17. 2022Monte Carlo Augmented Actor-Critic for Sparse Reward Deep Reinforcement Learning from Suboptimal Demonstrationscited by 14
  18. 2024Robust Fast Adaptation from Adversarially Explicit Task Distribution Generationcited by 12
  19. 2023A Simple Yet Effective Strategy to Robustify the Meta Learning Paradigmcited by 9
  20. 2024Stop Wandering, Find the Keys: LLMs Discriminate Key States for Efficient Multi-Agent Explorationcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 11; 2026: 9.

21 papers.

Sort:

By area:

  1. 2025FlyLoRA: Boosting Task Decoupling and Parameter Efficiency via Implicit Rank-Wise Mixture-of-Expertscited by 34
  2. 2025Fast and Robust: Task Sampling with Posterior and Diversity Synergies for Adaptive Decision-Makers in Randomized Environmentscited by 11
  3. 2026Small Generalizable Prompt Predictive Models Can Steer Efficient RL Post-Training of Large Reasoning Modelscited by 11
  4. 2025Fly-CL: A Fly-Inspired Framework for Enhancing Efficient Decorrelation and Reduced Training Time in Pre-trained Model-based Continual Representation Learningcited by 10
  5. 2025Dual-Robust Cross-Domain Offline Reinforcement Learning Against Dynamics Shiftscited by 5
  6. 2024Goal-Conditioned Data Augmentation for Offline Reinforcement Learningcited by 4
  7. 2025TrojanTO: Action-Level Backdoor Attacks against Trajectory Optimization Modelscited by 2
  8. 2025Adaptive Neighborhood-Constrained Q Learning for Offline Reinforcement Learningcited by 2
  9. 2026Robust Regularized Policy Iteration under Transition Uncertaintycited by 2
  10. 2025Enhancing Generative Auto-bidding with Offline Reward Evaluation and Policy Searchcited by 1
  11. 2026Epistemic Robust Offline Reinforcement Learningcited by 1
  12. 2026Offline Policy Optimization with Posterior Samplingcited by 1
9 more
  1. 2025In-Context Compositional Q-Learning for Offline Reinforcement Learningcited by 0
  2. 2025VAO: Validation-Aligned Optimization for Cross-Task Generative Auto-Biddingcited by 0
  3. 2025From Static Constraints to Dynamic Adaptation: Sample-Level Constraint Relaxation for Offline-to-Online Reinforcement Learningcited by 0
  4. 2025Quantile Q-Learning: Revisiting Offline Extreme Q-Learning with Quantile Regressioncited by 0
  5. 2026Efficient Anti-exploration via VQVAE and Fuzzy Clustering in Offline Reinforcement Learningcited by 0
  6. 2026Regularized Offline Policy Optimization with Posterior Hybrid Bayesian Beliefcited by 0
  7. 2026Generalization in offline RL: The structure is more important than the amount of pessimismcited by 0
  8. 2026Convex-Hull-Neighborhood Smooth Dual Generalization: Controlling Local Correction Propagation in Offline RLcited by 0
  9. 2026Lifted Bellman Linear Programming for Offline Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021Improving Zero-shot Generalization in Offline Reinforcement Learning using Generalized Similarity Functions
  2. 2025Provable Zero-Shot Generalization in Offline Reinforcement Learning
  3. 2022Domain Generalization for Robust Model-Based Offline Reinforcement Learning
  4. 2023The Generalization Gap in Offline Reinforcement Learning
  5. 2023Equivariant Data Augmentation for Generalization in Offline Reinforcement Learning
  6. 2023Beyond Uniform Sampling: Offline Reinforcement Learning with Imbalanced Datasets
  7. 2024On Sample-Efficient Offline Reinforcement Learning: Data Diversity, Posterior Sampling, and Beyond
  8. 2025Mildly Conservative Regularized Evaluation for Offline Reinforcement Learning