mlsanityData to 7 Oct 2026

Offline Reinforcement Learning with OOD State Correction and OOD Action Suppression

arXiv 2410.19400 ↗First posted 25 Oct 2024cs.LG cs.AIcited by 17

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

In offline reinforcement learning (RL), addressing the out-of-distribution (OOD) action issue has been a focus, but we argue that there exists an OOD state issue that also impairs performance yet has been underexplored. Such an issue describes the scenario when the agent encounters states out of the offline dataset during the test phase, leading to uncontrolled behavior and performance degradation. To this end, we propose SCAS, a simple yet effective approach that unifies OOD state correction and OOD action suppression in offline RL. Technically, SCAS achieves value-aware OOD state correction, capable of correcting the agent from OOD states to high-value in-distribution states. Theoretical and empirical results show that SCAS also exhibits the effect of suppressing OOD actions. On standard offline RL benchmarks, SCAS achieves excellent performance without additional hyperparameter tuning. Moreover, benefiting from its OOD state correction feature, SCAS demonstrates enhanced robustness against environmental perturbations.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 51 of its 77 references that are in the map. First posted on arXiv 25 Oct 2024.

Builds on

51 of 77 references.

Closest ancestors · 29

Same direction, newest first

  1. 2024Hokoff: Real Game Dataset from Honor of Kings and its Offline Reinforcement Learning Benchmarks20 Aug
  2. 2024ODICE: Revealing the Mystery of Distribution Correction Estimation via Orthogonal-gradient Update1 Feb
  3. 2023Supported Trust Region Optimization for Offline Reinforcement Learning15 Nov
  4. 2023Beyond Uniform Sampling: Offline Reinforcement Learning with Imbalanced Datasets6 Oct
  5. 2023Counterfactual Conservative Q Learning for Offline Multi-agent Reinforcement Learning22 Sep
  6. 2023Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularization28 Mar
  7. 2023Extreme Q-Learning: MaxEnt RL without Entropy5 Jan
  8. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learning9 Jun
  9. 2022RORL: Robust Offline Reinforcement Learning via Conservative Smoothing6 Jun
  10. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning23 Feb
  11. 2022Supported Policy Optimization for Offline Reinforcement Learning13 Feb
  12. 2022Adversarially Trained Actor Critic for Offline Reinforcement Learning5 Feb
17 more
  1. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  2. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  3. 2021OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimation21 Jun
  4. 2021Offline RL Without Off-Policy Evaluation16 Jun
  5. 2021Bellman-consistent Pessimism for Offline Reinforcement Learning13 Jun
  6. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  7. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
  8. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  9. 2021NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning1 Feb
  10. 2020EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL21 Jul
  11. 2020Critic Regularized Regression26 Jun
  12. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  13. 2020MOPO: Model-based Offline Policy Optimization27 May
  14. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  15. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  16. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  17. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 22

  1. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  4. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,519
  5. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  6. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  7. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
  8. 2017Constrained Policy Optimizationcited by 928
  9. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  10. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  11. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  12. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
10 more
  1. 2019Model-Based Reinforcement Learning for Ataricited by 495
  2. 2020Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 475
  3. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 187
  4. 2022A Review of Safe Reinforcement Learning: Methods, Theory and Applicationscited by 169
  5. 2020Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimizationcited by 120
  6. 2023A Tutorial on Meta-Reinforcement Learningcited by 102
  7. 2021Model-based Meta Reinforcement Learning using Graph Structured Surrogate Modelscited by 16
  8. 2024Robust Fast Adaptation from Adversarially Explicit Task Distribution Generationcited by 12
  9. 2023A Simple Yet Effective Strategy to Robustify the Meta Learning Paradigmcited by 9
  10. 2024Stop Wandering, Find the Keys: LLMs Discriminate Key States for Efficient Multi-Agent Explorationcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 10; 2026: 5.

17 papers.

Sort:

By area:

  1. 2025FlyLoRA: Boosting Task Decoupling and Parameter Efficiency via Implicit Rank-Wise Mixture-of-Expertscited by 34
  2. 2024Latent Reward: LLM-Empowered Credit Assignment in Episodic Reinforcement Learningcited by 22
  3. 2024Doubly Mild Generalization for Offline Reinforcement Learningcited by 21
  4. 2025Fast and Robust: Task Sampling with Posterior and Diversity Synergies for Adaptive Decision-Makers in Randomized Environmentscited by 11
  5. 2026Small Generalizable Prompt Predictive Models Can Steer Efficient RL Post-Training of Large Reasoning Modelscited by 11
  6. 2025Fly-CL: A Fly-Inspired Framework for Enhancing Efficient Decorrelation and Reduced Training Time in Pre-trained Model-based Continual Representation Learningcited by 10
  7. 2026Manifold-Constrained Energy-Based Transition Models for Offline Reinforcement Learningcited by 6
  8. 2025Offline Guarded Safe Reinforcement Learning for Medical Treatment Optimization Strategiescited by 4
  9. 2026Improving Policy Exploitation in Online Reinforcement Learning with Instant Retrospect Actioncited by 4
  10. 2025Beyond Non-Expert Demonstrations: Outcome-Driven Action Constraint for Offline Reinforcement Learningcited by 2
  11. 2025TrojanTO: Action-Level Backdoor Attacks against Trajectory Optimization Modelscited by 2
  12. 2025Adaptive Neighborhood-Constrained Q Learning for Offline Reinforcement Learningcited by 2
5 more
  1. 2025Enhancing Generative Auto-bidding with Offline Reward Evaluation and Policy Searchcited by 1
  2. 2026Epistemic Robust Offline Reinforcement Learningcited by 1
  3. 2025Dual Alignment Maximin Optimization for Offline Model-based RLcited by 0
  4. 2025VAO: Validation-Aligned Optimization for Cross-Task Generative Auto-Biddingcited by 0
  5. 2026DiDrive: A Risk-Aware Hierarchical Diffusion Framework for Safe Offline Reinforcement Learning in Autonomous Drivingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Beyond OOD State Actions: Supported Cross-Domain Offline Reinforcement Learning
  2. 2025Variational OOD State Correction for Offline Reinforcement Learning
  3. 2025Taming OOD Actions for Offline Reinforcement Learning: An Advantage-Based Approach
  4. 2026Leveraging Offline Supervision for Efficient and Generalizable Reinforcement Learning in Large-Scale Vision-Language-Action Models
  5. 2024Augmenting Offline RL with Unlabeled Data
  6. 2024SAMG: Offline-to-Online Reinforcement Learning via State-Action-Conditional Offline Model Guidance
  7. 2023Towards Robust Offline-to-Online Reinforcement Learning via Uncertainty and Smoothness
  8. 2023Offline Retraining for Online RL: Decoupled Policy Learning to Mitigate Exploration Bias