mlsanityData to 14 Oct 2026

Taming OOD Actions for Offline Reinforcement Learning: An Advantage-Based Approach

arXiv 2505.05126 ↗First posted 8 May 2025cs.LGcited by 1

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning (RL) learns policies from fixed datasets without online interactions, but suffers from distribution shift, causing inaccurate evaluation and overestimation of out-of-distribution (OOD) actions. Existing methods counter this by conservatively discouraging all OOD actions, which limits generalization. We propose Advantage-based Diffusion Actor-Critic (ADAC), which evaluates OOD actions via an advantage-like function and uses it to modulate the Q-function update discriminatively. Our key insight is that the (state) value function is generally learned more reliably than the action-value function; we thus use the next-state value to indirectly assess each action. We develop a PointMaze environment to clearly visualize that advantage modulation effectively selects superior OOD actions while discouraging inferior ones. Moreover, extensive experiments on the D4RL benchmark show that ADAC achieves state-of-the-art performance, with especially strong gains on challenging tasks.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 33 of its 40 references that are in the map. First posted on arXiv 8 May 2025.

Builds on

33 of 40 references.

Closest ancestors · 17

Same direction, newest first

  1. 2025VIPO: Value Function Inconsistency Penalized Offline Reinforcement Learning16 Apr
  2. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learning9 Jun
  3. 2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learning26 Apr
  4. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems2 Mar
  5. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning23 Feb
  6. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  7. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  8. 2021Offline RL Without Off-Policy Evaluation16 Jun
  9. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  10. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  11. 2020Offline Reinforcement Learning from Images with Latent Space Models21 Dec
  12. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
5 more
  1. 2020MOPO: Model-based Offline Policy Optimization27 May
  2. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  3. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  4. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  5. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 16

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  3. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  5. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  6. 2016Variational Inference: A Review for Statisticianscited by 1,472
  7. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  8. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,061
  9. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  10. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  11. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 743
  12. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learningcited by 562
4 more
  1. 2022Is Conditional Generative Modeling all you need for Decision-Making?cited by 516
  2. 2023Efficient Diffusion Policies for Offline Reinforcement Learningcited by 157
  3. 2020COG: Connecting New Skills to Past Experience with Offline Reinforcement Learningcited by 83
  4. 2023Elastic Decision Transformercited by 67

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025HOFLON: Hybrid Offline Learning and Online Optimization for Process Start-Up and Grade-Transition Controlcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Offline-Boosted Actor-Critic: Adaptively Blending Optimal Historical Behaviors in Deep Off-Policy RL
  2. 2024Adaptive Advantage-Guided Policy Regularization for Offline Reinforcement Learning
  3. 2026Decoupling Policy Extraction for Offline Reinforcement Learning
  4. 2024Preferred-Action-Optimized Diffusion Policies for Offline Reinforcement Learning
  5. 2024A2PO: Towards Effective Offline Reinforcement Learning from an Advantage-aware Perspective
  6. 2024Offline Reinforcement Learning with OOD State Correction and OOD Action Suppression
  7. 2023Beyond OOD State Actions: Supported Cross-Domain Offline Reinforcement Learning
  8. 2026Diffusion Policy with Behavioral Advantage Correction for Offline Reinforcement Learning