mlsanityData to 7 Oct 2026

Beyond Non-Expert Demonstrations: Outcome-Driven Action Constraint for Offline Reinforcement Learning

arXiv 2504.01719 ↗First posted 2 Apr 2025cs.LG cs.ROcited by 2

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

We address the challenge of offline reinforcement learning using realistic data, specifically non-expert data collected through sub-optimal behavior policies. Under such circumstance, the learned policy must be safe enough to manage distribution shift while maintaining sufficient flexibility to deal with non-expert (bad) demonstrations from offline data.To tackle this issue, we introduce a novel method called Outcome-Driven Action Flexibility (ODAF), which seeks to reduce reliance on the empirical action distribution of the behavior policy, hence reducing the negative impact of those bad demonstrations.To be specific, a new conservative reward mechanism is developed to deal with distribution shift by evaluating actions according to whether their outcomes meet safety requirements - remaining within the state support area, rather than solely depending on the actions' likelihood based on offline data.Besides theoretical justification, we provide empirical evidence on widely used MuJoCo and various maze benchmarks, demonstrating that our ODAF method, implemented using uncertainty quantification techniques, effectively tolerates unseen transitions for improved "trajectory stitching," while enhancing the agent's ability to learn from realistic non-expert data.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 28 of its 44 references that are in the map. First posted on arXiv 2 Apr 2025.

Builds on

28 of 44 references.

Closest ancestors · 15

Same direction, newest first

  1. 2024Offline Reinforcement Learning with OOD State Correction and OOD Action Suppression25 Oct
  2. 2022RORL: Robust Offline Reinforcement Learning via Conservative Smoothing6 Jun
  3. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning23 Feb
  4. 2022Supported Policy Optimization for Offline Reinforcement Learning13 Feb
  5. 2022Adversarially Trained Actor Critic for Offline Reinforcement Learning5 Feb
  6. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  7. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  8. 2021Bellman-consistent Pessimism for Offline Reinforcement Learning13 Jun
  9. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  10. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  11. 2020Is Pessimism Provably Efficient for Offline RL?30 Dec
  12. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
3 more
  1. 2020MOPO: Model-based Offline Policy Optimization27 May
  2. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  3. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 13

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2013Auto-Encoding Variational Bayescited by 3,980
  5. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  6. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  7. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  8. 2017Robust Adversarial Reinforcement Learningcited by 439
  9. 2020Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observationscited by 235
  10. 2023Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functionscited by 129
  11. 2016Safe Policy Improvement by Minimizing Robust Baseline Regretcited by 51
  12. 2021Learning Markov State Abstractions for Deep Reinforcement Learningcited by 42
1 more
  1. 2023Free from Bellman Completeness: Trajectory Stitching via Model-based Return-conditioned Supervised Learningcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Beyond Static Datasets: Robust Offline Policy Optimization via Vetted Synthetic Transitionscited by 0
  2. 2026Improving Online Reinforcement Learning via Bidirectional Behavior Prior Distillationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Offline Robot Reinforcement Learning with Uncertainty-Guided Human Expert Sampling
  2. 2023Bayesian Inverse Transition Learning for Offline Settings
  3. 2020Expert-Supervised Reinforcement Learning for Offline Policy Learning and Evaluation
  4. 2021Revisiting Design Choices in Offline Model-Based Reinforcement Learning
  5. 2025Evaluation-Time Policy Switching for Offline Reinforcement Learning
  6. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism
  7. 2023Beyond OOD State Actions: Supported Cross-Domain Offline Reinforcement Learning
  8. 2026When Does Trajectory-Level Supervision Permit Efficient Offline Reinforcement Learning?