mlsanityData to 7 Oct 2026

Offline Imitation Learning with Suboptimal Demonstrations via Relaxed Distribution Matching

arXiv 2303.02569 ↗First posted 5 Mar 2023cs.LGcited by 24

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

Offline imitation learning (IL) promises the ability to learn performant policies from pre-collected demonstrations without interactions with the environment. However, imitating behaviors fully offline typically requires numerous expert data. To tackle this issue, we study the setting where we have limited expert data and supplementary suboptimal data. In this case, a well-known issue is the distribution shift between the learned policy and the behavior policy that collects the offline data. Prior works mitigate this issue by regularizing the KL divergence between the stationary state-action distributions of the learned policy and the behavior policy. We argue that such constraints based on exact distribution matching can be overly conservative and hamper policy learning, especially when the imperfect offline data is highly suboptimal. To resolve this issue, we present RelaxDICE, which employs an asymmetrically-relaxed f-divergence for explicit support regularization. Specifically, instead of driving the learned policy to exactly match the behavior policy, we impose little penalty whenever the density ratio between their stationary state-action distributions is upper bounded by a constant. Note that such formulation leads to a nested min-max optimization problem, which causes instability in practice. RelaxDICE addresses this challenge by supporting a closed-form solution for the inner maximization problem. Extensive empirical study shows that our method significantly outperforms the best prior offline IL method in six standard continuous control environments with over 30% performance gain on average, across 22 settings where the imperfect dataset is highly suboptimal.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 31 of its 45 references that are in the map. First posted on arXiv 5 Mar 2023.

Builds on

31 of 45 references.

Closest ancestors · 6

Same direction, newest first

  1. 2021Feedback in Imitation Learning: The Three Regimes of Covariate Shift4 Feb
  2. 2019Imitation Learning via Off-Policy Distribution Matching10 Dec
  3. 2019A Divergence Minimization Perspective on Imitation Learning Methods6 Nov
  4. 2019Imitation Learning as f-Divergence Minimization30 May
  5. 2019Imitation Learning from Imperfect Demonstration27 Jan
  6. 2018Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning9 Sep

Other references · 25

  1. 2021Convex optimizationcited by 2,479
  2. 2017Improved Training of Wasserstein GANscited by 2,223
  3. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  4. 2016Generative Adversarial Imitation Learningcited by 1,576
  5. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  6. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  7. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  8. 2019Behavior Regularized Offline Reinforcement Learningcited by 654
  9. 2016f-GAN: Training Generative Neural Samplers using Variational Divergence Minimizationcited by 504
  10. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
  11. 2021COMBO: Conservative Offline Model-Based Policy Optimizationcited by 365
  12. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 285
13 more
  1. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 256
  2. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  3. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 187
  4. 2020GenDICE: Generalized Offline Estimation of Stationary Valuescited by 122
  5. 2022Offline Reinforcement Learning with Realizability and Single-policy Concentrabilitycited by 122
  6. 2021OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimationcited by 121
  7. 2018Supervised Reinforcement Learning with Recurrent Neural Network for Dynamic Treatment Recommendationcited by 101
  8. 2020Off-Policy Evaluation via the Regularized Lagrangiancited by 86
  9. 2020GradientDICE: Rethinking Generalized Offline Estimation of Stationary Valuescited by 70
  10. 2019Domain Adaptation with Asymmetrically-Relaxed Distribution Alignmentcited by 61
  11. 2021Showing Your Offline Reinforcement Learning Work: Online Evaluation Budget Matterscited by 24
  12. 2020Offline Reinforcement Learning Hands-Oncited by 6
  13. 2021A Unified Framework for Multi-distribution Density Ratio Estimationcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 7; 2025: 8; 2026: 5.

24 papers.

Sort:

By area:

  1. 2024SEABO: A Simple Search-Based Method for Offline Imitation Learningcited by 14
  2. 2024Learning from Imperfect Demonstrations with Self-Supervision for Robotic Manipulationcited by 13
  3. 2024SPRINQL: Sub-optimal Demonstrations driven Offline Imitation Learningcited by 7
  4. 2025Inverse Reinforcement Learning Meets Large Language Model Post-Training: Basics, Advances, and Opportunitiescited by 7
  5. 2023Marginalized Importance Sampling for Off-Environment Policy Evaluationcited by 5
  6. 2024Adversarial Imitation Learning via Boostingcited by 5
  7. 2024Hybrid Reinforcement Learning from Offline Observation Alonecited by 5
  8. 2025IL-SOAR : Imitation Learning with Soft Optimistic Actor cRiticcited by 5
  9. 2023Learning Distributions via Monte-Carlo Marginalizationcited by 4
  10. 2024Simple Ingredients for Offline Reinforcement Learningcited by 4
  11. 2023Imitation Learning from Purified Demonstrationscited by 3
  12. 2025AEGIS: Human Attention-based Explainable Guidance for Intelligent Vehicle Systemscited by 2
12 more
  1. 2025Demonstration Sidetracks: Categorizing Systematic Non-Optimality in Human Demonstrationscited by 2
  2. 2026Supervised Mixture-of-Experts for Surgical Grasping and Retractioncited by 2
  3. 2026Neuro-Cognitive Reward Modeling for Human-Centered Autonomous Vehicle Controlcited by 2
  4. 2023Mildly Constrained Evaluation Policy for Offline Reinforcement Learningcited by 1
  5. 2025AAPA: Adversarially Anchored Preference Alignment for Post-Training of Large Language Modelscited by 1
  6. 2025Restoring Noisy Demonstration for Imitation Learning With Diffusion Modelscited by 1
  7. 2026Semi-Supervised Cross-Domain Imitation Learningcited by 1
  8. 2024UNIQ: Offline Inverse Q-learning for Avoiding Undesirable Demonstrationscited by 0
  9. 2025Learning from Observation: A Survey of Recent Advancescited by 0
  10. 2025When a Robot is More Capable than a Human: Learning from Constrained Demonstratorscited by 0
  11. 2026General Flexible f-divergence for Challenging Offline RL Datasets with Low Stochasticity and Diverse Behavior Policiescited by 0
  12. 2026Language-Critique Imitation Learning from Suboptimal Demonstrationscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Learning What to Do and What Not To Do: Offline Imitation from Expert and Undesirable Demonstrations
  2. 2021Mitigating Covariate Shift in Imitation Learning via Offline Data Without Great Coverage
  3. 2024How to Leverage Diverse Demonstrations in Offline Imitation Learning
  4. 2024Offline Imitation Learning by Controlling the Effective Planning Horizon
  5. 2025Using Non-Expert Data to Robustify Imitation Learning via Offline Reinforcement Learning
  6. 2025Offline Imitation Learning upon Arbitrary Demonstrations by Pre-Training Dynamics Representations
  7. 2022Discriminator-Weighted Offline Imitation Learning from Suboptimal Demonstrations
  8. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism