mlsanityData to 14 Oct 2026

Learning What to Do and What Not To Do: Offline Imitation from Expert and Undesirable Demonstrations

arXiv 2505.21182 ↗First posted 27 May 2025cs.LG cs.AIcited by 0

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

Offline imitation learning typically learns from expert and unlabeled demonstrations, yet often overlooks the valuable signal in explicitly undesirable behaviors. In this work, we study offline imitation learning from contrasting behaviors, where the dataset contains both expert and undesirable demonstrations. We propose a novel formulation that optimizes a difference of KL divergences over the state-action visitation distributions of expert and undesirable (or bad) data. Although the resulting objective is a DC (Difference-of-Convex) program, we prove that it becomes convex when expert demonstrations outweigh undesirable demonstrations, enabling a practical and stable non-adversarial training objective. Our method avoids adversarial training and handles both positive and negative demonstrations in a unified framework. Extensive experiments on standard offline imitation learning benchmarks demonstrate that our approach consistently outperforms state-of-the-art baselines.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 34 of its 44 references that are in the map. First posted on arXiv 27 May 2025.

Builds on

34 of 44 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024How to Leverage Diverse Demonstrations in Offline Imitation Learning24 May
  2. 2024SPRINQL: Sub-optimal Demonstrations driven Offline Imitation Learning20 Feb
  3. 2022Discriminator-Weighted Offline Imitation Learning from Suboptimal Demonstrations20 Jul
  4. 2022Versatile Offline Imitation from Observations and Examples via Regularized State-Occupancy Matching4 Feb
  5. 2021Confidence-Aware Imitation Learning from Demonstrations with Varying Optimality27 Oct
  6. 2021IQ-Learn: Inverse soft-Q Learning for Imitation23 Jun
  7. 2020Non-Adversarial Imitation Learning and its Connections to Adversarial Methods8 Aug
  8. 2019Imitation Learning via Off-Policy Distribution Matching10 Dec
  9. 2019Imitation Learning from Imperfect Demonstration27 Jan
  10. 2018Behavioral Cloning from Observation4 May

Other references · 24

  1. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,838
  2. 2016Generative Adversarial Imitation Learningcited by 1,597
  3. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,568
  4. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  5. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  6. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  7. 2017Value-Decomposition Networks For Cooperative Multi-Agent Learningcited by 503
  8. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 260
  9. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 188
  10. 2021OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimationcited by 121
  11. 2020The Ingredients of Real-World Robotic Reinforcement Learningcited by 117
  12. 2023Extreme Q-Learning: MaxEnt RL without Entropycited by 116
12 more
  1. 2023Preference Transformer: Modeling Human Preferences using Transformers for RLcited by 79
  2. 2023Inverse Preference Learning: Preference-based RL without a Reward Functioncited by 68
  3. 2023Dual RL: Unification and New Methods for Reinforcement and Imitation Learningcited by 48
  4. 2021Learning Multimodal Rewards from Rankingscited by 41
  5. 2023Beyond Reward: Offline Preference-guided Policy Optimizationcited by 35
  6. 2023LS-IQ: Implicit Reward Regularization for Inverse Reinforcement Learningcited by 29
  7. 2022LobsDICE: Offline Learning from Observation via Stationary Distribution Correction Estimationcited by 23
  8. 2024ODICE: Revealing the Mystery of Distribution Correction Estimation via Orthogonal-gradient Updatecited by 21
  9. 2024Learning from negative feedback, or positive feedback or bothcited by 13
  10. 2024Offline Safe Reinforcement Learning Using Trajectory Classificationcited by 9
  11. 2023Imitate the Good and Avoid the Bad: An Incremental Approach to Safe Reinforcement Learningcited by 7
  12. 2024Semantic Loss Guided Data Efficient Supervised Fine Tuning for Safe Responses in LLMscited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Using Non-Expert Data to Robustify Imitation Learning via Offline Reinforcement Learning
  2. 2023Offline Imitation Learning with Suboptimal Demonstrations via Relaxed Distribution Matching
  3. 2024UNIQ: Offline Inverse Q-learning for Avoiding Undesirable Demonstrations
  4. 2022Model-based Offline Imitation Learning with Non-expert Data
  5. 2023Unlabeled Imperfect Demonstrations in Adversarial Imitation Learning
  6. 2022Discriminator-Guided Model-Based Offline Imitation Learning
  7. 2024Offline Imitation Learning by Controlling the Effective Planning Horizon
  8. 2021Mitigating Covariate Shift in Imitation Learning via Offline Data Without Great Coverage