mlsanityData to 7 Oct 2026

UNIQ: Offline Inverse Q-learning for Avoiding Undesirable Demonstrations

arXiv 2410.08307 ↗First posted 10 Oct 2024cs.LG cs.AIcited by 0

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

We address the problem of offline learning a policy that avoids undesirable demonstrations. Unlike conventional offline imitation learning approaches that aim to imitate expert or near-optimal demonstrations, our setting involves avoiding undesirable behavior (specified using undesirable demonstrations). To tackle this problem, unlike standard imitation learning where the aim is to minimize the distance between learning policy and expert demonstrations, we formulate the learning task as maximizing a statistical distance, in the space of state-action stationary distributions, between the learning policy and the undesirable policy. This significantly different approach results in a novel training objective that necessitates a new algorithm to address it. Our algorithm, UNIQ, tackles these challenges by building on the inverse Q-learning framework, framing the learning problem as a cooperative (non-adversarial) task. We then demonstrate how to efficiently leverage unlabeled data for practical training. Our method is evaluated on standard benchmark environments, where it consistently outperforms state-of-the-art baselines. The code implementation can be accessed at: https://github.com/hmhuy0/UNIQ.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 23 of its 35 references that are in the map. First posted on arXiv 10 Oct 2024.

Builds on

23 of 35 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024SPRINQL: Sub-optimal Demonstrations driven Offline Imitation Learning20 Feb
  2. 2023Offline Imitation Learning with Suboptimal Demonstrations via Relaxed Distribution Matching5 Mar
  3. 2022Discriminator-Weighted Offline Imitation Learning from Suboptimal Demonstrations20 Jul
  4. 2022Versatile Offline Imitation from Observations and Examples via Regularized State-Occupancy Matching4 Feb
  5. 2021IQ-Learn: Inverse soft-Q Learning for Imitation23 Jun
  6. 2020Non-Adversarial Imitation Learning and its Connections to Adversarial Methods8 Aug
  7. 2020Primal Wasserstein Imitation Learning8 Jun
  8. 2019Imitation Learning via Off-Policy Distribution Matching10 Dec
  9. 2018HG-DAgger: Interactive Imitation Learning with Human Experts5 Oct

Other references · 14

  1. 2016Generative Adversarial Imitation Learningcited by 1,576
  2. 2018ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worstcited by 391
  3. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 298
  4. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 256
  5. 2017Positive-Unlabeled Learning with Non-Negative Risk Estimatorcited by 217
  6. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 187
  7. 2023Safety-Gymnasium: A Unified Safe Reinforcement Learning Benchmarkcited by 135
  8. 2021B-Pref: Benchmarking Preference-Based Reinforcement Learningcited by 106
  9. 2023Preference Transformer: Modeling Human Preferences using Transformers for RLcited by 77
  10. 2023Inverse Preference Learning: Preference-based RL without a Reward Functioncited by 68
  11. 2023Beyond Reward: Offline Preference-guided Policy Optimizationcited by 35
  12. 2023LS-IQ: Implicit Reward Regularization for Inverse Reinforcement Learningcited by 28
2 more
  1. 2022LobsDICE: Offline Learning from Observation via Stationary Distribution Correction Estimationcited by 22
  2. 2023Imitate the Good and Avoid the Bad: An Incremental Approach to Safe Reinforcement Learningcited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Learning What to Do and What Not To Do: Offline Imitation from Expert and Undesirable Demonstrations
  2. 2025Inverse Q-Learning Done Right: Offline Imitation Learning in Qπ-Realizable MDPs
  3. 2020Offline Learning from Demonstrations and Unlabeled Experience
  4. 2025Using Non-Expert Data to Robustify Imitation Learning via Offline Reinforcement Learning
  5. 2023Aligning Human Intent from Imperfect Demonstrations with Confidence-based Inverse soft-Q Learning
  6. 2023Skill Disentanglement for Imitation Learning from Suboptimal Demonstrations
  7. 2021Offline Inverse Reinforcement Learning
  8. 2024How to Leverage Diverse Demonstrations in Offline Imitation Learning