mlsanityData to 14 Oct 2026

Discriminator-Weighted Offline Imitation Learning from Suboptimal Demonstrations

arXiv 2207.10050 ↗First posted 20 Jul 2022cs.LG cs.AIcited by 89

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

We study the problem of offline Imitation Learning (IL) where an agent aims to learn an optimal expert behavior policy without additional online environment interactions. Instead, the agent is provided with a supplementary offline dataset from suboptimal behaviors. Prior works that address this problem either require that expert data occupies the majority proportion of the offline dataset, or need to learn a reward function and perform offline reinforcement learning (RL) afterwards. In this paper, we aim to address the problem without additional steps of reward learning and offline RL training for the case when demonstrations contain a large proportion of suboptimal data. Built upon behavioral cloning (BC), we introduce an additional discriminator to distinguish expert and non-expert data. We propose a cooperation framework to boost the learning of both tasks, Based on this framework, we design a new IL algorithm, where the outputs of discriminator serve as the weights of the BC loss. Experimental results show that our proposed algorithm achieves higher returns and faster training speed compared to baseline algorithms.

Key facts. Cited by 89 papers in the map (data through 14 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 39 of its 68 references that are in the map. First posted on arXiv 20 Jul 2022.

Builds on

39 of 68 references.

Closest ancestors · 10

Same direction, newest first

  1. 2021Confidence-Aware Imitation Learning from Demonstrations with Varying Optimality27 Oct
  2. 2021Implicit Behavioral Cloning1 Sep
  3. 2021IQ-Learn: Inverse soft-Q Learning for Imitation23 Jun
  4. 2021Mitigating Covariate Shift in Imitation Learning via Offline Data Without Great Coverage6 Jun
  5. 2021SoftDICE for Imitation Learning: Rethinking Off-policy Distribution Matching6 Jun
  6. 2020Offline Learning from Demonstrations and Unlabeled Experience27 Nov
  7. 2020Strictly Batch Imitation Learning by Energy-based Distribution Matching25 Jun
  8. 2020Combating False Negatives in Adversarial Imitation Learning2 Feb
  9. 2019Imitation Learning via Off-Policy Distribution Matching10 Dec
  10. 2019Imitation Learning from Imperfect Demonstration27 Jan

Other references · 29

  1. 2017Attention Is All You Needcited by 60,361
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  4. 2015Trust Region Policy Optimizationcited by 2,580
  5. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  6. 2016Generative Adversarial Imitation Learningcited by 1,597
  7. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  8. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  9. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  10. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  11. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 715
  12. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
17 more
  1. 2019Behavior Regularized Offline Reinforcement Learningcited by 662
  2. 2020MOPO: Model-based Offline Policy Optimizationcited by 636
  3. 2019On Evaluating Adversarial Robustnesscited by 447
  4. 2021COMBO: Conservative Offline Model-Based Policy Optimizationcited by 370
  5. 2020Critic Regularized Regressioncited by 275
  6. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 260
  7. 2020Hyperparameter Selection for Offline Reinforcement Learningcited by 114
  8. 2017Implicit Regularization in Deep Learningcited by 90
  9. 2021Benchmarks for Deep Off-Policy Evaluationcited by 89
  10. 2016Robustness of classifiers: from adversarial to random noisecited by 88
  11. 2021Model Selection for Offline Reinforcement Learning: Practical Considerations for Healthcare Settingscited by 68
  12. 2021Replacing Rewards with Examples: Example-Based Policy Search via Recursive Classificationcited by 38
  13. 2019Positive-Unlabeled Reward Learningcited by 37
  14. 2020Offline Policy Selection under Uncertaintycited by 31
  15. 2020Semi-supervised reward learning for offline reinforcement learningcited by 23
  16. 2021Offline Inverse Reinforcement Learningcited by 14
  17. 2021Offline Policy Comparison under Limited Historical Agent-Environment Interactionscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 26; 2024: 22; 2025: 19; 2026: 20.

89 papers.

Sort:

By area:

  1. 2023MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrationscited by 327
  2. 2023OpenChat: Advancing Open-source Language Models with Mixed-Quality Datacited by 252
  3. 2023Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularizationcited by 116
  4. 2023Diffusion Model is an Effective Planner and Data Synthesizer for Multi-Task Reinforcement Learningcited by 115
  5. 2022Robot Learning on the Job: Human-in-the-Loop Autonomy and Learning During Deploymentcited by 108
  6. 2023Policy Expansion for Bridging Offline-to-Online Reinforcement Learningcited by 104
  7. 2022A Policy-Guided Imitation Approach for Offline Reinforcement Learningcited by 59
  8. 2024Diffusion-DICE: In-Sample Diffusion Guidance for Offline Reinforcement Learningcited by 42
  9. 2023PROTO: Iterative Policy Regularized Offline-to-Online Reinforcement Learningcited by 29
  10. 2023Beyond Uniform Sampling: Offline Reinforcement Learning with Imbalanced Datasetscited by 24
  11. 2023Coherent Soft Imitation Learningcited by 22
  12. 2023Harnessing Mixed Offline Reinforcement Learning Datasets via Trajectory Weightingcited by 22
38 more
  1. 2024ODICE: Revealing the Mystery of Distribution Correction Estimation via Orthogonal-gradient Updatecited by 21
  2. 2026PALM: Progress-Aware Policy Learning via Affordance Reasoning for Long-Horizon Robotic Manipulationcited by 21
  3. 2023Survival Instinct in Offline Reinforcement Learningcited by 20
  4. 2023MAHALO: Unifying Offline Reinforcement Learning and Imitation Learning from Observationscited by 18
  5. 2025Curating Demonstrations using Online Experiencecited by 18
  6. 2024Instruction-Guided Visual Maskingcited by 17
  7. 2023Mind the Gap: Offline Policy Optimization for Imperfect Rewardscited by 15
  8. 2024Bielik 7B v0.1: A Polish Language Model -- Development, Insights, and Evaluationcited by 15
  9. 2025Reinforcement Learning for Flow-Matching Policiescited by 15
  10. 2024Learning from Imperfect Demonstrations with Self-Supervision for Robotic Manipulationcited by 14
  11. 2024SEABO: A Simple Search-Based Method for Offline Imitation Learningcited by 14
  12. 2024Offline Imitation Learning Through Graph Search and Retrievalcited by 14
  13. 2023CEIL: Generalized Contextual Imitation Learningcited by 13
  14. 2024Adaptive Advantage-Guided Policy Regularization for Offline Reinforcement Learningcited by 13
  15. 2023A Survey of Demonstration Learningcited by 12
  16. 2024How to Leverage Diverse Demonstrations in Offline Imitation Learningcited by 12
  17. 2023CLUE: Calibrated Latent Guidance for Offline Reinforcement Learningcited by 11
  18. 2024Inverse Reinforcement Learning by Estimating Expertise of Demonstratorscited by 11
  19. 2023Guarded Policy Optimization with Imperfect Online Demonstrationscited by 10
  20. 2026Dexora: Open-source VLA for High-DoF Bimanual Dexteritycited by 9
  21. 2023Improving Behavioural Cloning with Positive Unlabeled Learningcited by 8
  22. 2024Constrained Ensemble Exploration for Unsupervised Skill Discoverycited by 8
  23. 2024Offline Imitation Learning with Model-based Reverse Augmentationcited by 8
  24. 2024ComaDICE: Offline Cooperative Multi-Agent Reinforcement Learning with Stationary Distribution Shift Regularizationcited by 8
  25. 2025Posterior Behavioral Cloning: Pretraining BC Policies for Efficient RL Finetuningcited by 8
  26. 2023Offline Imitation Learning with Variational Counterfactual Reasoningcited by 7
  27. 2024SPRINQL: Sub-optimal Demonstrations driven Offline Imitation Learningcited by 7
  28. 2024OLLIE: Imitation Learning from Offline Pretraining to Online Finetuningcited by 6
  29. 2023Skill Disentanglement for Imitation Learning from Suboptimal Demonstrationscited by 5
  30. 2023Offline Imitation from Observation via Primal Wasserstein State Occupancy Matchingcited by 5
  31. 2024An Offline Adaptation Framework for Constrained Multi-Objective Reinforcement Learningcited by 5
  32. 2024Robust Offline Imitation Learning from Diverse Auxiliary Datacited by 5
  33. 2025An Optimal Discriminator Weighted Imitation Perspective for Reinforcement Learningcited by 4
  34. 2025PVPO: Pre-Estimated Value-Based Policy Optimization for Agentic Reasoningcited by 4
  35. 2023Unlabeled Imperfect Demonstrations in Adversarial Imitation Learningcited by 3
  36. 2023Contrastive Example-Based Controlcited by 3
  37. 2023Imitation Learning from Purified Demonstrationscited by 3
  38. 2023A Simple Solution for Offline Imitation from Observations and Examples with Possibly Incomplete Trajectoriescited by 3

50 most cited of 89 All 89 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022Model-based Offline Imitation Learning with Non-expert Data
  2. 2022Discriminator-Guided Model-Based Offline Imitation Learning
  3. 2025Using Non-Expert Data to Robustify Imitation Learning via Offline Reinforcement Learning
  4. 2023Offline Imitation Learning with Suboptimal Demonstrations via Relaxed Distribution Matching
  5. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism
  6. 2022When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning?
  7. 2018Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning
  8. 2024Is Behavior Cloning All You Need? Understanding Horizon in Imitation Learning