mlsanityData to 7 Oct 2026

Imitating from auxiliary imperfect demonstrations via Adversarial Density Weighted Regression

arXiv 2405.20351 ↗First posted 28 May 2024cs.LG cs.AIcited by 0

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

We propose a novel one-step supervised imitation learning (IL) framework called Adversarial Density Regression (ADR). This IL framework aims to correct the policy learned on unknown-quality to match the expert distribution by utilizing demonstrations, without relying on the Bellman operator. Specifically, ADR addresses several limitations in previous IL algorithms: First, most IL algorithms are based on the Bellman operator, which inevitably suffer from cumulative offsets from sub-optimal rewards during multi-step update processes. Additionally, off-policy training frameworks suffer from Out-of-Distribution (OOD) state-actions. Second, while conservative terms help solve the OOD issue, balancing the conservative term is difficult. To address these limitations, we fully integrate a one-step density-weighted Behavioral Cloning (BC) objective for IL with auxiliary imperfect demonstration. Theoretically, we demonstrate that this adaptation can effectively correct the distribution of policies trained on unknown-quality datasets to align with the expert policy's distribution. Moreover, the difference between the empirical and the optimal value function is proportional to the upper bound of ADR's objective, indicating that minimizing ADR's objective is akin to approaching the optimal value. Experimentally, we validated the performance of ADR by conducting extensive evaluations. Specifically, ADR outperforms all of the selected IL algorithms on tasks from the Gym-Mujoco domain. Meanwhile, it achieves an 89.5% improvement over IQL when utilizing ground truth rewards on tasks from the Adroit and Kitchen domains. Our codebase will be released at: https://github.com/stevezhangzA/Adverserial_Density_Regression.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 36 of its 47 references that are in the map. First posted on arXiv 28 May 2024.

Builds on

36 of 47 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023CEIL: Generalized Contextual Imitation Learning26 Jun
  2. 2022Imitation Learning by State-Only Distribution Matching9 Feb
  3. 2022Versatile Offline Imitation from Observations and Examples via Regularized State-Occupancy Matching4 Feb
  4. 2021IQ-Learn: Inverse soft-Q Learning for Imitation23 Jun
  5. 2020Offline Learning from Demonstrations and Unlabeled Experience27 Nov
  6. 2019Imitation Learning via Off-Policy Distribution Matching10 Dec
  7. 2019Recent Advances in Imitation Learning from Observation30 May

Other references · 29

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Neural Discrete Representation Learningcited by 4,650
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  5. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  6. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  7. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  8. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  9. 2016Generative Adversarial Imitation Learningcited by 1,576
  10. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  11. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  12. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
17 more
  1. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  2. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  3. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  4. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  5. 2016Automatic chemical design using a data-driven continuous representation of moleculescited by 762
  6. 2019Behavior Regularized Offline Reinforcement Learningcited by 654
  7. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  8. 2015Policy Distillationcited by 414
  9. 2015MADE: Masked Autoencoder for Distribution Estimationcited by 306
  10. 2017Imitation from Observation: Learning to Imitate Behaviors from Raw Video via Context Translationcited by 167
  11. 2022CORL: Research-oriented Deep Offline Reinforcement Learning Librarycited by 117
  12. 2022Supported Policy Optimization for Offline Reinforcement Learningcited by 73
  13. 2023Optimal Transport for Offline Imitation Learningcited by 42
  14. 2023CLUE: Calibrated Latent Guidance for Offline Reinforcement Learningcited by 11
  15. 2024Learning from Sparse Offline Datasets via Conservative Density Estimationcited by 9
  16. 2023Controlling Structured Output Representations from Attributes using Conditional Generative Modelscited by 4
  17. 2024Context-Former: Stitching via Latent Conditioned Sequence Modelingcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2020Adversarial Soft Advantage Fitting: Imitation Learning without Policy Optimization
  2. 2025On Discovering Algorithms for Adversarial Imitation Learning
  3. 2023Sample-efficient Adversarial Imitation Learning
  4. 2026Latent Wasserstein Adversarial Imitation Learning
  5. 2021Deterministic and Discriminative Imitation (D2-Imitation): Revisiting Adversarial Imitation for Sample Efficiency
  6. 2023Unlabeled Imperfect Demonstrations in Adversarial Imitation Learning
  7. 2024Efficient Imitation Learning with Conservative World Models
  8. 2021Robust Maximum Entropy Behavior Cloning