Deterministic and Discriminative Imitation (D2-Imitation): Revisiting Adversarial Imitation for Sample Efficiency
On the map: Offline RL and imitation learning from data → Imitation learning methods
Abstract (arXiv)
Sample efficiency is crucial for imitation learning methods to be applicable in real-world applications. Many studies improve sample efficiency by extending adversarial imitation to be off-policy regardless of the fact that these off-policy extensions could either change the original objective or involve complicated optimization. We revisit the foundation of adversarial imitation and propose an off-policy sample efficient approach that requires no adversarial training or min-max optimization. Our formulation capitalizes on two key insights: (1) the similarity between the Bellman equation and the stationary state-action distribution equation allows us to derive a novel temporal difference (TD) learning approach; and (2) the use of a deterministic policy simplifies the TD learning. Combined, these insights yield a practical algorithm, Deterministic and Discriminative Imitation (D2-Imitation), which operates by first partitioning samples into two replay buffers and then learning a deterministic policy via off-policy reinforcement learning. Our empirical results show that D2-Imitation is effective in achieving good sample efficiency, outperforming several off-policy extension approaches of adversarial imitation on many control tasks.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 19 of its 31 references that are in the map. First posted on arXiv 11 Dec 2021.
Builds on
19 of 31 references.
Closest ancestors · 8
Same direction, newest first
- 2021SoftDICE for Imitation Learning: Rethinking Off-policy Distribution Matching6 Jun
- 2020Non-Adversarial Imitation Learning and its Connections to Adversarial Methods8 Aug
- 2020Primal Wasserstein Imitation Learning8 Jun
- 2019Imitation Learning via Off-Policy Distribution Matching10 Dec
- 2019A Divergence Minimization Perspective on Imitation Learning Methods6 Nov
- 2019Imitation Learning as f-Divergence Minimization30 May
- 2019Random Expert Distillation: Imitation Learning via Expert Policy Support Estimation16 May
- 2018Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning9 Sep
Other references · 11
- 2014Adam: A Method for Stochastic Optimizationcited by 37,240
- 2017Proximal Policy Optimization Algorithmscited by 15,215
- 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
- 2015Continuous control with deep reinforcement learningcited by 3,407
- 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
- 2017Improved Training of Wasserstein GANscited by 2,235
- 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
- 2016OpenAI Gymcited by 1,736
- 2016Generative Adversarial Imitation Learningcited by 1,597
- 2018Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimationcited by 227
- 2015On Convergence of Emphatic Temporal-Difference Learningcited by 21
Built on by
Citing papers per year: 2022: 1; 2023: 1; 2024: 0; 2025: 1.
3 papers.
Similar, not cited
Close in wording, no citation either way
- 2023Sample-efficient Adversarial Imitation Learning
- 2018Sample-Efficient Imitation Learning via Generative Adversarial Nets
- 2022Planning for Sample Efficient Imitation Learning
- 2024Provably Efficient Off-Policy Adversarial Imitation Learning with Convergence Guarantees
- 2019Sample-efficient Adversarial Imitation Learning from Observation
- 2023Provably Efficient Adversarial Imitation Learning with Unknown Transitions
- 2020Adversarial Soft Advantage Fitting: Imitation Learning without Policy Optimization
- 2021On Generalization of Adversarial Imitation Learning and Beyond