On Discovering Algorithms for Adversarial Imitation Learning
On the map: Offline RL and imitation learning from data → Imitation learning methods
Abstract (arXiv)
Adversarial Imitation Learning (AIL) methods, while effective in settings with limited expert demonstrations, are often considered unstable. These approaches typically decompose into two components: Density Ratio (DR) estimation fracρEρπ, where a discriminator estimates the relative occupancy of state-action pairs under the policy versus the expert; and Reward Assignment (RA), where this ratio is transformed into a reward signal used to train the policy. While significant research has focused on improving density estimation, the role of reward assignment in influencing training dynamics and final policy performance has been largely overlooked. RA functions in AIL are typically derived from divergence minimization objectives, relying heavily on human design and ingenuity. In this work, we take a different approach: we investigate the discovery of data-driven RA functions, i.e, based directly on the performance of the resulting imitation policy. To this end, we leverage an LLM-guided evolutionary framework that efficiently explores the space of RA functions, yielding Discovered Adversarial Imitation Learning (DAIL), the first meta-learnt AIL algorithm. Remarkably, DAIL generalises across unseen environments and policy optimization algorithms, outperforming the current state-of-the-art of human-designed baselines. Finally, we analyse why DAIL leads to more stable training, offering novel insights into the role of RA functions in the stability of AIL. Code is publicly available: https://github.com/shshnkreddy/DAIL.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 43 of its 66 references that are in the map. First posted on arXiv 1 Oct 2025.
Builds on
43 of 66 references.
Closest ancestors · 14
Same direction, newest first
- 2024Zero-Shot Offline Imitation Learning via Optimal Transport11 Oct
- 2024EvIL: Evolution Strategies for Generalisable Imitation Learning15 Jun
- 2024Diffusion-Reward Adversarial Imitation Learning25 May
- 2024C-GAIL: Stabilizing Generative Adversarial Imitation Learning with Control Theory26 Feb
- 2023DiffAIL: Diffusion Adversarial Imitation Learning11 Dec
- 2022imitation: Clean Imitation Learning Implementations22 Nov
- 2021IQ-Learn: Inverse soft-Q Learning for Imitation23 Jun
- 2021What Matters for Adversarial Imitation Learning?1 Jun
- 2020f-GAIL: Learning f-Divergence for Generative Adversarial Imitation Learning2 Oct
- 2019Imitation Learning via Off-Policy Distribution Matching10 Dec
- 2019A Divergence Minimization Perspective on Imitation Learning Methods6 Nov
- 2019Wasserstein Adversarial Imitation Learning19 Jun
2 more
Other references · 29
- 2017Proximal Policy Optimization Algorithmscited by 14,441
- 2017Improved Training of Wasserstein GANscited by 2,223
- 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
- 2016Generative Adversarial Imitation Learningcited by 1,576
- 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
- 2017Evolution Strategies as a Scalable Alternative to Reinforcement Learningcited by 555
- 2023Eureka: Human-Level Reward Design via Coding Large Language Modelscited by 542
- 2023Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolutioncited by 521
- 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 478
- 2023Symbolic Discovery of Optimization Algorithmscited by 423
- 2021Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulationcited by 375
- 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 322
17 more
- 2024ReEvo: Large Language Models as Hyper-Heuristics with Reflective Evolutioncited by 266
- 2023EvoPrompting: Language Models for Code-Level Neural Architecture Searchcited by 129
- 2022Discovered Policy Optimisationcited by 103
- 2023A Tutorial on Meta-Reinforcement Learningcited by 102
- 2022Optimal Transport Tools (OTT): A JAX Toolbox for all things Wassersteincited by 96
- 2016Learning values across many orders of magnitudecited by 80
- 2024DrEureka: Language Model Guided Sim-To-Real Transfercited by 78
- 2020Discovering Reinforcement Learning Algorithmscited by 75
- 2024OMNI-EPIC: Open-endedness via Models of human Notions of Interestingness with Environments Programmed in Codecited by 67
- 2023Optimal Transport for Offline Imitation Learningcited by 42
- 2022Misspecification in Inverse Reinforcement Learningcited by 27
- 2024Discovering Temporally-Aware Reinforcement Learning Algorithmscited by 18
- 2024Can Learned Optimization Make Reinforcement Learning Less Difficult?cited by 14
- 2024Non-Adversarial Inverse Reinforcement Learning via Successor Feature Matchingcited by 10
- 2025On Generalization Across Environments In Multi-Objective Reinforcement Learningcited by 8
- 2025How Should We Meta-Learn Reinforcement Learning Algorithms?cited by 5
- 2024Preserving the Privacy of Reward Functions in MDPs through Deceptioncited by 1
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2026Adversarial Imitation Learning with General Function Approximation: Theoretical Analysis and Practical Algorithms
- 2018Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning
- 2026Provably Efficient Policy-Reward Co-Pretraining for Adversarial Imitation Learning
- 2021Reward function shape exploration in adversarial imitation learning: an empirical study
- 2019A New Framework for Query Efficient Active Imitation Learning
- 2021Learning from Guided Play: A Scheduled Hierarchical Approach for Improving Exploration in Adversarial Imitation Learning
- 2023Provably Efficient Adversarial Imitation Learning with Unknown Transitions
- 2024Imitating from auxiliary imperfect demonstrations via Adversarial Density Weighted Regression