Non-Adversarial Imitation Learning Provably Free of Compounding Errors: The Value Flow Mechanism
On the map: Offline RL and imitation learning from data → Imitation learning methods
Abstract (arXiv)
Adversarial imitation learning (AIL) achieves high-quality imitation by mitigating compounding errors inherent to behavioral cloning (BC), yet its adversarial optimization frequently leads to training instability. A class of non-adversarial Q-based imitation learning (IL) methods, exemplified by IQ-Learn, has emerged to address this instability and is widely believed to outperform BC by leveraging online environment interactions. In this paper, we revisit IQ-Learn and prove that it in fact reduces to BC: it admits an imitation gap lower bound with quadratic dependence on the horizon and therefore remains susceptible to compounding errors. Our theoretical analysis reveals why online interactions fail to help: IQ-Learn uniformly suppresses Q-values for all actions at states not covered by demonstrations, preventing generalization beyond demonstrations. To address this fundamental limitation, we introduce Dual Q-DM, a new Q-based IL method built on Bellman constraints. Crucially, Bellman constraints drive value flow: Q-values propagate from demonstrated to unvisited states through environment dynamics, enabling generalization beyond demonstrations. We prove that Dual Q-DM is equivalent to AIL and can recover expert actions at unvisited states, thereby mitigating compounding errors. To the best of our knowledge, Dual Q-DM is the first non-adversarial IL method that is theoretically guaranteed to eliminate compounding errors. Experimental results further corroborate our theoretical findings.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 33 of its 51 references that are in the map. First posted on arXiv 24 Mar 2026.
Builds on
33 of 51 references.
Closest ancestors · 23
Same direction, newest first
- 2025Inverse Q-Learning Done Right: Offline Imitation Learning in Qπ-Realizable MDPs26 May
- 2025The Pitfalls of Imitation Learning when Actions are Continuous12 Mar
- 2025IL-SOAR : Imitation Learning with Soft Optimistic Actor cRitic27 Feb
- 2024Provably and Practically Efficient Adversarial Imitation Learning with General Function Approximation1 Nov
- 2024Is Behavior Cloning All You Need? Understanding Horizon in Imitation Learning20 Jul
- 2024Imitation Learning in Discounted Linear MDPs without exploration assumptions3 May
- 2023Provably Efficient Adversarial Imitation Learning with Unknown Transitions11 Jun
- 2023Coherent Soft Imitation Learning25 May
- 2022Proximal Point Imitation Learning22 Sep
- 2022Understanding Adversarial Imitation Learning in Small Sample Regime: A Stage-coupled Analysis3 Aug
- 2021Provably Efficient Generative Adversarial Imitation Learning for Online and Offline Setting with Linear Function Approximation19 Aug
- 2021IQ-Learn: Inverse soft-Q Learning for Imitation23 Jun
11 more
- 2021What Matters for Adversarial Imitation Learning?1 Jun
- 2020Error Bounds of Imitating Policies and Environments22 Oct
- 2020Toward the Fundamental Limits of Imitation Learning13 Sep
- 2020Primal Wasserstein Imitation Learning8 Jun
- 2020On Computation and Generalization of Generative Adversarial Imitation Learning9 Jan
- 2019Imitation Learning via Off-Policy Distribution Matching10 Dec
- 2019A Divergence Minimization Perspective on Imitation Learning Methods6 Nov
- 2019Imitation Learning as f-Divergence Minimization30 May
- 2019Provably Efficient Imitation Learning from Observation Alone27 May
- 2018An Algorithmic Perspective on Imitation Learning16 Nov
- 2018Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning9 Sep
Other references · 10
- 2022Training language models to follow instructions with human feedbackcited by 15,712
- 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
- 2021Convex optimizationcited by 2,479
- 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,297
- 2016Generative Adversarial Imitation Learningcited by 1,576
- 2023Dual RL: Unification and New Methods for Reinforcement and Imitation Learningcited by 48
- 2023Inverse Reinforcement Learning without Reinforcement Learningcited by 35
- 2024Hybrid Inverse Reinforcement Learningcited by 31
- 2023LS-IQ: Implicit Reward Regularization for Inverse Reinforcement Learningcited by 28
- 2024Imitating Language via Scalable Inverse Reinforcement Learningcited by 21
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Near-Optimal Second-Order Guarantees for Model-Based Adversarial Imitation Learning
- 2021On Generalization of Adversarial Imitation Learning and Beyond
- 2026Adversarial Imitation Learning with General Function Approximation: Theoretical Analysis and Practical Algorithms
- 2026Provably Efficient Policy-Reward Co-Pretraining for Adversarial Imitation Learning
- 2022Rethinking ValueDice: Does It Really Improve Performance?
- 2024Reward-free World Models for Online Imitation Learning
- 2022ABC: Adversarial Behavioral Cloning for Offline Mode-Seeking Imitation Learning
- 2024Efficient Imitation Learning with Conservative World Models