mlsanityData to 7 Oct 2026

Of Moments and Matching: A Game-Theoretic Framework for Closing the Imitation Gap

arXiv 2103.03236 ↗First posted 4 Mar 2021cs.LG cs.RO stat.MLcited by 80

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

We provide a unifying view of a large family of previous imitation learning algorithms through the lens of moment matching. At its core, our classification scheme is based on whether the learner attempts to match (1) reward or (2) action-value moments of the expert's behavior, with each option leading to differing algorithmic approaches. By considering adversarially chosen divergences between learner and expert behavior, we are able to derive bounds on policy performance that apply for all algorithms in each of these classes, the first to our knowledge. We also introduce the notion of moment recoverability, implicit in many previous analyses of imitation learning, which allows us to cleanly delineate how well each algorithmic family is able to mitigate compounding errors. We derive three novel algorithm templates (AdVIL, AdRIL, and DAeQuIL) with strong guarantees, simple implementation, and competitive empirical performance.

Key facts. Cited by 80 papers in the map (data through 7 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 21 of its 49 references that are in the map. First posted on arXiv 4 Mar 2021.

Builds on

21 of 49 references.

Closest ancestors · 10

Same direction, newest first

  1. 2021Feedback in Imitation Learning: The Three Regimes of Covariate Shift4 Feb
  2. 2020Toward the Fundamental Limits of Imitation Learning13 Sep
  3. 2020Non-Adversarial Imitation Learning and its Connections to Adversarial Methods8 Aug
  4. 2020Strictly Batch Imitation Learning by Energy-based Distribution Matching25 Jun
  5. 2020Adversarial Soft Advantage Fitting: Imitation Learning without Policy Optimization23 Jun
  6. 2020Primal Wasserstein Imitation Learning8 Jun
  7. 2019Imitation Learning via Off-Policy Distribution Matching10 Dec
  8. 2019Imitation Learning as f-Divergence Minimization30 May
  9. 2019Provably Efficient Imitation Learning from Observation Alone27 May
  10. 2018Multi-Agent Generative Adversarial Imitation Learning26 Jul

Other references · 11

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  4. 2017Improved Training of Wasserstein GANscited by 2,223
  5. 2016Generative Adversarial Imitation Learningcited by 1,576
  6. 2016End to End Learning for Self-Driving Carscited by 1,000
  7. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
  8. 2018Formal Limitations on the Measurement of Mutual Informationcited by 189
  9. 2014Reinforcement and Imitation Learning via Interactive No-Regret Learningcited by 116
  10. 2018Dual Policy Iterationcited by 22
  11. 2019Scaled Autonomy: Enabling Human Operators to Control Robot Fleetscited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 12; 2023: 20; 2024: 18; 2025: 20; 2026: 9.

80 papers.

Sort:

By area:

  1. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 244
  2. 2024A Minimaximalist Approach to Reinforcement Learning from Human Feedbackcited by 141
  3. 2022Adversarially Trained Actor Critic for Offline Reinforcement Learningcited by 111
  4. 2024REBEL: Reinforcement Learning via Regressing Relative Rewardscited by 66
  5. 2025All Roads Lead to Likelihood: The Value of Reinforcement Learning in Fine-Tuningcited by 58
  6. 2025From Foresight to Forethought: VLM-In-the-Loop Policy Steering via Latent Alignmentcited by 51
  7. 2025Zero-Shot Whole-Body Humanoid Control via Behavioral Foundation Modelscited by 51
  8. 2023Dual RL: Unification and New Methods for Reinforcement and Imitation Learningcited by 48
  9. 2022Hierarchical Model-Based Imitation Learning for Planning in Autonomous Drivingcited by 43
  10. 2023Inverse Reinforcement Learning without Reinforcement Learningcited by 35
  11. 2022Sequence Model Imitation Learning with Unobserved Contextscited by 34
  12. 2024Hybrid Inverse Reinforcement Learningcited by 31
38 more
  1. 2022Causal Imitation Learning under Temporally Correlated Noisecited by 28
  2. 2023LS-IQ: Implicit Reward Regularization for Inverse Reinforcement Learningcited by 28
  3. 2025Multi-Turn Code Generation Through Single-Step Rewardscited by 27
  4. 2023CCIL: Continuity-based Data Augmentation for Corrective Imitation Learningcited by 26
  5. 2022Minimax Optimal Online Imitation Learning via Replay Estimationcited by 25
  6. 2023SMORE: Score Models for Offline Goal-Conditioned Reinforcement Learningcited by 21
  7. 2023CLARE: Conservative Model-Based Reward Learning for Offline Inverse Reinforcement Learningcited by 20
  8. 2022A Ranking Game for Imitation Learningcited by 18
  9. 2022Discriminator-Guided Model-Based Offline Imitation Learningcited by 17
  10. 2022Embedding Synthetic Off-Policy Experience for Autonomous Driving via Zero-Shot Curriculacited by 17
  11. 2023Identifiability and Generalizability in Constrained Inverse Reinforcement Learningcited by 16
  12. 2024Diffusing States and Matching Scores: A New Framework for Imitation Learningcited by 16
  13. 2025A Smooth Sea Never Made a Skilled SAILOR: Robust Imitation via Learning to Searchcited by 16
  14. 2022Rethinking ValueDice: Does It Really Improve Performance?cited by 15
  15. 2023Learning Shared Safety Constraints from Multi-task Demonstrationscited by 15
  16. 2023What Matters to You? Towards Visual Representation Alignment for Robot Learningcited by 14
  17. 2022Understanding Adversarial Imitation Learning in Small Sample Regime: A Stage-coupled Analysiscited by 13
  18. 2023The Virtues of Laziness in Model-based RL: A Unified Objective and Algorithmscited by 12
  19. 2023Reward-Consistent Dynamics Models are Strongly Generalizable for Offline Reinforcement Learningcited by 12
  20. 2023Massively Scalable Inverse Reinforcement Learning in Google Mapscited by 11
  21. 2023CLUE: Calibrated Latent Guidance for Offline Reinforcement Learningcited by 11
  22. 2024How to Leverage Diverse Demonstrations in Offline Imitation Learningcited by 11
  23. 2024Multi-Agent Imitation Learning: Value is Easy, Regret is Hardcited by 11
  24. 2023On Imitation in Mean-field Gamescited by 10
  25. 2024A Dual Approach to Imitation Learning from Observations with Offline Datasetscited by 10
  26. 2024Non-Adversarial Inverse Reinforcement Learning via Successor Feature Matchingcited by 10
  27. 2021A Pragmatic Look at Deep Imitation Learningcited by 9
  28. 2023Offline Imitation Learning with Variational Counterfactual Reasoningcited by 7
  29. 2024EvIL: Evolution Strategies for Generalisable Imitation Learningcited by 7
  30. 2025Imitation Learning from a Single Temporally Misaligned Videocited by 7
  31. 2025Learning to Answer from Correct Demonstrationscited by 6
  32. 2022SHAIL: Safety-Aware Hierarchical Adversarial Imitation Learning for Autonomous Driving in Urban Environmentscited by 5
  33. 2023A Game-Theoretic Framework for Joint Forecasting and Planningcited by 5
  34. 2024Adversarial Imitation Learning via Boostingcited by 5
  35. 2025IL-SOAR : Imitation Learning with Soft Optimistic Actor cRiticcited by 5
  36. 2025Inverse Q-Learning Done Right: Offline Imitation Learning in Qπ-Realizable MDPscited by 5
  37. 2023ManiCast: Collaborative Manipulation with Cost-Aware Human Forecastingcited by 4
  38. 2025Efficient Imitation under Misspecificationcited by 4

50 most cited of 80 All 80 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Imitation Learning in the Deep Learning Era: A Novel Taxonomy and Recent Advances
  2. 2021Imitation Learning by Reinforcement Learning
  3. 2018An Algorithmic Perspective on Imitation Learning
  4. 2026Matching Multiple Experts: On the Exploitability of Multi-Agent Imitation Learning
  5. 2020State-only Imitation with Transition Dynamics Mismatch
  6. 2023Provably Efficient Adversarial Imitation Learning with Unknown Transitions
  7. 2022Imitation Learning by Estimating Expertise of Demonstrators
  8. 2018Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning