# Of Moments and Matching: A Game-Theoretic Framework for Closing the Imitation Gap

arXiv: https://arxiv.org/abs/2103.03236 · HTML: https://mlsanity.com/paper/2103.03236/

## Key facts

- Cited by 81 papers in the map (data through 14 Oct 2026).
- Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 21 of its 49 references that are in the map.
- First posted on arXiv 4 Mar 2021.

## Placed in

- Offline RL and imitation learning from data / [Imitation learning methods](https://mlsanity.com/offlinerl/imitation-learning-methods/)

## Builds on (most cited first, 21 of 21)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [Improved Training of Wasserstein GANs](https://mlsanity.com/paper/1704.00028/) (2017)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [End to End Learning for Self-Driving Cars](https://mlsanity.com/paper/1604.07316/) (2016)
- [Learning Robust Rewards with Adversarial Inverse Reinforcement Learning](https://mlsanity.com/paper/1710.11248/) (2017)
- [Formal Limitations on the Measurement of Mutual Information](https://mlsanity.com/paper/1811.04251/) (2018)
- [Imitation Learning via Off-Policy Distribution Matching](https://mlsanity.com/paper/1912.05032/) (2019)
- [Imitation Learning as f-Divergence Minimization](https://mlsanity.com/paper/1905.12888/) (2019)
- [Primal Wasserstein Imitation Learning](https://mlsanity.com/paper/2006.04678/) (2020)
- [Reinforcement and Imitation Learning via Interactive No-Regret Learning](https://mlsanity.com/paper/1406.5979/) (2014)
- [Multi-Agent Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1807.09936/) (2018)
- [Toward the Fundamental Limits of Imitation Learning](https://mlsanity.com/paper/2009.05990/) (2020)
- [Provably Efficient Imitation Learning from Observation Alone](https://mlsanity.com/paper/1905.10948/) (2019)
- [Feedback in Imitation Learning: The Three Regimes of Covariate Shift](https://mlsanity.com/paper/2102.02872/) (2021)
- [Strictly Batch Imitation Learning by Energy-based Distribution Matching](https://mlsanity.com/paper/2006.14154/) (2020)
- [Dual Policy Iteration](https://mlsanity.com/paper/1805.10755/) (2018)
- [Adversarial Soft Advantage Fitting: Imitation Learning without Policy Optimization](https://mlsanity.com/paper/2006.13258/) (2020)
- [Non-Adversarial Imitation Learning and its Connections to Adversarial Methods](https://mlsanity.com/paper/2008.03525/) (2020)
- [Scaled Autonomy: Enabling Human Operators to Control Robot Fleets](https://mlsanity.com/paper/1910.02910/) (2019)

## Built on by (most cited first, 30 of 81)

- [Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization](https://mlsanity.com/paper/2210.01241/) (2022)
- [A Minimaximalist Approach to Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2401.04056/) (2024)
- [Adversarially Trained Actor Critic for Offline Reinforcement Learning](https://mlsanity.com/paper/2202.02446/) (2022)
- [REBEL: Reinforcement Learning via Regressing Relative Rewards](https://mlsanity.com/paper/2404.16767/) (2024)
- [All Roads Lead to Likelihood: The Value of Reinforcement Learning in Fine-Tuning](https://mlsanity.com/paper/2503.01067/) (2025)
- [Zero-Shot Whole-Body Humanoid Control via Behavioral Foundation Models](https://mlsanity.com/paper/2504.11054/) (2025)
- [From Foresight to Forethought: VLM-In-the-Loop Policy Steering via Latent Alignment](https://mlsanity.com/paper/2502.01828/) (2025)
- [Dual RL: Unification and New Methods for Reinforcement and Imitation Learning](https://mlsanity.com/paper/2302.08560/) (2023)
- [Hierarchical Model-Based Imitation Learning for Planning in Autonomous Driving](https://mlsanity.com/paper/2210.09539/) (2022)
- [Inverse Reinforcement Learning without Reinforcement Learning](https://mlsanity.com/paper/2303.14623/) (2023)
- [Sequence Model Imitation Learning with Unobserved Contexts](https://mlsanity.com/paper/2208.02225/) (2022)
- [Hybrid Inverse Reinforcement Learning](https://mlsanity.com/paper/2402.08848/) (2024)
- [LS-IQ: Implicit Reward Regularization for Inverse Reinforcement Learning](https://mlsanity.com/paper/2303.00599/) (2023)
- [CCIL: Continuity-based Data Augmentation for Corrective Imitation Learning](https://mlsanity.com/paper/2310.12972/) (2023)
- [Causal Imitation Learning under Temporally Correlated Noise](https://mlsanity.com/paper/2202.01312/) (2022)
- [Multi-Turn Code Generation Through Single-Step Rewards](https://mlsanity.com/paper/2502.20380/) (2025)
- [Minimax Optimal Online Imitation Learning via Replay Estimation](https://mlsanity.com/paper/2205.15397/) (2022)
- [SMORE: Score Models for Offline Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2311.02013/) (2023)
- [CLARE: Conservative Model-Based Reward Learning for Offline Inverse Reinforcement Learning](https://mlsanity.com/paper/2302.04782/) (2023)
- [A Ranking Game for Imitation Learning](https://mlsanity.com/paper/2202.03481/) (2022)
- [Discriminator-Guided Model-Based Offline Imitation Learning](https://mlsanity.com/paper/2207.00244/) (2022)
- [Embedding Synthetic Off-Policy Experience for Autonomous Driving via Zero-Shot Curricula](https://mlsanity.com/paper/2212.01375/) (2022)
- [A Smooth Sea Never Made a Skilled SAILOR: Robust Imitation via Learning to Search](https://mlsanity.com/paper/2506.05294/) (2025)
- [Identifiability and Generalizability in Constrained Inverse Reinforcement Learning](https://mlsanity.com/paper/2306.00629/) (2023)
- [Diffusing States and Matching Scores: A New Framework for Imitation Learning](https://mlsanity.com/paper/2410.13855/) (2024)
- [Rethinking ValueDice: Does It Really Improve Performance?](https://mlsanity.com/paper/2202.02468/) (2022)
- [Learning Shared Safety Constraints from Multi-task Demonstrations](https://mlsanity.com/paper/2309.00711/) (2023)
- [What Matters to You? Towards Visual Representation Alignment for Robot Learning](https://mlsanity.com/paper/2310.07932/) (2023)
- [Understanding Adversarial Imitation Learning in Small Sample Regime: A Stage-coupled Analysis](https://mlsanity.com/paper/2208.01899/) (2022)
- [The Virtues of Laziness in Model-based RL: A Unified Objective and Algorithms](https://mlsanity.com/paper/2303.00694/) (2023)

## Abstract (arXiv)

We provide a unifying view of a large family of previous imitation learning algorithms through the lens of moment matching. At its core, our classification scheme is based on whether the learner attempts to match (1) reward or (2) action-value moments of the expert's behavior, with each option leading to differing algorithmic approaches. By considering adversarially chosen divergences between learner and expert behavior, we are able to derive bounds on policy performance that apply for all algorithms in each of these classes, the first to our knowledge. We also introduce the notion of moment recoverability, implicit in many previous analyses of imitation learning, which allows us to cleanly delineate how well each algorithmic family is able to mitigate compounding errors. We derive three novel algorithm templates (AdVIL, AdRIL, and DAeQuIL) with strong guarantees, simple implementation, and competitive empirical performance.