# Diffusing States and Matching Scores: A New Framework for Imitation Learning

arXiv: https://arxiv.org/abs/2410.13855 · HTML: https://mlsanity.com/paper/2410.13855/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models.
- Builds on 65 of its 83 references that are in the map.
- First posted on arXiv 17 Oct 2024.

## Placed in

- Diffusion models / [Diffusion and flow policies for robotics](https://mlsanity.com/diffusion/diffusion-and-flow-policies-for-robotics/)
- Offline RL and imitation learning from data / [Imitation learning methods](https://mlsanity.com/offlinerl/imitation-learning-methods/)

## Builds on (most cited first, 30 of 65)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [Spectral Normalization for Generative Adversarial Networks](https://mlsanity.com/paper/1802.05957/) (2018)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Octo: An Open-Source Generalist Robot Policy](https://mlsanity.com/paper/2405.12213/) (2024)
- [DiffWave: A Versatile Diffusion Model for Audio Synthesis](https://mlsanity.com/paper/2009.09761/) (2020)
- [Mastering Diverse Domains through World Models](https://mlsanity.com/paper/2301.04104/) (2023)
- [Planning with Diffusion for Flexible Behavior Synthesis](https://mlsanity.com/paper/2205.09991/) (2022)
- [DeepMind Control Suite](https://mlsanity.com/paper/1801.00690/) (2018)
- [Is Conditional Generative Modeling all you need for Decision-Making?](https://mlsanity.com/paper/2211.15657/) (2022)
- [Behavioral Cloning from Observation](https://mlsanity.com/paper/1805.01954/) (2018)
- [Learning Robust Rewards with Adversarial Inverse Reinforcement Learning](https://mlsanity.com/paper/1710.11248/) (2017)
- [Towards Principled Methods for Training Generative Adversarial Networks](https://mlsanity.com/paper/1701.04862/) (2017)
- [An Algorithmic Perspective on Imitation Learning](https://mlsanity.com/paper/1811.06711/) (2018)
- [Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions](https://mlsanity.com/paper/2209.11215/) (2022)
- [Information-Theoretic Considerations in Batch Reinforcement Learning](https://mlsanity.com/paper/1905.00360/) (2019)
- [Imitating Human Behaviour with Diffusion Models](https://mlsanity.com/paper/2301.10677/) (2023)
- [Improved Analysis of Score-based Generative Modeling: User-Friendly Bounds under Minimal Smoothness Assumptions](https://mlsanity.com/paper/2211.01916/) (2022)
- [Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning](https://mlsanity.com/paper/1809.02925/) (2018)
- [IQ-Learn: Inverse soft-Q Learning for Imitation](https://mlsanity.com/paper/2106.12142/) (2021)
- [A Divergence Minimization Perspective on Imitation Learning Methods](https://mlsanity.com/paper/1911.02256/) (2019)
- [Convergence for score-based generative modeling with polynomial complexity](https://mlsanity.com/paper/2206.06227/) (2022)
- [A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Models](https://mlsanity.com/paper/1611.03852/) (2016)
- [Diffusion Models are Minimax Optimal Distribution Estimators](https://mlsanity.com/paper/2303.01861/) (2023)

## Built on by (most cited first, 16 of 16)

- [Scaling Offline RL via Efficient and Expressive Shortcut Models](https://mlsanity.com/paper/2505.22866/) (2025)
- [The Pitfalls of Imitation Learning when Actions are Continuous](https://mlsanity.com/paper/2503.09722/) (2025)
- [Temporal Difference Flows](https://mlsanity.com/paper/2503.09817/) (2025)
- [A Smooth Sea Never Made a Skilled SAILOR: Robust Imitation via Learning to Search](https://mlsanity.com/paper/2506.05294/) (2025)
- [SMP: Reusable Score-Matching Motion Priors for Physics-Based Character Control](https://mlsanity.com/paper/2512.03028/) (2025)
- [Optimistically Optimistic Exploration for Provably Efficient Infinite-Horizon Reinforcement and Imitation Learning](https://mlsanity.com/paper/2502.13900/) (2025)
- [Expressive Value Learning for Scalable Offline Reinforcement Learning](https://mlsanity.com/paper/2510.08218/) (2025)
- [IL-SOAR : Imitation Learning with Soft Optimistic Actor cRitic](https://mlsanity.com/paper/2502.19859/) (2025)
- [Your Learned Constraint is Secretly a Backward Reachable Tube](https://mlsanity.com/paper/2501.15618/) (2025)
- [Fourier Features Let Agents Learn High Precision Policies with Imitation Learning](https://mlsanity.com/paper/2606.12334/) (2026)
- [RIZE: Adaptive Regularization for Imitation Learning](https://mlsanity.com/paper/2502.20089/) (2025)
- [Distributional Inverse Reinforcement Learning](https://mlsanity.com/paper/2510.03013/) (2025)
- [SD2AIL: Adversarial Imitation Learning from Synthetic Demonstrations via Diffusion Models](https://mlsanity.com/paper/2512.18583/) (2025)
- [Matching Multiple Experts: On the Exploitability of Multi-Agent Imitation Learning](https://mlsanity.com/paper/2602.21020/) (2026)
- [DSD: Learning Diverse and Reusable Motor Skills via Diffusion Skill Discovery](https://mlsanity.com/paper/2609.17682/) (2026)
- [Loop-Free Inverse Reinforcement Learning via Sequential Value Recovery with Q-Score Matching](https://mlsanity.com/paper/2609.38955/) (2026)

## Abstract (arXiv)

Adversarial Imitation Learning is traditionally framed as a two-player zero-sum game between a learner and an adversarially chosen cost function, and can therefore be thought of as the sequential generalization of a Generative Adversarial Network (GAN). However, in recent years, diffusion models have emerged as a non-adversarial alternative to GANs that merely require training a score function via regression, yet produce generations of higher quality. In response, we investigate how to lift insights from diffusion modeling to the sequential setting. We propose diffusing states and performing score-matching along diffused states to measure the discrepancy between the expert's and learner's states. Thus, our approach only requires training score functions to predict noises via standard regression, making it significantly easier and more stable to train than adversarial methods. Theoretically, we prove first- and second-order instance-dependent bounds with linear scaling in the horizon, proving that our approach avoids the compounding errors that stymie offline approaches to imitation learning. Empirically, we show our approach outperforms both GAN-style imitation learning baselines and discriminator-free imitation learning baselines across various continuous control problems, including complex tasks like controlling humanoids to walk, sit, crawl, and navigate through obstacles.