# Imitation Learning by State-Only Distribution Matching

arXiv: https://arxiv.org/abs/2202.04332 · HTML: https://mlsanity.com/paper/2202.04332/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 29 of its 49 references that are in the map.
- First posted on arXiv 9 Feb 2022.

## Placed in

- Offline RL and imitation learning from data / [Imitation learning methods](https://mlsanity.com/offlinerl/imitation-learning-methods/)

## Builds on (most cited first, 29 of 29)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Density estimation using Real NVP](https://mlsanity.com/paper/1605.08803/) (2016)
- [Spectral Normalization for Generative Adversarial Networks](https://mlsanity.com/paper/1802.05957/) (2018)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Glow: Generative Flow with Invertible 1x1 Convolutions](https://mlsanity.com/paper/1807.03039/) (2018)
- [Normalizing Flows for Probabilistic Modeling and Inference](https://mlsanity.com/paper/1912.02762/) (2019)
- [Soft Actor-Critic Algorithms and Applications](https://mlsanity.com/paper/1812.05905/) (2018)
- [Behavioral Cloning from Observation](https://mlsanity.com/paper/1805.01954/) (2018)
- [Learning Robust Rewards with Adversarial Inverse Reinforcement Learning](https://mlsanity.com/paper/1710.11248/) (2017)
- [An Algorithmic Perspective on Imitation Learning](https://mlsanity.com/paper/1811.06711/) (2018)
- [Analyzing Inverse Problems with Invertible Neural Networks](https://mlsanity.com/paper/1808.04730/) (2018)
- [IQ-Learn: Inverse soft-Q Learning for Imitation](https://mlsanity.com/paper/2106.12142/) (2021)
- [A Divergence Minimization Perspective on Imitation Learning Methods](https://mlsanity.com/paper/1911.02256/) (2019)
- [Generative Adversarial Imitation from Observation](https://mlsanity.com/paper/1807.06158/) (2018)
- [Imitation Learning via Off-Policy Distribution Matching](https://mlsanity.com/paper/1912.05032/) (2019)
- [Imitating Driver Behavior with Generative Adversarial Networks](https://mlsanity.com/paper/1701.06699/) (2017)
- [Recent Advances in Imitation Learning from Observation](https://mlsanity.com/paper/1905.13566/) (2019)
- [Provably Efficient Imitation Learning from Observation Alone](https://mlsanity.com/paper/1905.10948/) (2019)
- [Off-Policy Imitation Learning from Observations](https://mlsanity.com/paper/2102.13185/) (2021)
- [f-IRL: Inverse Reinforcement Learning via State Marginal Matching](https://mlsanity.com/paper/2011.04709/) (2020)
- [Imitation Learning from Observations by Minimizing Inverse Dynamics Disagreement](https://mlsanity.com/paper/1910.04417/) (2019)
- [Random Expert Distillation: Imitation Learning via Expert Policy Support Estimation](https://mlsanity.com/paper/1905.06750/) (2019)
- [Energy-Based Imitation Learning](https://mlsanity.com/paper/2004.09395/) (2020)
- [Generative predecessor models for sample-efficient imitation learning](https://mlsanity.com/paper/1904.01139/) (2019)
- [Imitation by Predicting Observations](https://mlsanity.com/paper/2107.03851/) (2021)
- [Imitation with Neural Density Models](https://mlsanity.com/paper/2010.09808/) (2020)
- [On Duality Gap as a Measure for Monitoring GAN Training](https://mlsanity.com/paper/2012.06723/) (2020)
- [Deterministic and Discriminative Imitation (D2-Imitation): Revisiting Adversarial Imitation for Sample Efficiency](https://mlsanity.com/paper/2112.06054/) (2021)

## Built on by (most cited first, 6 of 6)

- [Deep Generative Models for Offline Policy Learning: Tutorial, Survey, and Perspectives on Future Directions](https://mlsanity.com/paper/2402.13777/) (2024)
- [CEIL: Generalized Contextual Imitation Learning](https://mlsanity.com/paper/2306.14534/) (2023)
- [Context-Former: Stitching via Latent Conditioned Sequence Modeling](https://mlsanity.com/paper/2401.16452/) (2024)
- [Action-Constrained Imitation Learning](https://mlsanity.com/paper/2508.14379/) (2025)
- [Curriculum Imitation Learning of Distributed Multi-Robot Policies](https://mlsanity.com/paper/2509.25097/) (2025)
- [Imitating from auxiliary imperfect demonstrations via Adversarial Density Weighted Regression](https://mlsanity.com/paper/2405.20351/) (2024)

## Abstract (arXiv)

Imitation Learning from observation describes policy learning in a similar way to human learning. An agent's policy is trained by observing an expert performing a task. While many state-only imitation learning approaches are based on adversarial imitation learning, one main drawback is that adversarial training is often unstable and lacks a reliable convergence estimator. If the true environment reward is unknown and cannot be used to select the best-performing model, this can result in bad real-world policy performance. We propose a non-adversarial learning-from-observations approach, together with an interpretable convergence and performance metric. Our training objective minimizes the Kulback-Leibler divergence (KLD) between the policy and expert state transition trajectories which can be optimized in a non-adversarial fashion. Such methods demonstrate improved robustness when learned density models guide the optimization. We further improve the sample efficiency by rewriting the KLD minimization as the Soft Actor Critic objective based on a modified reward using additional density models that estimate the environment's forward and backward dynamics. Finally, we evaluate the effectiveness of our approach on well-known continuous control environments and show state-of-the-art performance while having a reliable performance estimator compared to several recent learning-from-observation methods.