mlsanityData to 7 Oct 2026

A Coupled Flow Approach to Imitation Learning

arXiv 2305.00303 ↗First posted 29 Apr 2023cs.LG stat.MLcited by 12

On the map: Offline RL and imitation learning from data → Diffusion and flow policies for offline RL

Abstract (arXiv)

In reinforcement learning and imitation learning, an object of central importance is the state distribution induced by the policy. It plays a crucial role in the policy gradient theorem, and references to it--along with the related state-action distribution--can be found all across the literature. Despite its importance, the state distribution is mostly discussed indirectly and theoretically, rather than being modeled explicitly. The reason being an absence of appropriate density estimation tools. In this work, we investigate applications of a normalizing flow-based model for the aforementioned distributions. In particular, we use a pair of flows coupled through the optimality point of the Donsker-Varadhan representation of the Kullback-Leibler (KL) divergence, for distribution matching based imitation learning. Our algorithm, Coupled Flow Imitation Learning (CFIL), achieves state-of-the-art performance on benchmark tasks with a single expert trajectory and extends naturally to a variety of other settings, including the subsampled and state-only regimes.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Placed in Diffusion and flow policies for offline RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 44 of its 57 references that are in the map. First posted on arXiv 29 Apr 2023.

Builds on

44 of 57 references.

References · 44

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2022Generative Adversarial Networkscited by 5,603
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  6. 2016Density estimation using Real NVPcited by 2,029
  7. 2016Generative Adversarial Imitation Learningcited by 1,576
  8. 2018Glow: Generative Flow with Invertible 1x1 Convolutionscited by 1,539
  9. 2019Normalizing Flows for Probabilistic Modeling and Inferencecited by 1,315
  10. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  11. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  12. 2014NICE: Non-linear Independent Components Estimationcited by 1,052
32 more
  1. 2017Masked Autoregressive Flow for Density Estimationcited by 680
  2. 2019Neural Spline Flowscited by 590
  3. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
  4. 2015MADE: Masked Autoencoder for Distribution Estimationcited by 306
  5. 2019Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Designcited by 258
  6. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  7. 2018Provably Efficient Maximum Entropy Explorationcited by 240
  8. 2018Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimationcited by 226
  9. 2020Why Normalizing Flows Fail to Detect Out-of-Distribution Datacited by 220
  10. 2018Neural Autoregressive Flowscited by 207
  11. 2018Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learningcited by 203
  12. 2019A Divergence Minimization Perspective on Imitation Learning Methodscited by 188
  13. 2019Efficient Exploration via State Marginal Matchingcited by 186
  14. 2018Generative Adversarial Imitation from Observationcited by 183
  15. 2019Imitation Learning via Off-Policy Distribution Matchingcited by 182
  16. 2019Imitation Learning as f-Divergence Minimizationcited by 121
  17. 2020Primal Wasserstein Imitation Learningcited by 120
  18. 2020Variational Policy Gradient Method for Reinforcement Learning with General Utilitiescited by 108
  19. 2021Reward is enough for convex MDPscited by 82
  20. 2021Off-Policy Imitation Learning from Observationscited by 70
  21. 2019Is the Policy Gradient a Gradient?cited by 40
  22. 2022Challenging Common Assumptions in Convex Reinforcement Learningcited by 29
  23. 2021Density Constrained Reinforcement Learningcited by 25
  24. 2022Rethinking ValueDice: Does It Really Improve Performance?cited by 15
  25. 2019Generative predecessor models for sample-efficient imitation learningcited by 14
  26. 2021SoftDICE for Imitation Learning: Rethinking Off-policy Distribution Matchingcited by 14
  27. 2020Universal Value Density Estimation for Imitation Learning and Goal-Conditioned Reinforcement Learningcited by 12
  28. 2022IL-flOw: Imitation Learning from Observation using Normalizing Flowscited by 12
  29. 2020Imitation with Neural Density Modelscited by 8
  30. 2020Harnessing Distribution Ratio Estimators for Learning Agents with Quality and Diversitycited by 7
  31. 2019Off-Policy Policy Gradient Algorithms by Constraining the State Distribution Shiftcited by 4
  32. 2021DEALIO: Data-Efficient Adversarial Learning for Imitation from Observationcited by 4

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 7; 2025: 3.

12 papers.

  1. 2024AdaFlow: Imitation Learning with Variance-Adaptive Flow-Based Policiescited by 50
  2. 2024Deep Generative Models for Offline Policy Learning: Tutorial, Survey, and Perspectives on Future Directionscited by 20
  3. 2023DiffAIL: Diffusion Adversarial Imitation Learningcited by 18
  4. 2024Diffusion-Reward Adversarial Imitation Learningcited by 18
  5. 2024SEABO: A Simple Search-Based Method for Offline Imitation Learningcited by 14
  6. 2024Reward-free World Models for Online Imitation Learningcited by 7
  7. 2023A Simple Solution for Offline Imitation from Observations and Examples with Possibly Incomplete Trajectoriescited by 3
  8. 2024Expert Proximity as Surrogate Rewards for Single Demonstration Imitation Learningcited by 2
  9. 2025Action-Constrained Imitation Learningcited by 2
  10. 2024Rethinking Adversarial Inverse Reinforcement Learning: Policy Imitation, Transferable Reward Recovery and Algebraic Equilibrium Proofcited by 0
  11. 2025Coupled Distributional Random Expert Distillation for World Model Online Imitation Learningcited by 0
  12. 2025SD2AIL: Adversarial Imitation Learning from Synthetic Demonstrations via Diffusion Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Extremum Flow Matching for Offline Goal Conditioned Reinforcement Learning
  2. 2022Imitation Learning by State-Only Distribution Matching
  3. 2025Flow Matching Policy Gradients
  4. 2025Reinforcement Learning for Flow-Matching Policies
  5. 2025Streaming Flow Policy: Simplifying diffusion/flow-matching policies by treating action trajectories as flow trajectories
  6. 2023Offline Imitation Learning with Suboptimal Demonstrations via Relaxed Distribution Matching
  7. 2023Conditional Kernel Imitation Learning for Continuous State Environments
  8. 2025FM-IRL: Flow-Matching for Reward Modeling and Policy Regularization in Reinforcement Learning