Data to 14 Oct 2026

f-IRL: Inverse Reinforcement Learning via State Marginal Matching

arXiv 2011.04709 ↗First posted 9 Nov 2020cs.LG cs.ROcited by 67

On the map: Offline RL and imitation learning from data → Inverse reinforcement learning

Abstract (arXiv)

Imitation learning is well-suited for robotic tasks where it is difficult to directly program the behavior or specify a cost for optimal control. In this work, we propose a method for learning the reward function (and the corresponding policy) to match the expert state density. Our main result is the analytic gradient of any f-divergence between the agent and expert state distribution w.r.t. reward parameters. Based on the derived gradient, we present an algorithm, f-IRL, that recovers a stationary reward function from the expert density by gradient descent. We show that f-IRL can learn behaviors from a hand-designed target state density or implicitly through expert observations. Our method outperforms adversarial imitation learning methods in terms of sample efficiency and the required number of expert trajectories on IRL benchmarks. Moreover, we show that the recovered reward function can be used to quickly solve downstream tasks, and empirically demonstrate its utility on hard-to-explore tasks and for behavior transfer across changes in dynamics.

Key facts. Cited by 67 papers in the map (data through 14 Oct 2026). Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 24 of its 46 references that are in the map. First posted on arXiv 9 Nov 2020.

Builds on

24 of 46 references.

Closest ancestors · 1

Same direction, newest first

2018Adversarial Imitation via Variational Inverse Reinforcement Learning17 Sep

Other references · 23

2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
2015Trust Region Policy Optimizationcited by 2,580
2016OpenAI Gymcited by 1,736
2016Generative Adversarial Imitation Learningcited by 1,597
2017Deep Reinforcement Learning that Matterscited by 754
2017Wasserstein GANcited by 639
2016f-GAN: Training Generative Neural Samplers using Variational Divergence Minimizationcited by 506
2019Learning agile and dynamic motor skills for legged robotscited by 504
2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 490
2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 482
2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 406
2016Cooperative Inverse Reinforcement Learningcited by 300

11 more

2017MMD GAN: Towards Deeper Understanding of Moment Matching Networkcited by 254
2018Safe Exploration in Continuous Action Spacescited by 247
2019A Divergence Minimization Perspective on Imitation Learning Methodscited by 190
2019Efficient Exploration via State Marginal Matchingcited by 188
2015Maximum Entropy Deep Inverse Reinforcement Learningcited by 176
2016A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Modelscited by 168
2019Imitation Learning as f-Divergence Minimizationcited by 123
2019State Alignment-based Imitation Learningcited by 73
2019Random Expert Distillation: Imitation Learning via Expert Policy Support Estimationcited by 42
2020Energy-Based Imitation Learningcited by 40
2016Analysis of k-Nearest Neighbor Distances with Application to Entropy Estimationcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 4; 2022: 14; 2023: 19; 2024: 19; 2025: 9; 2026: 2.

67 papers.

Sort:

By area:

2024RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedbackcited by 156
2023Dual RL: Unification and New Methods for Reinforcement and Imitation Learningcited by 48
2022Maximum-Likelihood Inverse Reinforcement Learning with Finite-Time Guaranteescited by 42
2021Adversarial Intrinsic Motivation for Reinforcement Learningcited by 40
2023State Regularized Policy Optimization on Data with Dynamics Shiftcited by 23
2023Coherent Soft Imitation Learningcited by 22
2024Real-World Offline Reinforcement Learning from Vision Language Model Feedbackcited by 22
2023On The Fragility of Learned Reward Functionscited by 21
2023SMORE: Score Models for Offline Goal-Conditioned Reinforcement Learningcited by 21
2024Deep Generative Models for Offline Policy Learning: Tutorial, Survey, and Perspectives on Future Directionscited by 20
2022A Ranking Game for Imitation Learningcited by 18
2024Diffusing States and Matching Scores: A New Framework for Imitation Learningcited by 16

38 more

2021OPIRL: Sample Efficient Off-Policy Inverse Reinforcement Learning via Distribution Matchingcited by 15
2024Imitation Learning in Discounted Linear MDPs without exploration assumptionscited by 15
2022Fast Lifelong Adaptive Inverse Reinforcement Learning from Demonstrationscited by 14
2023f-Policy Gradients: A General Framework for Goal Conditioned RL using f-Divergencescited by 14
2023CEIL: Generalized Contextual Imitation Learningcited by 13
2022IL-flOw: Imitation Learning from Observation using Normalizing Flowscited by 12
2022Structural Estimation of Markov Decision Processes in High-Dimensional State Space with Finite-Time Guaranteescited by 12
2023Reward-Consistent Dynamics Models are Strongly Generalizable for Offline Reinforcement Learningcited by 12
2022Receding Horizon Inverse Reinforcement Learningcited by 10
2023Learning to Reach Goals via Diffusioncited by 10
2023Inverse Decision Modeling: Learning Interpretable Representations of Behaviorcited by 10
2024A Dual Approach to Imitation Learning from Observations with Offline Datasetscited by 10
2023Time-series Generation by Contrastive Imitationcited by 9
2025CREStE: Scalable Mapless Navigation with Internet Scale Priors and Counterfactual Guidancecited by 9
2023Inverse Reinforcement Learning with the Average Reward Criterioncited by 8
2025Posterior Behavioral Cloning: Pretraining BC Policies for Efficient RL Finetuningcited by 8
2023BC-IRL: Learning Generalizable Reward Functions from Demonstrationscited by 7
2023Imitation Learning from Observation through Optimal Transportcited by 7
2024RLZero: Direct Policy Inference from Language Without In-Domain Supervisioncited by 7
2025Inverse Reinforcement Learning Meets Large Language Model Post-Training: Basics, Advances, and Opportunitiescited by 7
2022Imitation Learning by State-Only Distribution Matchingcited by 6
2022Robust Imitation via Mirror Descent Inverse Reinforcement Learningcited by 6
2024OLLIE: Imitation Learning from Offline Pretraining to Online Finetuningcited by 6
2022Transferable Reward Learning by Dynamics-Agnostic Discriminator Ensemblecited by 5
2023Infer and Adapt: Bipedal Locomotion Reward Learning from Demonstrations via Inverse Reinforcement Learningcited by 5
2023Learning Reward for Physical Skills using Large Language Modelcited by 5
2024Robot Air Hockey: A Manipulation Testbed for Robot Learning with Reinforcement Learningcited by 5
2025IL-SOAR : Imitation Learning with Soft Optimistic Actor cRiticcited by 5
2021Back to Reality for Imitation Learningcited by 3
2022ARC - Actor Residual Critic for Adversarial Imitation Learningcited by 3
2022Co-Imitation: Learning Design and Behaviour by Imitationcited by 3
2024Language-Model-Assisted Bi-Level Programming for Reward Learning from Internet Videoscited by 3
2024Rethinking Inverse Reinforcement Learning: from Data Alignment to Task Alignmentcited by 3
2024Towards Sample-Efficiency and Generalization of Transfer and Inverse Reinforcement Learning: A Comprehensive Literature Reviewcited by 3
2024Expert Proximity as Surrogate Rewards for Single Demonstration Imitation Learningcited by 2
2024Learning Causally Invariant Reward Functions from Diverse Demonstrationscited by 2
2022Modeling Human Behavior Part I -- Learning and Belief Approachescited by 1
2022Accelerating Self-Imitation Learning from Demonstrations via Policy Constraints and Q-Ensemblecited by 1

50 most cited of 67 All 67 (JSON)

Similar, not cited

Close in wording, no citation either way