A Novel Variational Lower Bound for Inverse Reinforcement Learning
On the map: Offline RL and imitation learning from data → Inverse reinforcement learning
Abstract (arXiv)
Inverse reinforcement learning (IRL) seeks to learn the reward function from expert trajectories, to understand the task for imitation or collaboration thereby removing the need for manual reward engineering. However, IRL in the context of large, high-dimensional problems with unknown dynamics has been particularly challenging. In this paper, we present a new Variational Lower Bound for IRL (VLB-IRL), which is derived under the framework of a probabilistic graphical model with an optimality node. Our method simultaneously learns the reward function and policy under the learned reward function by maximizing the lower bound, which is equivalent to minimizing the reverse Kullback-Leibler divergence between an approximated distribution of optimality given the reward function and the true distribution of optimality given trajectories. This leads to a new IRL method that learns a valid reward function such that the policy under the learned reward achieves expert-level performance on several known domains. Importantly, the method outperforms the existing state-of-the-art IRL algorithms on these domains by demonstrating better reward from the learned policy.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 19 of its 36 references that are in the map. First posted on arXiv 7 Nov 2023.
Builds on
19 of 36 references.
Closest ancestors · 7
Same direction, newest first
- 2023Identifiability and Generalizability in Constrained Inverse Reinforcement Learning1 Jun
- 2021Scalable Bayesian Inverse Reinforcement Learning12 Feb
- 2020f-IRL: Inverse Reinforcement Learning via State Marginal Matching9 Nov
- 2020Robust Inverse Reinforcement Learning under Transition Dynamics Mismatch2 Jul
- 2018Adversarial Imitation via Variational Inverse Reinforcement Learning17 Sep
- 2018A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progress18 Jun
- 2018Variational Inverse Control with Events: A General Framework for Data-Driven Reward Definition29 May
Other references · 12
- 2017Proximal Policy Optimization Algorithmscited by 14,441
- 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
- 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
- 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
- 2016Generative Adversarial Imitation Learningcited by 1,576
- 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 466
- 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 400
- 2021IQ-Learn: Inverse soft-Q Learning for Imitationcited by 191
- 2019A Divergence Minimization Perspective on Imitation Learning Methodscited by 188
- 2019Assistive Gym: A Physics Simulation Framework for Assistive Roboticscited by 44
- 2020Energy-Based Imitation Learningcited by 39
- 2020When Will Generative Adversarial Imitation Learning Algorithms Attain Global Convergencecited by 12
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Inverse Reinforcement Learning via Convex Optimization
- 2020Efficient Exploration of Reward Functions in Inverse Reinforcement Learning via Bayesian Optimization
- 2026Inverse Reinforcement Learning without an Optimal Demonstrator: A Feasible Reward Set Approach
- 2025Reward Compatibility: A Framework for Inverse RL
- 2023Towards Theoretical Understanding of Inverse Reinforcement Learning
- 2020Inverse Reinforcement Learning via Matching of Optimality Profiles
- 2023Is Inverse Reinforcement Learning Harder than Standard Reinforcement Learning? A Theoretical Perspective
- 2026Fast Rates for Inverse Reinforcement Learning