# BC-IRL: Learning Generalizable Reward Functions from Demonstrations

arXiv: https://arxiv.org/abs/2303.16194 · HTML: https://mlsanity.com/paper/2303.16194/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 31 of its 45 references that are in the map.
- First posted on arXiv 28 Mar 2023.

## Placed in

- Offline RL and imitation learning from data / [Inverse reinforcement learning](https://mlsanity.com/offlinerl/inverse-reinforcement-learning/)

## Builds on (most cited first, 30 of 31)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation](https://mlsanity.com/paper/1806.10293/) (2018)
- [Habitat 2.0: Training Home Assistants to Rearrange their Habitat](https://mlsanity.com/paper/2106.14405/) (2021)
- [RMA: Rapid Motor Adaptation for Legged Robots](https://mlsanity.com/paper/2107.04034/) (2021)
- [Learning Robust Rewards with Adversarial Inverse Reinforcement Learning](https://mlsanity.com/paper/1710.11248/) (2017)
- [An Algorithmic Perspective on Imitation Learning](https://mlsanity.com/paper/1811.06711/) (2018)
- [Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization](https://mlsanity.com/paper/1603.00448/) (2016)
- [DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Frames](https://mlsanity.com/paper/1911.00357/) (2019)
- [A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Models](https://mlsanity.com/paper/1611.03852/) (2016)
- [Primal Wasserstein Imitation Learning](https://mlsanity.com/paper/2006.04678/) (2020)
- [CausalWorld: A Robotic Manipulation Benchmark for Causal Structure and Transfer Learning](https://mlsanity.com/paper/2010.04296/) (2020)
- [Generalized Inner Loop Meta-Learning](https://mlsanity.com/paper/1910.01727/) (2019)
- [What Matters for Adversarial Imitation Learning?](https://mlsanity.com/paper/2106.00672/) (2021)
- [f-IRL: Inverse Reinforcement Learning via State Marginal Matching](https://mlsanity.com/paper/2011.04709/) (2020)
- [Meta-Learning via Learned Loss](https://mlsanity.com/paper/1906.05374/) (2019)
- [Model-Based Inverse Reinforcement Learning from Visual Demonstrations](https://mlsanity.com/paper/2010.09034/) (2020)
- [Meta-Inverse Reinforcement Learning with Probabilistic Context Variables](https://mlsanity.com/paper/1909.09314/) (2019)
- [Policies Modulating Trajectory Generators](https://mlsanity.com/paper/1910.02812/) (2019)
- [Task-Relevant Adversarial Imitation Learning](https://mlsanity.com/paper/1910.01077/) (2019)
- [Positive-Unlabeled Reward Learning](https://mlsanity.com/paper/1911.00459/) (2019)
- [Learning a Prior over Intent via Meta-Inverse Reinforcement Learning](https://mlsanity.com/paper/1805.12573/) (2018)
- [Adversarial Soft Advantage Fitting: Imitation Learning without Policy Optimization](https://mlsanity.com/paper/2006.13258/) (2020)
- [Multi-task Maximum Entropy Inverse Reinforcement Learning](https://mlsanity.com/paper/1805.08882/) (2018)
- [Imitation by Predicting Observations](https://mlsanity.com/paper/2107.03851/) (2021)
- [Meta-Adversarial Inverse Reinforcement Learning for Decision-making Tasks](https://mlsanity.com/paper/2103.12694/) (2021)

## Built on by (most cited first, 7 of 7)

- [Coherent Soft Imitation Learning](https://mlsanity.com/paper/2305.16498/) (2023)
- [Rating-based Reinforcement Learning](https://mlsanity.com/paper/2307.16348/) (2023)
- [Enhancing Rating-Based Reinforcement Learning to Effectively Leverage Feedback from Large Vision-Language Models](https://mlsanity.com/paper/2506.12822/) (2025)
- [FoMo Rewards: Can we cast foundation models as reward functions?](https://mlsanity.com/paper/2312.03881/) (2023)
- [Trust Region Reward Optimization and Proximal Inverse Reward Optimization Algorithm](https://mlsanity.com/paper/2509.23135/) (2025)
- [Rethinking Adversarial Inverse Reinforcement Learning: Policy Imitation, Transferable Reward Recovery and Algebraic Equilibrium Proof](https://mlsanity.com/paper/2403.14593/) (2024)
- [On Reward Transferability in Adversarial Inverse Reinforcement Learning: Insights from Random Matrix Theory](https://mlsanity.com/paper/2410.07643/) (2024)

## Abstract (arXiv)

How well do reward functions learned with inverse reinforcement learning (IRL) generalize? We illustrate that state-of-the-art IRL algorithms, which maximize a maximum-entropy objective, learn rewards that overfit to the demonstrations. Such rewards struggle to provide meaningful rewards for states not covered by the demonstrations, a major detriment when using the reward to learn policies in new situations. We introduce BC-IRL a new inverse reinforcement learning method that learns reward functions that generalize better when compared to maximum-entropy IRL approaches. In contrast to the MaxEnt framework, which learns to maximize rewards around demonstrations, BC-IRL updates reward parameters such that the policy trained with the new reward matches the expert demonstrations better. We show that BC-IRL learns rewards that generalize better on an illustrative simple task and two continuous robotic control tasks, achieving over twice the success rate of baselines in challenging generalization settings.