# Basis for Intentions: Efficient Inverse Reinforcement Learning using Past Experience

arXiv: https://arxiv.org/abs/2208.04919 · HTML: https://mlsanity.com/paper/2208.04919/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 34 of its 56 references that are in the map.
- First posted on arXiv 9 Aug 2022.

## Placed in

- Offline RL and imitation learning from data / [Inverse reinforcement learning](https://mlsanity.com/offlinerl/inverse-reinforcement-learning/)

## Builds on (most cited first, 30 of 34)

- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Reinforcement Learning with Deep Energy-Based Policies](https://mlsanity.com/paper/1702.08165/) (2017)
- [Behavioral Cloning from Observation](https://mlsanity.com/paper/1805.01954/) (2018)
- [Learning Robust Rewards with Adversarial Inverse Reinforcement Learning](https://mlsanity.com/paper/1710.11248/) (2017)
- [Time-Contrastive Networks: Self-Supervised Learning from Video](https://mlsanity.com/paper/1704.06888/) (2017)
- [Successor Features for Transfer in Reinforcement Learning](https://mlsanity.com/paper/1606.05312/) (2016)
- [Cooperative Inverse Reinforcement Learning](https://mlsanity.com/paper/1606.03137/) (2016)
- [Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observations](https://mlsanity.com/paper/1904.06387/) (2019)
- [Machine Theory of Mind](https://mlsanity.com/paper/1802.07740/) (2018)
- [Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning](https://mlsanity.com/paper/1809.02925/) (2018)
- [IQ-Learn: Inverse soft-Q Learning for Imitation](https://mlsanity.com/paper/2106.12142/) (2021)
- [Imitation Learning via Off-Policy Distribution Matching](https://mlsanity.com/paper/1912.05032/) (2019)
- [Maximum Entropy Deep Inverse Reinforcement Learning](https://mlsanity.com/paper/1507.04888/) (2015)
- [Imitation from Observation: Learning to Imitate Behaviors from Raw Video via Context Translation](https://mlsanity.com/paper/1707.03374/) (2017)
- [Fast Task Inference with Variational Intrinsic Successor Features](https://mlsanity.com/paper/1906.05030/) (2019)
- [Count-Based Exploration with the Successor Representation](https://mlsanity.com/paper/1807.11622/) (2018)
- [Universal Successor Features Approximators](https://mlsanity.com/paper/1812.07626/) (2018)
- [Eigenoption Discovery through the Deep Successor Representation](https://mlsanity.com/paper/1710.11089/) (2017)
- [Scalable Bayesian Inverse Reinforcement Learning](https://mlsanity.com/paper/2102.06483/) (2021)
- [Strictly Batch Imitation Learning by Energy-based Distribution Matching](https://mlsanity.com/paper/2006.14154/) (2020)
- [Meta-Inverse Reinforcement Learning with Probabilistic Context Variables](https://mlsanity.com/paper/1909.09314/) (2019)
- [Driving Behavior Modeling using Naturalistic Human Driving Data with Inverse Reinforcement Learning](https://mlsanity.com/paper/2010.03118/) (2020)
- [Visual Semantic Planning using Deep Successor Representations](https://mlsanity.com/paper/1705.08080/) (2017)
- [Learning a Prior over Intent via Meta-Inverse Reinforcement Learning](https://mlsanity.com/paper/1805.12573/) (2018)
- [Successor Uncertainties: Exploration and Uncertainty in Temporal Difference Learning](https://mlsanity.com/paper/1810.06530/) (2018)
- [Model-based Adversarial Imitation Learning](https://mlsanity.com/paper/1612.02179/) (2016)
- [Truly Batch Apprenticeship Learning with Deep Successor Features](https://mlsanity.com/paper/1903.10077/) (2019)
- [PsiPhi-Learning: Reinforcement Learning with Demonstrations using Successor Features and Inverse Temporal Difference Learning](https://mlsanity.com/paper/2102.12560/) (2021)
- [Multi-task Maximum Entropy Inverse Reinforcement Learning](https://mlsanity.com/paper/1805.08882/) (2018)
- [Deep Inverse Q-learning with Constraints](https://mlsanity.com/paper/2008.01712/) (2020)

## Built on by (most cited first, 4 of 4)

- [When Demonstrations Meet Generative World Models: A Maximum Likelihood Framework for Offline Inverse Reinforcement Learning](https://mlsanity.com/paper/2302.07457/) (2023)
- [Non-Adversarial Inverse Reinforcement Learning via Successor Feature Matching](https://mlsanity.com/paper/2411.07007/) (2024)
- [OMG-RL:Offline Model-based Guided Reward Learning for Heparin Treatment](https://mlsanity.com/paper/2409.13299/) (2024)
- [Do as the Romans Do: Learning Universal Behaviors from Heterogeneous Agents](https://mlsanity.com/paper/2606.18537/) (2026)

## Abstract (arXiv)

This paper addresses the problem of inverse reinforcement learning (IRL) -- inferring the reward function of an agent from observing its behavior. IRL can provide a generalizable and compact representation for apprenticeship learning, and enable accurately inferring the preferences of a human in order to assist them. %and provide for more accurate prediction. However, effective IRL is challenging, because many reward functions can be compatible with an observed behavior. We focus on how prior reinforcement learning (RL) experience can be leveraged to make learning these preferences faster and more efficient. We propose the IRL algorithm BASIS (Behavior Acquisition through Successor-feature Intention inference from Samples), which leverages multi-task RL pre-training and successor features to allow an agent to build a strong basis for intentions that spans the space of possible goals in a given domain. When exposed to just a few expert demonstrations optimizing a novel goal, the agent uses its basis to quickly and effectively infer the reward function. Our experiments reveal that our method is highly effective at inferring and optimizing demonstrated reward functions, accurately inferring reward functions from less than 100 trajectories.