# Learning a Prior over Intent via Meta-Inverse Reinforcement Learning

arXiv: https://arxiv.org/abs/1805.12573 · HTML: https://mlsanity.com/paper/1805.12573/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 31 of its 65 references that are in the map.
- First posted on arXiv 31 May 2018.

## Placed in

- Offline RL and imitation learning from data / [Inverse reinforcement learning](https://mlsanity.com/offlinerl/inverse-reinforcement-learning/)

## Builds on (most cited first, 30 of 31)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [Matching Networks for One Shot Learning](https://mlsanity.com/paper/1606.04080/) (2016)
- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [End-to-End Training of Deep Visuomotor Policies](https://mlsanity.com/paper/1504.00702/) (2015)
- [Learning to learn by gradient descent by gradient descent](https://mlsanity.com/paper/1606.04474/) (2016)
- [RL2: Fast Reinforcement Learning via Slow Reinforcement Learning](https://mlsanity.com/paper/1611.02779/) (2016)
- [Learning Robust Rewards with Adversarial Inverse Reinforcement Learning](https://mlsanity.com/paper/1710.11248/) (2017)
- [Semantic Scene Completion from a Single Depth Image](https://mlsanity.com/paper/1611.08974/) (2016)
- [Learning to reinforcement learn](https://mlsanity.com/paper/1611.05763/) (2016)
- [One-Shot Imitation Learning](https://mlsanity.com/paper/1703.07326/) (2017)
- [One-Shot Visual Imitation Learning via Meta-Learning](https://mlsanity.com/paper/1709.04905/) (2017)
- [Recurrent Neural Network Regularization](https://mlsanity.com/paper/1409.2329/) (2014)
- [Recasting Gradient-Based Meta-Learning as Hierarchical Bayes](https://mlsanity.com/paper/1801.08930/) (2018)
- [Maximum Entropy Deep Inverse Reinforcement Learning](https://mlsanity.com/paper/1507.04888/) (2015)
- [Multi-Modal Imitation Learning from Unstructured Demonstrations using Generative Adversarial Nets](https://mlsanity.com/paper/1705.10479/) (2017)
- [One-Shot Generalization in Deep Generative Models](https://mlsanity.com/paper/1603.05106/) (2016)
- [Learning to Optimize Neural Nets](https://mlsanity.com/paper/1703.00441/) (2017)
- [Watch This: Scalable Cost-Function Learning for Path Planning in Urban Environments](https://mlsanity.com/paper/1607.02329/) (2016)
- [Few-shot Autoregressive Density Estimation: Towards Learning to Learn Distributions](https://mlsanity.com/paper/1710.10304/) (2017)
- [Early Stopping is Nonparametric Variational Inference](https://mlsanity.com/paper/1504.01344/) (2015)
- [A Simple Neural Attentive Meta-Learner](https://mlsanity.com/paper/1707.03141/) (2017)
- [Attentive Recurrent Comparators](https://mlsanity.com/paper/1703.00767/) (2017)
- [TorchCraft: a Library for Machine Learning Research on Real-Time Strategy Games](https://mlsanity.com/paper/1611.00625/) (2016)
- [InfoGAIL: Interpretable Imitation Learning from Visual Demonstrations](https://mlsanity.com/paper/1703.08840/) (2017)
- [Burn-In Demonstrations for Multi-Modal Imitation Learning](https://mlsanity.com/paper/1710.05090/) (2017)
- [Incorporating Human Domain Knowledge into Large Scale Cost Function Learning](https://mlsanity.com/paper/1612.04318/) (2016)

## Built on by (most cited first, 30 of 36)

- [Getting aligned on representational alignment](https://mlsanity.com/paper/2310.13018/) (2023)
- [Generalized Decision Transformer for Offline Hindsight Information Matching](https://mlsanity.com/paper/2111.10364/) (2021)
- [Meta-Inverse Reinforcement Learning with Probabilistic Context Variables](https://mlsanity.com/paper/1909.09314/) (2019)
- [Baby Intuitions Benchmark (BIB): Discerning the goals, preferences, and actions of others](https://mlsanity.com/paper/2102.11938/) (2021)
- [Bayesian Robust Optimization for Imitation Learning](https://mlsanity.com/paper/2007.12315/) (2020)
- [Robust Imitation Learning against Variations in Environment Dynamics](https://mlsanity.com/paper/2206.09314/) (2022)
- [MORAL: Aligning AI with Human Norms through Multi-Objective Reinforced Active Learning](https://mlsanity.com/paper/2201.00012/) (2021)
- [NOPA: Neurally-guided Online Probabilistic Assistance for Building Socially Intelligent Home Assistants](https://mlsanity.com/paper/2301.05223/) (2023)
- [Learning Safety Constraints from Demonstrations with Unknown Rewards](https://mlsanity.com/paper/2305.16147/) (2023)
- [Theory of Mind as Intrinsic Motivation for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2307.01158/) (2023)
- [OPIRL: Sample Efficient Off-Policy Inverse Reinforcement Learning via Distribution Matching](https://mlsanity.com/paper/2109.04307/) (2021)
- [Learning Shared Safety Constraints from Multi-task Demonstrations](https://mlsanity.com/paper/2309.00711/) (2023)
- [SIRL: Similarity-based Implicit Representation Learning](https://mlsanity.com/paper/2301.00810/) (2023)
- [Inverse Reinforcement Learning in Contextual MDPs](https://mlsanity.com/paper/1905.09710/) (2019)
- [Policy Gradient Bayesian Robust Optimization for Imitation Learning](https://mlsanity.com/paper/2106.06499/) (2021)
- [Aligning Robot and Human Representations](https://mlsanity.com/paper/2302.01928/) (2023)
- [The Role of Predictive Uncertainty and Diversity in Embodied AI and Robot Learning](https://mlsanity.com/paper/2405.03164/) (2024)
- [Designing Fiduciary Artificial Intelligence](https://mlsanity.com/paper/2308.02435/) (2023)
- [Combining Reward Information from Multiple Sources](https://mlsanity.com/paper/2103.12142/) (2021)
- [Transfering Hierarchical Structure with Dual Meta Imitation Learning](https://mlsanity.com/paper/2201.11981/) (2022)
- [BC-IRL: Learning Generalizable Reward Functions from Demonstrations](https://mlsanity.com/paper/2303.16194/) (2023)
- [In-Trajectory Inverse Reinforcement Learning: Learn Incrementally Before An Ongoing Trajectory Terminates](https://mlsanity.com/paper/2410.15612/) (2024)
- [Meta-Adversarial Inverse Reinforcement Learning for Decision-making Tasks](https://mlsanity.com/paper/2103.12694/) (2021)
- [Basis for Intentions: Efficient Inverse Reinforcement Learning using Past Experience](https://mlsanity.com/paper/2208.04919/) (2022)
- [Discovering Generalizable Spatial Goal Representations via Graph-based Active Reward Learning](https://mlsanity.com/paper/2211.15339/) (2022)
- [Meta-Inverse Reinforcement Learning for Mean Field Games via Probabilistic Context Variables](https://mlsanity.com/paper/2509.03845/) (2025)
- [Multi Task Inverse Reinforcement Learning for Common Sense Reward](https://mlsanity.com/paper/2402.11367/) (2024)
- [SS-MAIL: Self-Supervised Multi-Agent Imitation Learning](https://mlsanity.com/paper/2110.08963/) (2021)
- [Aligning Robot Representations with Humans](https://mlsanity.com/paper/2205.07882/) (2022)
- [Kernel Density Bayesian Inverse Reinforcement Learning](https://mlsanity.com/paper/2303.06827/) (2023)

## Abstract (arXiv)

A significant challenge for the practical application of reinforcement learning in the real world is the need to specify an oracle reward function that correctly defines a task. Inverse reinforcement learning (IRL) seeks to avoid this challenge by instead inferring a reward function from expert behavior. While appealing, it can be impractically expensive to collect datasets of demonstrations that cover the variation common in the real world (e.g. opening any type of door). Thus in practice, IRL must commonly be performed with only a limited set of demonstrations where it can be exceedingly difficult to unambiguously recover a reward function. In this work, we exploit the insight that demonstrations from other tasks can be used to constrain the set of possible reward functions by learning a "prior" that is specifically optimized for the ability to infer expressive reward functions from limited numbers of demonstrations. We demonstrate that our method can efficiently recover rewards from images for novel tasks and provide intuition as to how our approach is analogous to learning a prior.