# Meta-Inverse Reinforcement Learning with Probabilistic Context Variables

arXiv: https://arxiv.org/abs/1909.09314 · HTML: https://mlsanity.com/paper/1909.09314/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 26 of its 42 references that are in the map.
- First posted on arXiv 20 Sep 2019.

## Placed in

- Offline RL and imitation learning from data / [Inverse reinforcement learning](https://mlsanity.com/offlinerl/inverse-reinforcement-learning/)

## Builds on (most cited first, 26 of 26)

- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Learning to learn by gradient descent by gradient descent](https://mlsanity.com/paper/1606.04474/) (2016)
- [RL2: Fast Reinforcement Learning via Slow Reinforcement Learning](https://mlsanity.com/paper/1611.02779/) (2016)
- [Learning Robust Rewards with Adversarial Inverse Reinforcement Learning](https://mlsanity.com/paper/1710.11248/) (2017)
- [Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review](https://mlsanity.com/paper/1805.00909/) (2018)
- [Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization](https://mlsanity.com/paper/1603.00448/) (2016)
- [Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables](https://mlsanity.com/paper/1903.08254/) (2019)
- [Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperation](https://mlsanity.com/paper/1710.04615/) (2017)
- [Meta Networks](https://mlsanity.com/paper/1703.00837/) (2017)
- [One-Shot Imitation Learning](https://mlsanity.com/paper/1703.07326/) (2017)
- [One-Shot Visual Imitation Learning via Meta-Learning](https://mlsanity.com/paper/1709.04905/) (2017)
- [A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Models](https://mlsanity.com/paper/1611.03852/) (2016)
- [One-Shot Imitation from Observing Humans via Domain-Adaptive Meta-Learning](https://mlsanity.com/paper/1802.01557/) (2018)
- [Variational Discriminator Bottleneck: Improving Imitation Learning, Inverse RL, and GANs by Constraining Information Flow](https://mlsanity.com/paper/1810.00821/) (2018)
- [Multi-Modal Imitation Learning from Unstructured Demonstrations using Generative Adversarial Nets](https://mlsanity.com/paper/1705.10479/) (2017)
- [Meta Reinforcement Learning with Latent Variable Gaussian Processes](https://mlsanity.com/paper/1803.07551/) (2018)
- [Learning to Optimize Neural Nets](https://mlsanity.com/paper/1703.00441/) (2017)
- [Directed-Info GAIL: Learning Hierarchical Policies from Unsegmented Demonstrations using Directed Information](https://mlsanity.com/paper/1810.01266/) (2018)
- [Learning a Prior over Intent via Meta-Inverse Reinforcement Learning](https://mlsanity.com/paper/1805.12573/) (2018)
- [One-Shot Hierarchical Imitation Learning of Compound Visuomotor Tasks](https://mlsanity.com/paper/1810.11043/) (2018)
- [Multi-task Maximum Entropy Inverse Reinforcement Learning](https://mlsanity.com/paper/1805.08882/) (2018)
- [Burn-In Demonstrations for Multi-Modal Imitation Learning](https://mlsanity.com/paper/1710.05090/) (2017)

## Built on by (most cited first, 30 of 50)

- [IQ-Learn: Inverse soft-Q Learning for Imitation](https://mlsanity.com/paper/2106.12142/) (2021)
- [Generalized Decision Transformer for Offline Hindsight Information Matching](https://mlsanity.com/paper/2111.10364/) (2021)
- [Why So Pessimistic? Estimating Uncertainties for Offline RL through Ensembles, and Why Their Independence Matters](https://mlsanity.com/paper/2205.13703/) (2022)
- [Procedure Planning in Instructional Videos via Contextual Modeling and Model-based Policy Learning](https://mlsanity.com/paper/2110.01770/) (2021)
- [Baby Intuitions Benchmark (BIB): Discerning the goals, preferences, and actions of others](https://mlsanity.com/paper/2102.11938/) (2021)
- [Hard Choices in Artificial Intelligence](https://mlsanity.com/paper/2106.11022/) (2021)
- [Robust Imitation Learning against Variations in Environment Dynamics](https://mlsanity.com/paper/2206.09314/) (2022)
- [MORAL: Aligning AI with Human Norms through Multi-Objective Reinforced Active Learning](https://mlsanity.com/paper/2201.00012/) (2021)
- [Deep Generative Models for Offline Policy Learning: Tutorial, Survey, and Perspectives on Future Directions](https://mlsanity.com/paper/2402.13777/) (2024)
- [Learning Safety Constraints from Demonstrations with Unknown Rewards](https://mlsanity.com/paper/2305.16147/) (2023)
- [OPIRL: Sample Efficient Off-Policy Inverse Reinforcement Learning via Distribution Matching](https://mlsanity.com/paper/2109.04307/) (2021)
- [Multi-task Hierarchical Adversarial Inverse Reinforcement Learning](https://mlsanity.com/paper/2305.12633/) (2023)
- [Learning Shared Safety Constraints from Multi-task Demonstrations](https://mlsanity.com/paper/2309.00711/) (2023)
- [SIRL: Similarity-based Implicit Representation Learning](https://mlsanity.com/paper/2301.00810/) (2023)
- [A Survey of Demonstration Learning](https://mlsanity.com/paper/2303.11191/) (2023)
- [Aligning Robot and Human Representations](https://mlsanity.com/paper/2302.01928/) (2023)
- [Adversarial Inverse Reinforcement Learning for Mean Field Games](https://mlsanity.com/paper/2104.14654/) (2021)
- [Transfering Hierarchical Structure with Dual Meta Imitation Learning](https://mlsanity.com/paper/2201.11981/) (2022)
- [BC-IRL: Learning Generalizable Reward Functions from Demonstrations](https://mlsanity.com/paper/2303.16194/) (2023)
- [GoIRL: Graph-Oriented Inverse Reinforcement Learning for Multimodal Trajectory Prediction](https://mlsanity.com/paper/2506.21121/) (2025)
- [Grounded Relational Inference: Domain Knowledge Driven Explainable Autonomous Driving](https://mlsanity.com/paper/2102.11905/) (2021)
- [Meta-Adversarial Inverse Reinforcement Learning for Decision-making Tasks](https://mlsanity.com/paper/2103.12694/) (2021)
- [Learning without Knowing: Unobserved Context in Continuous Transfer Reinforcement Learning](https://mlsanity.com/paper/2106.03833/) (2021)
- [Basis for Intentions: Efficient Inverse Reinforcement Learning using Past Experience](https://mlsanity.com/paper/2208.04919/) (2022)
- [Diversifying Policy Behaviors with Extrinsic Behavioral Curiosity](https://mlsanity.com/paper/2410.06151/) (2024)
- [Identifying Selections for Unsupervised Subtask Discovery](https://mlsanity.com/paper/2410.21616/) (2024)
- [Off-Dynamics Inverse Reinforcement Learning from Hetero-Domain](https://mlsanity.com/paper/2110.11443/) (2021)
- [Meta-Inverse Reinforcement Learning for Mean Field Games via Probabilistic Context Variables](https://mlsanity.com/paper/2509.03845/) (2025)
- [LiMIIRL: Lightweight Multiple-Intent Inverse Reinforcement Learning](https://mlsanity.com/paper/2106.01777/) (2021)
- [Sequential Decision Making with Expert Demonstrations under Unobserved Heterogeneity](https://mlsanity.com/paper/2404.07266/) (2024)

## Abstract (arXiv)

Providing a suitable reward function to reinforcement learning can be difficult in many real world applications. While inverse reinforcement learning (IRL) holds promise for automatically learning reward functions from demonstrations, several major challenges remain. First, existing IRL methods learn reward functions from scratch, requiring large numbers of demonstrations to correctly infer the reward for each task the agent may need to perform. Second, existing methods typically assume homogeneous demonstrations for a single behavior or task, while in practice, it might be easier to collect datasets of heterogeneous but related behaviors. To this end, we propose a deep latent variable model that is capable of learning rewards from demonstrations of distinct but related tasks in an unsupervised way. Critically, our model can infer rewards for new, structurally-similar tasks from a single demonstration. Our experiments on multiple continuous control tasks demonstrate the effectiveness of our approach compared to state-of-the-art imitation and inverse reinforcement learning methods.