# Versatile Inverse Reinforcement Learning via Cumulative Rewards

arXiv: https://arxiv.org/abs/2111.07667 · HTML: https://mlsanity.com/paper/2111.07667/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 14 of its 33 references that are in the map.
- First posted on arXiv 15 Nov 2021.

## Placed in

- Offline RL and imitation learning from data / [Inverse reinforcement learning](https://mlsanity.com/offlinerl/inverse-reinforcement-learning/)

## Builds on (most cited first, 14 of 14)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Variational Inference: A Review for Statisticians](https://mlsanity.com/paper/1601.00670/) (2016)
- [Dota 2 with Large Scale Deep Reinforcement Learning](https://mlsanity.com/paper/1912.06680/) (2019)
- [Reinforcement Learning with Deep Energy-Based Policies](https://mlsanity.com/paper/1702.08165/) (2017)
- [Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization](https://mlsanity.com/paper/1603.00448/) (2016)
- [Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observations](https://mlsanity.com/paper/1904.06387/) (2019)
- [A Divergence Minimization Perspective on Imitation Learning Methods](https://mlsanity.com/paper/1911.02256/) (2019)
- [Non-Adversarial Imitation Learning and its Connections to Adversarial Methods](https://mlsanity.com/paper/2008.03525/) (2020)
- [Trust-Region Variational Inference with Gaussian Mixture Models](https://mlsanity.com/paper/1907.04710/) (2019)
- [Expected Information Maximization: Using the I-Projection for Mixture Density Estimation](https://mlsanity.com/paper/2001.08682/) (2020)

## Built on by (most cited first, 2 of 2)

- [Inferring Versatile Behavior from Demonstrations by Matching Geometric Descriptors](https://mlsanity.com/paper/2210.08121/) (2022)
- [Curriculum-Based Imitation of Versatile Skills](https://mlsanity.com/paper/2304.05171/) (2023)

## Abstract (arXiv)

Inverse Reinforcement Learning infers a reward function from expert demonstrations, aiming to encode the behavior and intentions of the expert. Current approaches usually do this with generative and uni-modal models, meaning that they encode a single behavior. In the common setting, where there are various solutions to a problem and the experts show versatile behavior this severely limits the generalization capabilities of these methods. We propose a novel method for Inverse Reinforcement Learning that overcomes these problems by formulating the recovered reward as a sum of iteratively trained discriminators. We show on simulated tasks that our approach is able to recover general, high-quality reward functions and produces policies of the same quality as behavioral cloning approaches designed for versatile behavior.