mlsanityData to 14 Oct 2026

Demonstration-efficient Inverse Reinforcement Learning in Procedurally Generated Environments

arXiv 2012.02527 ↗First posted 4 Dec 2020cs.LG cs.AIcited by 6

On the map: Offline RL and imitation learning from data → Inverse reinforcement learning

Abstract (arXiv)

Deep Reinforcement Learning achieves very good results in domains where reward functions can be manually engineered. At the same time, there is growing interest within the community in using games based on Procedurally Content Generation (PCG) as benchmark environments since this type of environment is perfect for studying overfitting and generalization of agents under domain shift. Inverse Reinforcement Learning (IRL) can instead extrapolate reward functions from expert demonstrations, with good results even on high-dimensional problems, however there are no examples of applying these techniques to procedurally-generated environments. This is mostly due to the number of demonstrations needed to find a good reward model. We propose a technique based on Adversarial Inverse Reinforcement Learning which can significantly decrease the need for expert demonstrations in PCG games. Through the use of an environment with a limited set of initial seed levels, plus some modifications to stabilize training, we show that our approach, DE-AIRL, is demonstration-efficient and still able to extrapolate reward functions which generalize to the fully procedural domain. We demonstrate the effectiveness of our technique on two procedural environments, MiniGrid and DeepCrawl, for a variety of tasks.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 22 of its 38 references that are in the map. First posted on arXiv 4 Dec 2020.

Builds on

22 of 38 references.

Closest ancestors · 2

Same direction, newest first

  1. 2019Adversarial recovery of agent rewards from latent spaces of the limit order book9 Dec
  2. 2018Inverse reinforcement learning for video games24 Oct

Other references · 20

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2017Deep reinforcement learning from human preferencescited by 3,912
  3. 2015Trust Region Policy Optimizationcited by 2,580
  4. 2015Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networkscited by 2,054
  5. 2016Concrete Problems in AI Safetycited by 1,642
  6. 2016Generative Adversarial Imitation Learningcited by 1,597
  7. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,103
  8. 2015End-to-End Training of Deep Visuomotor Policiescited by 875
  9. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 490
  10. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 483
  11. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 406
  12. 2018Reward learning from human preferences and demonstrations in Ataricited by 316
8 more
  1. 2019MineRL: A Large-Scale Dataset of Minecraft Demonstrationscited by 215
  2. 2020The NetHack Learning Environmentcited by 198
  3. 2016A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Modelscited by 168
  4. 2018Illuminating Generalization in Deep Reinforcement Learning through Procedural Level Generationcited by 81
  5. 2019Increasing Generality in Machine Learning through Procedural Content Generationcited by 59
  6. 2019Obstacle Tower: A Generalization Challenge in Vision, Control, and Planningcited by 54
  7. 2019High-Fidelity Image Generation With Fewer Labelscited by 52
  8. 2020DeepCrawl: Deep Reinforcement Learning for Turn-based Strategy Gamescited by 6

Built on by

Cited by year

Citing papers per year: 2021: 2; 2022: 0; 2023: 1; 2024: 2; 2025: 1.

6 papers.

  1. 2021A Survey of Zero-shot Generalisation in Deep Reinforcement Learningcited by 170
  2. 2021Policy Fusion for Adaptive and Customizable Reinforcement Learning Agentscited by 4
  3. 2023Improving Generalization in Game Agents with Data Augmentation in Imitation Learningcited by 4
  4. 2024ARMCHAIR: integrated inverse reinforcement learning and model predictive control for human-robot collaborationcited by 2
  5. 2024Rethinking Adversarial Inverse Reinforcement Learning: Policy Imitation, Transferable Reward Recovery and Algebraic Equilibrium Proofcited by 0
  6. 2025TROFI: Trajectory-Ranked Offline Inverse Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Non-Adversarial Inverse Reinforcement Learning via Successor Feature Matching
  2. 2020oIRL: Robust Adversarial Inverse Reinforcement Learning with Temporally Extended Actions
  3. 2019Multi-Agent Adversarial Inverse Reinforcement Learning
  4. 2022Active Exploration for Inverse Reinforcement Learning
  5. 2020Off-Policy Adversarial Inverse Reinforcement Learning
  6. 2021Off-Dynamics Inverse Reinforcement Learning from Hetero-Domain
  7. 2022Environment Design for Inverse Reinforcement Learning
  8. 2018Competitive Multi-agent Inverse Reinforcement Learning with Sub-optimal Demonstrations