mlsanityData to 7 Oct 2026

Diversifying Policy Behaviors with Extrinsic Behavioral Curiosity

arXiv 2410.06151 ↗First posted 8 Oct 2024cs.LG cs.AIcited by 4

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

Imitation learning (IL) has shown promise in various applications (e.g. robot locomotion) but is often limited to learning a single expert policy, constraining behavior diversity and robustness in unpredictable real-world scenarios. To address this, we introduce Quality Diversity Inverse Reinforcement Learning (QD-IRL), a novel framework that integrates quality-diversity optimization with IRL methods, enabling agents to learn diverse behaviors from limited demonstrations. This work introduces Extrinsic Behavioral Curiosity (EBC), which allows agents to receive additional curiosity rewards from an external critic based on how novel the behaviors are with respect to a large behavioral archive. To validate the effectiveness of EBC in exploring diverse locomotion behaviors, we evaluate our method on multiple robot locomotion tasks. EBC improves the performance of QD-IRL instances with GAIL, VAIL, and DiffAIL across all included environments by up to 185%, 42%, and 150%, even surpassing expert performance by 20% in Humanoid. Furthermore, we demonstrate that EBC is applicable to Gradient-Arborescence-based Quality Diversity Reinforcement Learning (QD-RL) algorithms, where it substantially improves performance and provides a generic technique for learning behavioral-diverse policies. The source code of this work is provided at https://github.com/vanzll/EBC.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 31 of its 47 references that are in the map. First posted on arXiv 8 Oct 2024.

Builds on

31 of 47 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023DiffAIL: Diffusion Adversarial Imitation Learning11 Dec
  2. 2023A Survey of Imitation Learning: Algorithms, Recent Developments, and Challenges5 Sep
  3. 2020Intrinsic Reward Driven Imitation Learning via Generative Model26 Jun
  4. 2018Variational Discriminator Bottleneck: Improving Imitation Learning, Inverse RL, and GANs by Constraining Information Flow1 Oct

Other references · 27

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2016Generative Adversarial Imitation Learningcited by 1,576
  4. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  5. 2016End to End Learning for Self-Driving Carscited by 1,000
  6. 2016Deep Variational Information Bottleneckcited by 915
  7. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  8. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
  9. 2015Illuminating search spaces by mapping elitescited by 406
  10. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 400
  11. 2021Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulationcited by 375
  12. 2016The CMA Evolution Strategy: A Tutorialcited by 364
15 more
  1. 2015Maximum Entropy Deep Inverse Reinforcement Learningcited by 173
  2. 2018Reinforcement and Imitation Learning for Diverse Visuomotor Skillscited by 124
  3. 2019Covariance Matrix Adaptation for the Rapid Illumination of Behavior Spacecited by 78
  4. 2021Differentiable Quality Diversitycited by 71
  5. 2016Using Centroidal Voronoi Tessellations to Scale Up the Multi-dimensional Archive of Phenotypic Elites Algorithmcited by 65
  6. 2020Quality-Diversity Optimization: a novel branch of stochastic optimizationcited by 59
  7. 2020Diversity Policy Gradient for Sample Efficient Quality-Diversity Optimizationcited by 53
  8. 2019Meta-Inverse Reinforcement Learning with Probabilistic Context Variablescited by 50
  9. 2022Approximating Gradients for Differentiable Quality Diversity in Reinforcement Learningcited by 37
  10. 2023pyribs: A Bare-Bones Python Library for Quality Diversity Optimizationcited by 29
  11. 2022Covariance Matrix Adaptation MAP-Annealingcited by 24
  12. 2022Learning Options via Compressioncited by 22
  13. 2023Proximal Policy Gradient Arborescence for Quality Diversity Reinforcement Learningcited by 20
  14. 2023Multi-task Hierarchical Adversarial Inverse Reinforcement Learningcited by 15
  15. 2024Quality-Diversity Actor-Critic: Learning High-Performing and Diverse Behaviors via Value and Successor Features Criticscited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2026Time-Annealed Perturbation Sampling: Diverse Generation for Diffusion Language Modelscited by 3
  2. 2025FM-IRL: Flow-Matching for Reward Modeling and Policy Regularization in Reinforcement Learningcited by 2
  3. 2026GroupToM-Bench: Benchmarking Group Theory of Mind and Nonlinear Social Emergence in MLLMscited by 1
  4. 2026Adversarial Dual On-Policy Distillation from Expressive Teachercited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Towards Diverse Behaviors: A Benchmark for Imitation Learning with Human Demonstrations
  2. 2019A Divergence Minimization Perspective on Imitation Learning Methods
  3. 2024Imitation from Diverse Behaviors: Wasserstein Quality Diversity Imitation Learning with Single-Step Archive Exploration
  4. 2023Coherent Soft Imitation Learning
  5. 2023Infer and Adapt: Bipedal Locomotion Reward Learning from Demonstrations via Inverse Reinforcement Learning
  6. 2024Diverse Policies Recovering via Pointwise Mutual Information Weighted Imitation Learning
  7. 2021IQ-Learn: Inverse soft-Q Learning for Imitation
  8. 2025Symmetry-Guided Multi-Agent Inverse Reinforcement Learning