mlsanityData to 14 Oct 2026

Offline Diversity Maximization Under Imitation Constraints

arXiv 2307.11373 ↗First posted 21 Jul 2023cs.LG cs.AI cs.ROcited by 4

Not on the map. Why?

Abstract (arXiv)

There has been significant recent progress in the area of unsupervised skill discovery, utilizing various information-theoretic objectives as measures of diversity. Despite these advances, challenges remain: current methods require significant online interaction, fail to leverage vast amounts of available task-agnostic data and typically lack a quantitative measure of skill utility. We address these challenges by proposing a principled offline algorithm for unsupervised skill discovery that, in addition to maximizing diversity, ensures that each learned skill imitates state-only expert demonstrations to a certain degree. Our main analytical contribution is to connect Fenchel duality, reinforcement learning, and unsupervised skill discovery to maximize a mutual information objective subject to KL-divergence state occupancy constraints. Furthermore, we demonstrate the effectiveness of our method on the standard offline benchmark D4RL and on a custom offline dataset collected from a 12-DoF quadruped robot for which the policies trained in simulation transfer well to the real robotic system.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 43 of its 60 references that are in the map. First posted on arXiv 21 Jul 2023.

Builds on

43 of 60 references.

References · 43

  1. 2017Improved Training of Wasserstein GANscited by 2,235
  2. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  3. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  4. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  5. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  6. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 998
  7. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 752
  8. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  9. 2018Reward Constrained Policy Optimizationcited by 417
  10. 2016Successor Features for Transfer in Reinforcement Learningcited by 340
  11. 2020Responsive Safety in Reinforcement Learning by PID Lagrangian Methodscited by 316
  12. 2019Dynamics-Aware Unsupervised Discovery of Skillscited by 313
31 more
  1. 2020Critic Regularized Regressioncited by 275
  2. 2016Variational Intrinsic Controlcited by 267
  3. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  4. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problemscited by 231
  5. 2018Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimationcited by 227
  6. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 188
  7. 2015Variational Information Maximisation for Intrinsically Motivated Reinforcement Learningcited by 144
  8. 2019Fast Task Inference with Variational Intrinsic Successor Featurescited by 133
  9. 2018Variational Option Discovery Algorithmscited by 122
  10. 2020Explore, Discover and Learn: Unsupervised Discovery of State-Covering Skillscited by 122
  11. 2021OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimationcited by 121
  12. 2023Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularizationcited by 116
  13. 2022Adversarially Trained Actor Critic for Offline Reinforcement Learningcited by 111
  14. 2020Reinforcement Learning via Fenchel-Rockafellar Dualitycited by 100
  15. 2019Transfer in Deep Reinforcement Learning Using Successor Features and Generalised Policy Improvementcited by 88
  16. 2021Reward is enough for convex MDPscited by 84
  17. 2019InfoBot: Transfer and Exploration via the Information Bottleneckcited by 78
  18. 2022COptiDICE: Offline Constrained Reinforcement Learning via Stationary Distribution Correction Estimationcited by 71
  19. 2020GradientDICE: Rethinking Generalized Offline Estimation of Stationary Valuescited by 70
  20. 2020CoinDICE: Off-Policy Confidence Interval Estimationcited by 64
  21. 2022Lipschitz-constrained Unsupervised Skill Discoverycited by 62
  22. 2023Controllability-Aware Unsupervised Skill Discoverycited by 49
  23. 2022Versatile Offline Imitation from Observations and Examples via Regularized State-Occupancy Matchingcited by 43
  24. 2021Learning more skills through optimistic explorationcited by 38
  25. 2021The Information Geometry of Unsupervised Reinforcement Learningcited by 37
  26. 2022Discovering Policies with DOMiNO: Diversity Optimization Maintaining Near Optimalitycited by 37
  27. 2021Unsupervised Skill Discovery with Bottleneck Option Learningcited by 28
  28. 2021Offline Reinforcement Learning with Soft Behavior Regularizationcited by 28
  29. 2019An Open Torque-Controlled Modular Robot Architecture for Legged Locomotion Researchcited by 22
  30. 2020Black-box Off-policy Estimation for Infinite-Horizon Reinforcement Learningcited by 16
  31. 2023Predictable MDP Abstraction for Unsupervised Model-Based RLcited by 8

Built on by

Cited by year

Citing papers per year: 2025: 4.

4 papers.

  1. 2025Zero-Shot Whole-Body Humanoid Control via Behavioral Foundation Modelscited by 56
  2. 2025Epistemically-guided forward-backward explorationcited by 8
  3. 2025Dual-Force: Enhanced Offline Diversity Maximization under Imitation Constraintscited by 2
  4. 2025Trajectory First: A Curriculum for Discovering Diverse Policiescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Diverse Skill Discovery for Quadruped Robots via Unsupervised Learning
  2. 2022Bayesian Nonparametrics for Offline Skill Discovery
  3. 2024Constrained Skill Discovery: Quadruped Locomotion with Unsupervised Reinforcement Learning
  4. 2025From Tabula Rasa to Emergent Abilities: Discovering Robot Skills via Real-World Unsupervised Quality-Diversity
  5. 2025Unsupervised Skill Discovery through Skill Regions Differentiation
  6. 2024Constrained Ensemble Exploration for Unsupervised Skill Discovery
  7. 2025Divide, Discover, Deploy: Factorized Skill Learning with Symmetry and Style Priors
  8. 2026SUSD: Structured Unsupervised Skill Discovery through State Factorization