mlsanityData to 7 Oct 2026

Diversify & Conquer: Outcome-directed Curriculum RL via Out-of-Distribution Disagreement

arXiv 2310.19261 ↗First posted 30 Oct 2023cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

Reinforcement learning (RL) often faces the challenges of uninformed search problems where the agent should explore without access to the domain knowledge such as characteristics of the environment or external rewards. To tackle these challenges, this work proposes a new approach for curriculum RL called Diversify for Disagreement & Conquer (D2C). Unlike previous curriculum learning methods, D2C requires only a few examples of desired outcomes and works in any environment, regardless of its geometry or the distribution of the desired outcome examples. The proposed method performs diversification of the goal-conditional classifiers to identify similarities between visited and desired outcome states and ensures that the classifiers disagree on states from out-of-distribution, which enables quantifying the unexplored region and designing an arbitrary goal-conditioned intrinsic reward signal in a simple and intuitive way. The proposed method then employs bipartite matching to define a curriculum learning objective that produces a sequence of well-adjusted intermediate goals, which enable the agent to automatically explore and conquer the unexplored region. We present experimental results demonstrating that D2C outperforms prior curriculum RL methods in both quantitative and qualitative aspects, even with the arbitrarily distributed desired outcome examples.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 43 of its 49 references that are in the map. First posted on arXiv 30 Oct 2023.

Builds on

43 of 49 references.

References · 43

  1. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  4. 2017Deep reinforcement learning from human preferencescited by 3,846
  5. 2020FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidencecited by 2,080
  6. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,304
  7. 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,232
  8. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  9. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  10. 2017Hindsight Experience Replaycited by 1,096
  11. 2018Exploration by Random Network Distillationcited by 966
  12. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
31 more
  1. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 632
  2. 2021Just Train Twice: Improving Group Robustness without Training Group Informationcited by 522
  3. 2019Dynamics-Aware Unsupervised Discovery of Skillscited by 308
  4. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 298
  5. 2019Self-Supervised Exploration via Disagreementcited by 263
  6. 2017Count-Based Exploration with Neural Density Modelscited by 255
  7. 2017Automatic Goal Generation for Reinforcement Learning Agentscited by 252
  8. 2020Prioritized Level Replaycited by 190
  9. 2021Reinforcement Learning with Prototypical Representationscited by 179
  10. 2021Behavior From the Void: Unsupervised Active Pre-Trainingcited by 173
  11. 2019Skew-Fit: State-Covering Self-Supervised Reinforcement Learningcited by 171
  12. 2016Correlation Alignment for Unsupervised Domain Adaptationcited by 146
  13. 2017Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Playcited by 139
  14. 2021Discovering and Achieving Goals via World Modelscited by 131
  15. 2019End-to-End Robotic Reinforcement Learning without Reward Engineeringcited by 115
  16. 2019Teacher algorithms for curriculum learning of Deep RL in continuously parameterized environmentscited by 113
  17. 2021APS: Active Pretraining with Successor Featurescited by 109
  18. 2018Unsupervised Control Through Non-Parametric Discriminative Rewardscited by 102
  19. 2020Automatic Curriculum Learning through Value Disagreementcited by 86
  20. 2022Agree to Disagree: Diversity through Disagreement for Better Transferabilitycited by 69
  21. 2018Variational Inverse Control with Events: A General Framework for Data-Driven Reward Definitioncited by 66
  22. 2022CIC: Contrastive Intrinsic Control for Unsupervised Skill Discoverycited by 66
  23. 2022Diversify and Disambiguate: Learning From Underspecified Datacited by 55
  24. 2019Exploration via Hindsight Goal Generationcited by 53
  25. 2021Replacing Rewards with Examples: Example-Based Policy Search via Recursive Classificationcited by 38
  26. 2022Choreographer: Learning and Adapting Skills in Imaginationcited by 31
  27. 2021Unsupervised Skill Discovery with Bottleneck Option Learningcited by 28
  28. 2023Planning Goals for Explorationcited by 28
  29. 2021MURAL: Meta-Learning Uncertainty-Aware Rewards for Outcome-Driven Reinforcement Learningcited by 24
  30. 2021Mutual Information State Intrinsic Controlcited by 18
  31. 2023Outcome-directed Reinforcement Learning by Uncertainty & Temporal Distance-Aware Curriculum Goal Generationcited by 14

Built on by

Cited by year

Citing papers per year: 2025: 4.

4 papers.

  1. 2025DISCOVER: Automated Curricula for Sparse-Reward Reinforcement Learningcited by 11
  2. 2025Diversify and Conquer: Open-set Disagreement for Robust Semi-supervised Learning with Outlierscited by 1
  3. 2025Causal-Paced Deep Reinforcement Learningcited by 1
  4. 2025Periodic Skill Discoverycited by 1