Data to 7 Oct 2026

Automatic Curriculum Learning through Value Disagreement

arXiv 2006.09641 ↗First posted 17 Jun 2020cs.LG cs.AI cs.ROcited by 86

Not on the map. Why?

Abstract (arXiv)

Continually solving new, unsolved tasks is the key to learning diverse behaviors. Through reinforcement learning (RL), we have made massive strides towards solving tasks that have a single goal. However, in the multi-task domain, where an agent needs to reach multiple goals, the choice of training goals can largely affect sample efficiency. When biological agents learn, there is often an organized and meaningful order to which learning happens. Inspired by this, we propose setting up an automatic curriculum for goals that the agent needs to solve. Our key insight is that if we can sample goals at the frontier of the set of goals that an agent is able to reach, it will provide a significantly stronger learning signal compared to randomly sampled goals. To operationalize this idea, we introduce a goal proposal module that prioritizes goals that maximize the epistemic uncertainty of the Q-function of the policy. This simple technique samples goals that are neither too hard nor too easy for the agent to solve, hence enabling continual improvement. We evaluate our method across 13 multi-goal robotic tasks and 5 navigation tasks, and demonstrate performance gains over current state-of-the-art methods.

Key facts. Cited by 86 papers in the map (data through 7 Oct 2026). First posted on arXiv 17 Jun 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 14; 2022: 19; 2023: 24; 2024: 9; 2025: 12; 2026: 8.

86 papers.

Sort:

By area:

2021Curriculum Learning: A Surveycited by 257
2022Evolving Curricula with Regret-Based Environment Designcited by 166
2022Goal-Conditioned Reinforcement Learning: Problems and Solutionscited by 145
2021Discovering and Achieving Goals via World Modelscited by 131
2021Replay-Guided Adversarial Environment Designcited by 108
2021Asymmetric self-play for automatic goal discovery in robotic manipulationcited by 71
2022BYOL-Explore: Exploration by Bootstrapped Predictioncited by 71
2022Automated Reinforcement Learning (AutoRL): A Survey and Open Problemscited by 69
2023OMNI: Open-endedness via Models of human Notions of Interestingnesscited by 66
2023Towards A Unified Agent with Foundation Modelscited by 46
2021Adversarial Intrinsic Motivation for Reinforcement Learningcited by 40
2021Learning more skills through optimistic explorationcited by 37

38 more

2023Decomposing the Enigma: Subgoal-based Demonstration Learning for Formal Theorem Provingcited by 37
2021Multi-task curriculum learning in a complex, visual, hard-exploration domain: Minecraftcited by 34
2022Environment Generation for Zero-Shot Compositional Reinforcement Learningcited by 32
2023Planning Goals for Explorationcited by 28
2021Self-Paced Context Evaluation for Contextual Reinforcement Learningcited by 25
2023ALAN: Autonomously Exploring Robotic Agents in the Real Worldcited by 23
2025Learning to Reason at the Frontier of Learnabilitycited by 20
2023Proximal Curriculum for Reinforcement Learning Agentscited by 19
2021TeachMyAgent: a Benchmark for Automatic Curriculum Learning in Deep RLcited by 18
2022Understanding the Complexity Gains of Single-Task RL with a Curriculumcited by 16
2022It Takes Four to Tango: Multiagent Selfplay for Automatic Curriculum Generationcited by 14
2023Outcome-directed Reinforcement Learning by Uncertainty & Temporal Distance-Aware Curriculum Goal Generationcited by 14
2024Acquiring Diverse Skills using Curriculum Reinforcement Learning with Mixture of Expertscited by 14
2023CQM: Curriculum Reinforcement Learning with a Quantized World Modelcited by 11
2024DRED: Zero-Shot Transfer in Reinforcement Learning via Data-Regularised Environment Designcited by 11
2025DISCOVER: Automated Curricula for Sparse-Reward Reinforcement Learningcited by 11
2025State-Covering Trajectory Stitching for Diffusion Plannerscited by 11
2023Imitation Learning from Observation with Automatic Discount Schedulingcited by 10
2023Robust Adversarial Reinforcement Learning via Bounded Rationality Curriculacited by 9
2023Variational Curriculum Reinforcement Learning for Unsupervised Discovery of Skillscited by 8
2021DisTop: Discovering a Topological representation to learn diverse and rewarding skillscited by 7
2023Visual-Policy Learning through Multi-Camera View to Single-Camera View Knowledge Distillation for Robot Manipulation Taskscited by 7
2025Beyond Accuracy: Dissecting Mathematical Reasoning for LLMs Under Reinforcement Learningcited by 7
2023Paced-Curriculum Distillation with Prediction and Label Uncertainty for Image Segmentationcited by 6
2023On the Benefit of Optimal Transport for Curriculum Reinforcement Learningcited by 6
2024Proximal Curriculum with Task Correlations for Deep Reinforcement Learningcited by 6
2024Learning World Models for Unconstrained Goal Navigationcited by 6
2022Open-Ended Reinforcement Learning with Neural Reward Functionscited by 5
2022Stein Variational Goal Generation for adaptive Exploration in Multi-Goal Reinforcement Learningcited by 5
2022Cell-Free Latent Go-Explorecited by 5
2024Grounded Curriculum Learningcited by 5
2026When Should We Introduce Safety Interventions During Pretraining?cited by 5
2026From Trainee to Trainer: LLM-Designed Training Environment for RL with Multi-Agent Reasoningcited by 5
2021Adaptive Multi-Goal Explorationcited by 4
2022Curiosity in Hindsight: Intrinsic Exploration in Stochastic Environmentscited by 4
2023Diversify & Conquer: Outcome-directed Curriculum RL via Out-of-Distribution Disagreementcited by 4
2022Active Task Randomization: Learning Robust Skills via Unsupervised Generation of Diverse and Feasible Taskscited by 3
2024Reset-free Reinforcement Learning with World Modelscited by 3

50 most cited of 86 All 86 (JSON)