mlsanityData to 14 Oct 2026

DISCOVER: Automated Curricula for Sparse-Reward Reinforcement Learning

arXiv 2505.19850 ↗First posted 26 May 2025cs.LG cs.AI cs.ROcited by 12

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Sparse-reward reinforcement learning (RL) can model a wide range of highly complex tasks. Solving sparse-reward tasks is RL's core premise, requiring efficient exploration coupled with long-horizon credit assignment, and overcoming these challenges is key for building self-improving agents with superhuman ability. Prior work commonly explores with the objective of solving many sparse-reward tasks, making exploration of individual high-dimensional, long-horizon tasks intractable. We argue that solving such challenging tasks requires solving simpler tasks that are relevant to the target task, i.e., whose achieval will teach the agent skills required for solving the target task. We demonstrate that this sense of direction, necessary for effective exploration, can be extracted from existing RL algorithms, without leveraging any prior information. To this end, we propose a method for directed sparse-reward goal-conditioned very long-horizon RL (DISCOVER), which selects exploratory goals in the direction of the target task. We connect DISCOVER to principled exploration in bandits, formally bounding the time until the target task becomes achievable in terms of the agent's initial distance to the target, but independent of the volume of the space of all tasks. We then perform a thorough evaluation in high-dimensional environments. We find that the directed goal selection of DISCOVER solves exploration problems that are beyond the reach of prior state-of-the-art exploration methods in RL.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 69 of its 93 references that are in the map. First posted on arXiv 26 May 2025.

Builds on

69 of 93 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024MaxInfoRL: Boosting exploration in reinforcement learning through information gain maximization16 Dec
  2. 2024Accelerating Goal-Conditioned RL Algorithms and Research20 Aug

Other references · 67

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  3. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  4. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  5. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  6. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
  7. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  8. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
  9. 2017Hindsight Experience Replaycited by 1,136
  10. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,103
  11. 2020Mastering Atari with Discrete World Modelscited by 1,007
  12. 2018Exploration by Random Network Distillationcited by 993
38 more
  1. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 752
  2. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 734
  3. 2016Deep Exploration via Bootstrapped DQNcited by 527
  4. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 455
  5. 2017A Tutorial on Thompson Samplingcited by 433
  6. 2024Learning to (Learn at Test Time): RNNs with Expressive Hidden Statescited by 355
  7. 2017On Kernelized Multi-armed Banditscited by 352
  8. 2025Absolute Zero: Reinforced Self-play Reasoning with Zero Datacited by 344
  9. 2016#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learningcited by 309
  10. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Examplecited by 295
  11. 2020Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Designcited by 275
  12. 2025TTRL: Test-Time Reinforcement Learningcited by 275
  13. 2020First return, then explorecited by 266
  14. 2018Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Researchcited by 260
  15. 2017Automatic Goal Generation for Reinforcement Learning Agentscited by 259
  16. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 253
  17. 2019Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutionscited by 192
  18. 2019Skew-Fit: State-Covering Self-Supervised Reinforcement Learningcited by 174
  19. 2014Learning to Optimize via Information-Directed Samplingcited by 168
  20. 2018Learning by Playing - Solving Sparse Reward Tasks from Scratchcited by 167
  21. 2022Exploration in Deep Reinforcement Learning: A Surveycited by 166
  22. 2017Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Playcited by 141
  23. 2025LIMR: Less is More for RL Scalingcited by 139
  24. 2021Discovering and Achieving Goals via World Modelscited by 131
  25. 2020Self-Supervised Policy Adaptation during Deploymentcited by 123
  26. 2023Optimal Goal-Reaching Reinforcement Learning via Quasimetric Learningcited by 121
  27. 2021Epistemic Neural Networkscited by 120
  28. 2014Generalization and Exploration via Randomized Value Functionscited by 112
  29. 2018Unsupervised Control Through Non-Parametric Discriminative Rewardscited by 102
  30. 2020Maximum Entropy Gain Exploration for Long Horizon Multi-goal Reinforcement Learningcited by 102
  31. 2025One-Minute Video Generation with Test-Time Trainingcited by 101
  32. 2024The Surprising Effectiveness of Test-Time Training for Few-Shot Learningcited by 96
  33. 2023Test-Time Training on Nearest Neighbors for Large Language Modelscited by 88
  34. 2020Automatic Curriculum Learning through Value Disagreementcited by 87
  35. 2020Efficient Model-Based Reinforcement Learning through Optimistic Policy Search and Planningcited by 86
  36. 2023METRA: Scalable Unsupervised RL with Metric-Aware Abstractioncited by 86
  37. 2019Keeping Your Distance: Solving Sparse Reward Tasks Using Self-Balancing Shaped Rewardscited by 78
  38. 2019Exploration via Hindsight Goal Generationcited by 54

50 most cited of 67 All 67 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 8.

12 papers.

  1. 2026Reinforcement Learning via Self-Distillationcited by 332
  2. 2025Learning on the Job: Test-Time Curricula for Targeted Reinforcement Learningcited by 9
  3. 2025Test-time Offline Reinforcement Learning on Goal-related Experiencecited by 8
  4. 2026Autonomous Continual Learning for Environment Adaptation of Computer-Use Agentscited by 6
  5. 2025Specialization after Generalization: Towards Understanding Test-Time Training in Foundation Modelscited by 5
  6. 2026Breaking the Solver Bottleneck: Training Task Generators at the Learnable Frontiercited by 3
  7. 2025AURA: Autonomous Upskilling with Retrieval-Augmented Agentscited by 2
  8. 2026Maximum Entropy Behavior Exploration for Sim2Real Zero-Shot Reinforcement Learningcited by 1
  9. 2026Per-Context Reverse Curricula for Sparse-Reward Reinforcement Learningcited by 0
  10. 2026SUN: Reaching for Novelty in Reinforcement Learningcited by 0
  11. 2026Deep Epistemic Value Functions for Optimistic Explorationcited by 0
  12. 2026Bidirectional Voronoi-biased Exploration Curriculum for Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Search Inspired Exploration in Reinforcement Learning
  2. 2024Dense Dynamics-Aware Reward Synthesis: Integrating Prior Experience with Demonstrations
  3. 2023Accelerating Exploration with Unlabeled Prior Data
  4. 2026Learning Process Rewards via Success Visitation Matching for Efficient RL
  5. 2026Beyond Scripted Search: Sample-Efficient Reward Discovery via Agentic Black-box Optimization
  6. 2024Revisiting Sparse Rewards for Goal-Reaching Reinforcement Learning
  7. 2025Fill in the Blanks: Accelerating Q-Learning with a Handful of Demonstrations in Sparse Reward Settings
  8. 2025Autonomous state-space segmentation for Deep-RL sparse reward scenarios