mlsanityData to 13 Oct 2026

Keeping Your Distance: Solving Sparse Reward Tasks Using Self-Balancing Shaped Rewards

arXiv 1911.01417 ↗First posted 4 Nov 2019cs.AIcited by 76

Not on the map. Why?

Cited by 22 in Deep reinforcement learning methods, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

While using shaped rewards can be beneficial when solving sparse reward tasks, their successful application often requires careful engineering and is problem specific. For instance, in tasks where the agent must achieve some goal state, simple distance-to-goal reward shaping often fails, as it renders learning vulnerable to local optima. We introduce a simple and effective model-free method to learn from shaped distance-to-goal rewards on tasks where success depends on reaching a goal state. Our method introduces an auxiliary distance-based reward based on pairs of rollouts to encourage diverse exploration. This approach effectively prevents learning dynamics from stabilizing around local optima induced by the naive distance-to-goal reward shaping and enables policies to efficiently solve sparse reward tasks. Our augmented objective does not require any additional reward engineering or domain expertise to implement and converges to the original sparse objective as the agent learns to solve the task. We demonstrate that our method successfully solves a variety of hard-exploration tasks (including maze navigation and 3D construction in a Minecraft environment), where naive distance-based reward shaping otherwise fails, and intrinsic curiosity and reward relabeling strategies exhibit poor performance.

Key facts. Cited by 76 papers in the map (data through 13 Oct 2026). First posted on arXiv 4 Nov 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2021: 15; 2022: 9; 2023: 14; 2024: 17; 2025: 15; 2026: 6.

76 papers.

Sort:

By area:

  1. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 339
  2. 2024HuatuoGPT-o1, Towards Medical Complex Reasoning with LLMscited by 260
  3. 2024ReFT: Reasoning with Reinforced Fine-Tuningcited by 237
  4. 2022Goal-Conditioned Reinforcement Learning: Problems and Solutionscited by 148
  5. 2021Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domaincited by 57
  6. 2021Self-Supervised Online Reward Shaping in Sparse-Reward Environmentscited by 34
  7. 2022Generalizing Goal-Conditioned Reinforcement Learning with Variational Causal Reasoningcited by 33
  8. 2023MCU: An Evaluation Framework for Open-Ended Game Agentscited by 33
  9. 2025Towards Better Alignment: Training Diffusion Models with Reinforcement Learning Against Sparse Rewardscited by 31
  10. 2023Planning Goals for Explorationcited by 28
  11. 2024LLM-Empowered State Representation for Reinforcement Learningcited by 28
  12. 2024Highly Efficient Self-Adaptive Reward Shaping for Reinforcement Learningcited by 22
38 more
  1. 2024A Single Goal is All You Need: Skills and Exploration Emerge from Contrastive RL without Rewards, Demonstrations, or Subgoalscited by 21
  2. 2021An Information-Theoretic Perspective on Credit Assignment in Reinforcement Learningcited by 20
  3. 2021GridToPix: Training Embodied Agents with Minimal Supervisioncited by 20
  4. 2022Reinforcement Learning for Branch-and-Bound Optimisation using Retrospective Trajectoriescited by 19
  5. 2023Reward Shaping for Happier Autonomous Cyber Security Agentscited by 19
  6. 2021PC-MLP: Model-based Reinforcement Learning with Policy Cover Guided Explorationcited by 14
  7. 2024Solving Minimum-Cost Reach Avoid using Reinforcement Learningcited by 14
  8. 2021Sparse Reward Exploration via Novelty Search and Emitterscited by 13
  9. 2021MapGo: Model-Assisted Policy Optimization for Goal-Oriented Taskscited by 13
  10. 2025DISCOVER: Automated Curricula for Sparse-Reward Reinforcement Learningcited by 12
  11. 2025Multi-Stage Manipulation with Demonstration-Augmented Reward, Policy, and World Model Learningcited by 10
  12. 2022Neuro-symbolic Natural Logic with Introspective Revision for Natural Language Inferencecited by 9
  13. 2024DrS: Learning Reusable Dense Rewards for Multi-Stage Taskscited by 9
  14. 2023Variational Curriculum Reinforcement Learning for Unsupervised Discovery of Skillscited by 8
  15. 2024Pre-trained Visual Dynamics Representations for Efficient Policy Learningcited by 8
  16. 2023Hierarchical Empowerment: Towards Tractable Empowerment-Based Skill Learningcited by 7
  17. 2024To the Max: Reinventing Reward in Reinforcement Learningcited by 7
  18. 2025ClinicalGPT-R1: Pushing reasoning capability of generalist disease diagnosis with large language modelcited by 7
  19. 2023Building Open-Ended Embodied Agent via Language-Policy Bidirectional Adaptationcited by 6
  20. 2024World Models with Hints of Large Language Models for Goal Achievingcited by 6
  21. 2024Learning World Models for Unconstrained Goal Navigationcited by 6
  22. 2021Discovering and Exploiting Sparse Rewards in a Learned Behavior Spacecited by 5
  23. 2022Stein Variational Goal Generation for adaptive Exploration in Multi-Goal Reinforcement Learningcited by 5
  24. 2023GOATS: Goal Sampling Adaptation for Scooping with Curriculum Reinforcement Learningcited by 5
  25. 2025Attention-Based Reward Shaping for Sparse and Delayed Rewardscited by 5
  26. 2021Unsupervised Skill-Discovery and Skill-Learning in Minecraftcited by 4
  27. 2025Tree-Guided Diffusion Plannercited by 4
  28. 2021Goal-constrained Sparse Reinforcement Learning for End-to-End Drivingcited by 3
  29. 2021Reward prediction for representation learning and reward shapingcited by 3
  30. 2023Learning a Generic Value-Selection Heuristic Inside a Constraint Programming Solvercited by 3
  31. 2024Informativeness of Reward Functions in Reinforcement Learningcited by 3
  32. 2021Reinforcement Learning With Sparse-Executing Actions via Sparsity Regularizationcited by 2
  33. 2021CubeTR: Learning to Solve The Rubiks Cube Using Transformerscited by 2
  34. 2022Goal Exploration Augmentation via Pre-trained Skills for Sparse-Reward Long-Horizon Goal-Conditioned Reinforcement Learningcited by 2
  35. 2023Magnetic Field-Based Reward Shaping for Goal-Conditioned Reinforcement Learningcited by 2
  36. 2024WayEx: Waypoint Exploration using a Single Demonstrationcited by 2
  37. 2024Goal-Reaching Policy Learning from Non-Expert Observations via Effective Subgoal Guidancecited by 2
  38. 2024ETGL-DDPG: A Deep Deterministic Policy Gradient Algorithm for Sparse Reward Continuous Controlcited by 2

50 most cited of 76 All 76 (JSON)