mlsanityData to 7 Oct 2026

Revisiting Sparse Rewards for Goal-Reaching Reinforcement Learning

arXiv 2407.00324 ↗First posted 29 Jun 2024cs.RO cs.LGcited by 12

Not on the map. Why?

Abstract (arXiv)

Many real-world robot learning problems, such as pick-and-place or arriving at a destination, can be seen as a problem of reaching a goal state as soon as possible. These problems, when formulated as episodic reinforcement learning tasks, can easily be specified to align well with our intended goal: -1 reward every time step with termination upon reaching the goal state, called minimum-time tasks. Despite this simplicity, such formulations are often overlooked in favor of dense rewards due to their perceived difficulty and lack of informativeness. Our studies contrast the two reward paradigms, revealing that the minimum-time task specification not only facilitates learning higher-quality policies but can also surpass dense-reward-based policies on their own performance metrics. Crucially, we also identify the goal-hit rate of the initial policy as a robust early indicator for learning success in such sparse feedback settings. Finally, using four distinct real-robotic platforms, we show that it is possible to learn pixel-based policies from scratch within two to three hours using constant negative rewards.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 27 of its 34 references that are in the map. First posted on arXiv 29 Jun 2024.

Builds on

27 of 34 references.

References · 27

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2016OpenAI Gymcited by 1,684
  3. 2016Concrete Problems in AI Safetycited by 1,582
  4. 2017Hindsight Experience Replaycited by 1,096
  5. 2018DeepMind Control Suitecited by 935
  6. 2015End-to-End Training of Deep Visuomotor Policiescited by 847
  7. 2023Eureka: Human-Level Reward Design via Coding Large Language Modelscited by 542
  8. 2020Reinforcement Learning with Augmented Datacited by 452
  9. 2018Visual Reinforcement Learning with Imagined Goalscited by 248
  10. 2018Learning by Playing - Solving Sparse Reward Tasks from Scratchcited by 164
  11. 2023Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learningcited by 155
  12. 2018Making Sense of Vision and Touch: Self-Supervised Learning of Multimodal Representations for Contact-Rich Taskscited by 150
15 more
  1. 2024RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedbackcited by 147
  2. 2021Reward (Mis)design for Autonomous Drivingcited by 102
  3. 2018Benchmarking Reinforcement Learning Algorithms on Real-World Robotscited by 64
  4. 2022Asynchronous Reinforcement Learning for Real-Time Control of Physical Robotscited by 19
  5. 2019Autoregressive Policies for Continuous Control Deep Reinforcement Learningcited by 13
  6. 2022Real-Time Reinforcement Learning for Vision-Based Robotics Utilizing Local and Remote Computerscited by 13
  7. 2022Dynamic Decision Frequency with Continuous Optionscited by 11
  8. 2021Model-free Policy Learning with Reward Gradientscited by 9
  9. 2021An Alternate Policy Gradient Estimator for Softmax Policiescited by 9
  10. 2023Correcting discount-factor mismatch in on-policy policy gradient methodscited by 9
  11. 2023MaDi: Learning to Mask Distractions for Generalization in Visual Deep Reinforcement Learningcited by 9
  12. 2022Learning Minimum-Time Flight in Cluttered Environmentscited by 8
  13. 2020Simple Sensor Intentions for Explorationcited by 5
  14. 2024Revisiting Scalable Hessian Diagonal Approximations for Applications in Reinforcement Learningcited by 5
  15. 2023Reducing the Cost of Cycle-Time Tuning for Real-World Policy Optimizationcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 6; 2026: 3.

12 papers.

  1. 2024Deep Policy Gradient Methods Without Batch Updates, Target Networks, or Replay Bufferscited by 16
  2. 2025ReDit: Reward Dithering for Improved LLM Policy Optimizationcited by 8
  3. 2025Planner-R1: Reward Shaping Enables Efficient Agentic RL with Smaller LLMscited by 7
  4. 2025Ethical AI: Towards Defining a Collective Evaluation Frameworkcited by 3
  5. 2025Driving Beyond Privilege: Distilling Dense-Reward Knowledge into Sparse-Reward Policiescited by 2
  6. 2024Efficient Imitation Without Demonstrations via Value-Penalized Auxiliary Control from Examplescited by 1
  7. 2025Dynamic Object Masks as Goal Representations for Visual Goal-Conditioned Reinforcement Learningcited by 1
  8. 2025Learning Without Time-Based Embodiment Resets in Soft-Actor Criticcited by 1
  9. 2026World-Action Models for Robot Learning and Control: A Surveycited by 1
  10. 2024Solving Rubik's Cube Without Tricky Samplingcited by 0
  11. 2026Sample Efficient Generative Molecular Optimization with Joint Self-Improvementcited by 0
  12. 2026Learning Reach-Avoid Task with Reinforcement Learning: Vectorized Simulation and Benchmarkcited by 0