mlsanityData to 7 Oct 2026

A Study on Dense and Sparse (Visual) Rewards in Robot Policy Learning

arXiv 2108.03222 ↗First posted 6 Aug 2021cs.RO cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

Deep Reinforcement Learning (DRL) is a promising approach for teaching robots new behaviour. However, one of its main limitations is the need for carefully hand-coded reward signals by an expert. We argue that it is crucial to automate the reward learning process so that new skills can be taught to robots by their users. To address such automation, we consider task success classifiers using visual observations to estimate the rewards in terms of task success. In this work, we study the performance of multiple state-of-the-art deep reinforcement learning algorithms under different types of reward: Dense, Sparse, Visual Dense, and Visual Sparse rewards. Our experiments in various simulation tasks (Pendulum, Reacher, Pusher, and Fetch Reach) show that while DRL agents can learn successful behaviours using visual rewards when the goal targets are distinguishable, their performance may decrease if the task goal is not clearly visible. Our results also show that visual dense rewards are more successful than visual sparse rewards and that there is no single best algorithm for all tasks.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 22 of its 30 references that are in the map. First posted on arXiv 6 Aug 2021.

Builds on

22 of 30 references.

References · 22

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2015Continuous control with deep reinforcement learningcited by 3,279
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  6. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 400
  7. 2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 400
  8. 2016Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collectioncited by 312
  9. 2018Visual Reinforcement Learning with Imagined Goalscited by 248
  10. 2019End-to-End Robotic Reinforcement Learning without Reward Engineeringcited by 115
  11. 2016Loss is its own Reward: Self-Supervision for Reinforcement Learningcited by 72
  12. 2018Variational Inverse Control with Events: A General Framework for Data-Driven Reward Definitioncited by 66
10 more
  1. 2016Unsupervised Perceptual Rewards for Imitation Learningcited by 64
  2. 2019Deep Reinforcement Learning for Industrial Insertion Tasks with Visual Inputs and Natural Rewardscited by 58
  3. 2018No Metrics Are Perfect: Adversarial Reward Learning for Visual Storytellingcited by 46
  4. 2019Contextual Imagined Goals for Self-Supervised Robotic Learningcited by 40
  5. 2018A Practical Approach to Insertion with Variable Socket Position Using Deep Reinforcement Learningcited by 35
  6. 2016Watch This: Scalable Cost-Function Learning for Path Planning in Urban Environmentscited by 33
  7. 2018Few-Shot Goal Inference for Visuomotor Learning and Planningcited by 30
  8. 2018Reward Learning from Narrated Demonstrationscited by 13
  9. 2016Perceptual Reward Functionscited by 3
  10. 2017Cross-Domain Perceptual Reward Functionscited by 2

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 2.

3 papers.

  1. 2023A Reinforcement Learning Approach for Scheduling Problems With Improved Generalization Through Order Swappingcited by 1
  2. 2023Boosting Feedback Efficiency of Interactive Reinforcement Learning by Adaptive Learning from Scorescited by 1
  3. 2022Robot Policy Learning from Demonstration Using Advantage Weighting and Early Terminationcited by 0