mlsanityData to 7 Oct 2026

DrS: Learning Reusable Dense Rewards for Multi-Stage Tasks

arXiv 2404.16779 ↗First posted 25 Apr 2024cs.LG cs.AI cs.ROcited by 9

Not on the map. Why?

Abstract (arXiv)

The success of many RL techniques heavily relies on human-engineered dense rewards, which typically demand substantial domain expertise and extensive trial and error. In our work, we propose DrS (Dense reward learning from Stages), a novel approach for learning reusable dense rewards for multi-stage tasks in a data-driven manner. By leveraging the stage structures of the task, DrS learns a high-quality dense reward from sparse rewards and demonstrations if given. The learned rewards can be textitreused in unseen tasks, thus reducing the human effort for reward engineering. Extensive experiments on three physical robot manipulation task families with 1000+ task variants demonstrate that our learned rewards can be reused in unseen tasks, resulting in improved performance and sample efficiency of RL algorithms. The learned rewards even achieve comparable performance to human-engineered rewards on some tasks. See our project page (https://sites.google.com/view/iclr24drs) for more details.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 48 of its 61 references that are in the map. First posted on arXiv 25 Apr 2024.

Builds on

48 of 61 references.

References · 48

  1. 2022Generative Adversarial Networkscited by 5,603
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2017Deep reinforcement learning from human preferencescited by 3,846
  4. 2015Trust Region Policy Optimizationcited by 2,503
  5. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,436
  6. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  7. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  8. 2016Generative Adversarial Imitation Learningcited by 1,576
  9. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  10. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  11. 2018Exploration by Random Network Distillationcited by 966
  12. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
36 more
  1. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 598
  2. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
  3. 2020robosuite: A Modular Simulation Framework and Benchmark for Robot Learningcited by 484
  4. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 436
  5. 2019Learning agile and dynamic motor skills for legged robotscited by 402
  6. 2018Reward learning from human preferences and demonstrations in Ataricited by 314
  7. 2023ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skillscited by 313
  8. 2023Language to Rewards for Robotic Skill Synthesiscited by 307
  9. 2019RoboNet: Large-Scale Multi-Robot Learningcited by 268
  10. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 256
  11. 2021MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scalecited by 226
  12. 2019Go-Explore: a New Approach for Hard-Exploration Problemscited by 207
  13. 2018Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learningcited by 203
  14. 2021ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrationscited by 193
  15. 2019A Divergence Minimization Perspective on Imitation Learning Methodscited by 188
  16. 2023Scaling Up and Distilling Down: Language-Guided Robot Skill Acquisitioncited by 156
  17. 2023Vision-Language Models as Success Detectorscited by 138
  18. 2019End-to-End Robotic Reinforcement Learning without Reward Engineeringcited by 115
  19. 2019AVID: Learning Multi-Stage Tasks via Pixel-Level Translation of Human Videoscited by 105
  20. 2021XIRL: Cross-embodiment Inverse Reinforcement Learningcited by 101
  21. 2018Playing hard exploration games by watching YouTubecited by 98
  22. 2017Meta Learning Shared Hierarchiescited by 93
  23. 2019Keeping Your Distance: Solving Sparse Reward Tasks Using Self-Balancing Shaped Rewardscited by 76
  24. 2019State Alignment-based Imitation Learningcited by 73
  25. 2020Offline Learning from Demonstrations and Unlabeled Experiencecited by 72
  26. 2021What Matters for Adversarial Imitation Learning?cited by 70
  27. 2018Variational Inverse Control with Events: A General Framework for Data-Driven Reward Definitioncited by 66
  28. 2023Waypoint-Based Imitation Learning for Robotic Manipulationcited by 59
  29. 2022Multi-skill Mobile Manipulation for Object Rearrangementcited by 46
  30. 2021Adversarial Skill Chaining for Long-Horizon Robot Manipulation via Terminal State Regularizationcited by 44
  31. 2019Positive-Unlabeled Reward Learningcited by 36
  32. 2018A Practical Approach to Insertion with Variable Socket Position Using Deep Reinforcement Learningcited by 35
  33. 2021Self-Supervised Online Reward Shaping in Sparse-Reward Environmentscited by 34
  34. 2013Learning Trajectory Preferences for Manipulators via Iterative Improvementcited by 31
  35. 2020Learning Dense Rewards for Contact-Rich Manipulation Taskscited by 18
  36. 2022Tactile Gym 2.0: Sim-to-real Deep Reinforcement Learning for Comparing Low-cost High-Resolution Robot Touchcited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 3.

9 papers.

  1. 2025SRPO: Self-Referential Policy Optimization for Vision-Language-Action Modelscited by 25
  2. 2025Multi-Stage Manipulation with Demonstration-Augmented Reward, Policy, and World Model Learningcited by 10
  3. 2025Subtask-Aware Visual Reward Learning from Segmented Demonstrationscited by 7
  4. 2025ELMUR: External Layer Memory with Update/Rewrite for Long-Horizon RL Problemscited by 6
  5. 2025Attention-Based Reward Shaping for Sparse and Delayed Rewardscited by 5
  6. 2025RDD: Retrieval-Based Demonstration Decomposer for Planner Alignment in Long-Horizon Taskscited by 2
  7. 2026SARM2: Multi-Task Stage Aware Reward Modeling for Self Improving Robotic Manipulationcited by 2
  8. 2026SUN: Agentic Robot Policy Learning with Persistent Task Programscited by 0
  9. 2026Demonstration-Free Success-Probability Reward Learning for Generalist Robot Policiescited by 0