mlsanityData to 7 Oct 2026

Scaling data-driven robotics with reward sketching and batch reinforcement learning

arXiv 1909.12200 ↗First posted 26 Sep 2019cs.RO cs.LGcited by 18

On the map: Offline RL and imitation learning from data → Real-world robot offline RL

Abstract (arXiv)

We present a framework for data-driven robotics that makes use of a large dataset of recorded robot experience and scales to several tasks using learned reward functions. We show how to apply this framework to accomplish three different object manipulation tasks on a real robot platform. Given demonstrations of a task together with task-agnostic recorded experience, we use a special form of human annotation as supervision to learn a reward function, which enables us to deal with real-world tasks where the reward signal cannot be acquired directly. Learned rewards are used in combination with a large dataset of experience from different tasks to learn a robot policy offline using batch RL. We show that using our approach it is possible to train agents to perform a variety of challenging manipulation tasks including stacking rigid objects and handling cloth.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Placed in Real-world robot offline RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 40 of its 87 references that are in the map. First posted on arXiv 26 Sep 2019.

Builds on

40 of 87 references.

References · 40

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  5. 2016Layer Normalizationcited by 5,579
  6. 2017Deep reinforcement learning from human preferencescited by 3,846
  7. 2016Generative Adversarial Imitation Learningcited by 1,576
  8. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  9. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
  10. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  11. 2015End-to-End Training of Deep Visuomotor Policiescited by 847
  12. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
28 more
  1. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  2. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  3. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  4. 2018Learning Dexterous In-Hand Manipulationcited by 643
  5. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
  6. 2015Deep Speech 2: End-to-End Speech Recognition in English and Mandarincited by 449
  7. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 400
  8. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 322
  9. 2018Reward learning from human preferences and demonstrations in Ataricited by 314
  10. 2016Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collectioncited by 312
  11. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 302
  12. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  13. 2019Challenges of Real-World Reinforcement Learningcited by 282
  14. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 256
  15. 2017Asymmetric Actor Critic for Image-Based Robot Learningcited by 202
  16. 2018Distributed Distributional Deterministic Policy Gradientscited by 195
  17. 2017Using Simulation and Domain Adaptation to Improve Efficiency of Deep Robotic Graspingcited by 138
  18. 2018Reinforcement and Imitation Learning for Diverse Visuomotor Skillscited by 124
  19. 2018Sim-to-Real Reinforcement Learning for Deformable Object Manipulationcited by 124
  20. 2019End-to-End Robotic Reinforcement Learning without Reward Engineeringcited by 115
  21. 2017Vision-Based Multi-Task Manipulation for Inexpensive Robots Using End-To-End Learning from Demonstrationcited by 100
  22. 2017Learning human behaviors from motion capture by adversarial imitationcited by 75
  23. 2017Transferring End-to-End Visuomotor Control from Simulation to Real World for a Multi-Stage Taskcited by 66
  24. 2016Unsupervised Perceptual Rewards for Imitation Learningcited by 64
  25. 2018Observe and Look Further: Achieving Consistent Performance on Ataricited by 42
  26. 2019An Optimistic Perspective on Offline Reinforcement Learningcited by 37
  27. 2018A Practical Approach to Insertion with Variable Socket Position Using Deep Reinforcement Learningcited by 35
  28. 2019Compositional Transfer in Hierarchical Reinforcement Learningcited by 7

Built on by

Cited by year

Citing papers per year: 2019: 1; 2020: 5; 2021: 5; 2022: 3; 2023: 4.

18 papers.

Sort:

By area:

  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  2. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  3. 2021How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learnedcited by 201
  4. 2020Hyperparameter Selection for Offline Reinforcement Learningcited by 113
  5. 2023Policy Expansion for Bridging Offline-to-Online Reinforcement Learningcited by 103
  6. 2021S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learningcited by 85
  7. 2020COG: Connecting New Skills to Past Experience with Offline Reinforcement Learningcited by 83
  8. 2021Beyond Pick-and-Place: Tackling Robotic Stacking of Diverse Shapescited by 70
  9. 2021A Workflow for Offline Model-Free Robotic Reinforcement Learningcited by 68
  10. 2019Positive-Unlabeled Reward Learningcited by 36
  11. 2021A Dataset Perspective on Offline Reinforcement Learningcited by 27
  12. 2022The Challenges of Exploration for Offline Reinforcement Learningcited by 26
6 more
  1. 2022Dungeons and Data: A Large-Scale NetHack Datasetcited by 16
  2. 2022Offline Reinforcement Learning with Closed-Form Policy Improvement Operatorscited by 14
  3. 2023Learning a Universal Human Prior for Dexterous Manipulation from Human Preferencecited by 10
  4. 2020DERAIL: Diagnostic Environments for Reward And Imitation Learningcited by 6
  5. 2023Contrastive Example-Based Controlcited by 3
  6. 2023Towards Multi-User Activity Recognition through Facilitated Training Data and Deep Learning for Human-Robot Collaboration Applicationscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Reward Machine Inference for Robotic Manipulation
  2. 2026Large Reward Models: Generalizable Online Robot Reward Generation with Vision-Language Models
  3. 2024Learning Manipulation Skills through Robot Chain-of-Thought with Sparse Failure Guidance
  4. 2022How to Spend Your Robot Time: Bridging Kickstarting and Offline Reinforcement Learning for Vision-based Robotic Manipulation
  5. 2024EXTRACT: Efficient Policy Learning by Extracting Transferable Robot Skills from Offline Data
  6. 2024Data Scaling Laws in Imitation Learning for Robotic Manipulation
  7. 2021Efficient Robotic Manipulation Through Offline-to-Online Reinforcement Learning and Goal-Aware State Information
  8. 2023Offline Skill Generalization via Task and Motion Planning