mlsanityData to 7 Oct 2026

Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards

arXiv 1707.08817 ↗First posted 27 Jul 2017cs.AIcited by 302

Not on the map. Why?

Cited by 96 in Offline RL and imitation learning from data, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We propose a general and model-free approach for Reinforcement Learning (RL) on real robotics with sparse rewards. We build upon the Deep Deterministic Policy Gradient (DDPG) algorithm to use demonstrations. Both demonstrations and actual interactions are used to fill a replay buffer and the sampling ratio between demonstrations and transitions is automatically tuned via a prioritized replay mechanism. Typically, carefully engineered shaping rewards are required to enable the agents to efficiently explore on high dimensional control problems such as robotics. They are also required for model-based acceleration methods relying on local solvers such as iLQG (e.g. Guided Policy Search and Normalized Advantage Function). The demonstrations replace the need for carefully engineered rewards, and reduce the exploration problem encountered by classical RL approaches in these domains. Demonstrations are collected by a robot kinesthetically force-controlled by a human demonstrator. Results on four simulated insertion tasks show that DDPG from demonstrations out-performs DDPG, and does not require engineered rewards. Finally, we demonstrate the method on a real robotics task consisting of inserting a clip (flexible object) into a rigid object.

Key facts. Cited by 302 papers in the map (data through 7 Oct 2026). First posted on arXiv 27 Jul 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2018: 5; 2019: 7; 2020: 7; 2021: 60; 2022: 58; 2023: 55; 2024: 48; 2025: 38; 2026: 24.

302 papers.

Sort:

By area:

  1. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  2. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  3. 2023Efficient Online Reinforcement Learning with Offline Datacited by 332
  4. 2018Reward learning from human preferences and demonstrations in Ataricited by 314
  5. 2024Diffusion Policy Policy Optimizationcited by 255
  6. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 230
  7. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemblecited by 211
  8. 2018Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learningcited by 203
  9. 2021How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learnedcited by 201
  10. 2021DexMV: Imitation Learning for Dexterous Manipulation from Human Videoscited by 194
  11. 2025VLA-RL: Towards Masterful and General Robotic Manipulation with Scalable Reinforcement Learningcited by 166
  12. 2019Goal-conditioned Imitation Learningcited by 159
38 more
  1. 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 158
  2. 2024Deep Reinforcement Learning for Robotics: A Survey of Real-World Successescited by 153
  3. 2023Neural Combinatorial Optimization with Heavy Decoder: Toward Large Scale Generalizationcited by 140
  4. 2018Reinforcement and Imitation Learning for Diverse Visuomotor Skillscited by 124
  5. 2022Imitation Is Not Enough: Robustifying Imitation with Reinforcement Learning for Challenging Driving Scenarioscited by 124
  6. 2020Hyperparameter Selection for Offline Reinforcement Learningcited by 113
  7. 2025Improving Vision-Language-Action Model with Online Reinforcement Learningcited by 107
  8. 2023Policy Expansion for Bridging Offline-to-Online Reinforcement Learningcited by 103
  9. 2022Jump-Start Reinforcement Learningcited by 102
  10. 2021Coarse-to-Fine Imitation Learning: Robot Manipulation from a Single Demonstrationcited by 92
  11. 2020State-Only Imitation Learning for Dexterous Manipulationcited by 90
  12. 2021Q-attention: Enabling Efficient Learning for Vision-based Robotic Manipulationcited by 85
  13. 2022DexVIP: Learning Dexterous Grasping with Human Hand Pose Priors from Videocited by 83
  14. 2023Optimizing DDPM Sampling with Shortcut Fine-Tuningcited by 81
  15. 2022From One Hand to Multiple Hands: Imitation Learning for Dexterous Manipulation from Single-Camera Teleoperationcited by 79
  16. 2022Pre-Training for Robots: Offline RL Enables Learning New Tasks from a Handful of Trialscited by 78
  17. 2022Dexterous Imitation Made Easy: A Learning-Based Framework for Efficient Dexterous Manipulationcited by 75
  18. 2021Demonstration-Guided Reinforcement Learning with Learned Skillscited by 73
  19. 2020Offline Learning from Demonstrations and Unlabeled Experiencecited by 72
  20. 2025Self-Improving Vision-Language-Action Models with Data Generation via Residual RLcited by 72
  21. 2021Asymmetric self-play for automatic goal discovery in robotic manipulationcited by 71
  22. 2024Policy Decorator: Model-Agnostic Online Refinement for Large Policy Modelcited by 60
  23. 2025BREAD: Branched Rollouts from Expert Anchors Bridge SFT & RL for Reasoningcited by 57
  24. 2022Reinforcement Learning with Sparse Rewards using Guidance from Offline Demonstrationcited by 54
  25. 2023Imitation Bootstrapped Reinforcement Learningcited by 54
  26. 2024FLaRe: Achieving Masterful and Adaptive Robot Policies with Large-Scale Reinforcement Learning Fine-Tuningcited by 54
  27. 2022Retrieval-Augmented Reinforcement Learningcited by 53
  28. 2022VRL3: A Data-Driven Framework for Visual Deep Reinforcement Learningcited by 51
  29. 2021Modeling Strong and Human-Like Gameplay with KL-Regularized Searchcited by 50
  30. 2022Factory: Fast Contact for Robotic Assemblycited by 49
  31. 2023Dual RL: Unification and New Methods for Reinforcement and Imitation Learningcited by 48
  32. 2021Efficient Deep Reinforcement Learning with Imitative Expert Priors for Autonomous Drivingcited by 46
  33. 2022Learning Generalizable Dexterous Manipulation from Human Grasp Affordancecited by 46
  34. 2019Task-Relevant Adversarial Imitation Learningcited by 45
  35. 2022Equivariant Transporter Networkcited by 44
  36. 2023Adaptive Policy Learning for Offline-to-Online Reinforcement Learningcited by 40
  37. 2018Truncated Horizon Policy Search: Combining Reinforcement Learning & Imitation Learningcited by 39
  38. 2021Growing 3D Artefacts and Functional Machines with Neural Cellular Automatacited by 38

50 most cited of 302 All 302 (JSON)