Data to 7 Oct 2026

Overcoming Exploration in Reinforcement Learning with Demonstrations

arXiv 1709.10089 ↗First posted 28 Sep 2017cs.LG cs.AI cs.NEcited by 322

Not on the map. Why?

Cited by 100 in Offline RL and imitation learning from data, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Exploration in environments with sparse rewards has been a persistent problem in reinforcement learning (RL). Many tasks are natural to specify with a sparse reward, and manually shaping a reward function can result in suboptimal performance. However, finding a non-zero reward is exponentially more difficult with increasing task horizon or action dimensionality. This puts many real-world tasks out of practical reach of RL methods. In this work, we use demonstrations to overcome the exploration problem and successfully learn to perform long-horizon, multi-step robotics tasks with continuous control such as stacking blocks with a robot arm. Our method, which builds on top of Deep Deterministic Policy Gradients and Hindsight Experience Replay, provides an order of magnitude of speedup over RL on simulated robotics tasks. It is simple to implement and makes only the additional assumption that we can collect a small set of demonstrations. Furthermore, our method is able to solve tasks not solvable by either RL or behavior cloning alone, and often ends up outperforming the demonstrator policy.

Key facts. Cited by 322 papers in the map (data through 7 Oct 2026). First posted on arXiv 28 Sep 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2018: 8; 2019: 8; 2020: 4; 2021: 45; 2022: 79; 2023: 58; 2024: 54; 2025: 40; 2026: 26.

322 papers.

Sort:

By area:

2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 384
2023Efficient Online Reinforcement Learning with Offline Datacited by 332
2018Reward learning from human preferences and demonstrations in Ataricited by 314
2018Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learningcited by 203
2021How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learnedcited by 201
2019Goal-conditioned Imitation Learningcited by 159
2022Exploration in Deep Reinforcement Learning: A Surveycited by 158
2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 158
2022Goal-Conditioned Reinforcement Learning: Problems and Solutionscited by 145
2025ConRFT: A Reinforced Fine-tuning Method for VLA Models via Consistency Policycited by 139
2018Self-Imitation Learningcited by 135

38 more

2024Reconciling Reality through Simulation: A Real-to-Sim-to-Real Approach for Robust Manipulationcited by 135
2018Reinforcement and Imitation Learning for Diverse Visuomotor Skillscited by 124
2023Policy Expansion for Bridging Offline-to-Online Reinforcement Learningcited by 103
2022Jump-Start Reinforcement Learningcited by 102
2021Coarse-to-Fine Imitation Learning: Robot Manipulation from a Single Demonstrationcited by 92
2022Rethinking Goal-conditioned Supervised Learning and Its Connection to Offline RLcited by 89
2021Q-attention: Enabling Efficient Learning for Vision-based Robotic Manipulationcited by 85
2022DexVIP: Learning Dexterous Grasping with Human Hand Pose Priors from Videocited by 83
2024From Imitation to Refinement -- Residual RL for Precise Assemblycited by 80
2018Vision-based Navigation with Language-based Assistance via Imitation Learning with Indirect Interventioncited by 77
2021Augmenting Reinforcement Learning with Behavior Primitives for Diverse Manipulation Taskscited by 75
2023Teach a Robot to FISH: Versatile Imitation from One Minute of Demonstrationscited by 74
2021Demonstration-Guided Reinforcement Learning with Learned Skillscited by 73
2020Offline Learning from Demonstrations and Unlabeled Experiencecited by 72
2021Asymmetric self-play for automatic goal discovery in robotic manipulationcited by 71
2022Watch and Match: Supercharging Imitation with Regularized Optimal Transportcited by 64
2022Reincarnating Reinforcement Learning: Reusing Prior Computation to Accelerate Progresscited by 63
2024Training Large Language Models for Reasoning through Reverse Curriculum Reinforcement Learningcited by 63
2024Policy Decorator: Model-Agnostic Online Refinement for Large Policy Modelcited by 60
2022MoDem: Accelerating Visual Model-Based Reinforcement Learning with Demonstrationscited by 59
2021Imitation Learning: Progress, Taxonomies and Challengescited by 56
2022Reinforcement Learning with Sparse Rewards using Guidance from Offline Demonstrationcited by 54
2023Imitation Bootstrapped Reinforcement Learningcited by 54
2023C·ASE: Learning Conditional Adversarial Skill Embeddings for Physics-based Characterscited by 53
2021Modeling Strong and Human-Like Gameplay with KL-Regularized Searchcited by 50
2023Learning to Generate Better Than Your LLMcited by 49
2022Mastering the Game of No-Press Diplomacy via Human-Regularized Reinforcement Learning and Planningcited by 47
2020Overcoming Model Bias for Robust Offline Deep Reinforcement Learningcited by 42
2022Leveraging Offline Data in Online Reinforcement Learningcited by 42
2024Dataset Reset Policy Optimization for RLHFcited by 41
2018Truncated Horizon Policy Search: Combining Reinforcement Learning & Imitation Learningcited by 39
2018DQN-TAMER: Human-in-the-Loop Reinforcement Learning with Intractable Feedbackcited by 39
2023Query-Dependent Prompt Evaluation and Optimization with Offline Inverse RLcited by 37
2023Efficient Reinforcement Learning for Autonomous Driving with Parameterized Skills and Priorscited by 36
2023AMAGO: Scalable In-Context Reinforcement Learning for Adaptive Agentscited by 36
2021Deep reinforcement learning in medical imaging: A literature reviewcited by 34
2022LEAGUE: Guided Skill Learning and Abstraction for Long-Horizon Manipulationcited by 32
2022First-order Policy Optimization for Robust Markov Decision Processcited by 31

50 most cited of 322 All 322 (JSON)