mlsanityData to 7 Oct 2026

Improving Learning from Demonstrations by Learning from Experience

arXiv 2111.08156 ↗First posted 16 Nov 2021cs.AIcited by 0

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

How to make imitation learning more general when demonstrations are relatively limited has been a persistent problem in reinforcement learning (RL). Poor demonstrations lead to narrow and biased date distribution, non-Markovian human expert demonstration makes it difficult for the agent to learn, and over-reliance on sub-optimal trajectories can make it hard for the agent to improve its performance. To solve these problems we propose a new algorithm named TD3fG that can smoothly transition from learning from experts to learning from experience. Our algorithm achieves good performance in the MUJOCO environment with limited and sub-optimal demonstrations. We use behavior cloning to train the network as a reference action generator and utilize it in terms of both loss function and exploration noise. This innovation can help agents extract a priori knowledge from demonstrations while reducing the detrimental effects of the poor Markovian properties of the demonstrations. It has a better performance compared to the BC+ fine-tuning and DDPGfD approach, especially when the demonstrations are relatively limited. We call our method TD3fG meaning TD3 from a generator.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 17 of its 23 references that are in the map. First posted on arXiv 16 Nov 2021.

Builds on

17 of 23 references.

References · 17

  1. 2015Continuous control with deep reinforcement learningcited by 3,279
  2. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  3. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  4. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  5. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  6. 2015End-to-End Training of Deep Visuomotor Policiescited by 847
  7. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 698
  8. 2019Behavior Regularized Offline Reinforcement Learningcited by 654
  9. 2018Learning Dexterous In-Hand Manipulationcited by 643
  10. 2019Exploring the Limitations of Behavior Cloning for Autonomous Drivingcited by 343
  11. 2016Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updatescited by 342
  12. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 322
5 more
  1. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 302
  2. 2019Challenges of Real-World Reinforcement Learningcited by 282
  3. 2016Query-Efficient Imitation Learning for End-to-End Autonomous Drivingcited by 83
  4. 2013Tractable Inference for Complex Stochastic Processescited by 20
  5. 2019Integrating Behavior Cloning and Reinforcement Learning for Improved Performance in Dense and Sparse Reward Environmentscited by 17

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2022Robust Imitation of a Few Demonstrations with a Backwards Model
  2. 2022Self-Imitation Learning from Demonstrations
  3. 2024Imitation Learning from Suboptimal Demonstrations via Meta-Learning An Action Ranker
  4. 2019Watch, Try, Learn: Meta-Learning from Demonstrations and Reward
  5. 2021Learning from Imperfect Demonstrations from Agents with Varying Dynamics
  6. 2024A Bayesian Solution To The Imitation Gap
  7. 2023Skill Disentanglement for Imitation Learning from Suboptimal Demonstrations
  8. 2025Learning from Observation: A Survey of Recent Advances