mlsanityData to 7 Oct 2026

Reinforcement Learning from Imperfect Demonstrations

arXiv 1802.05313 ↗First posted 14 Feb 2018cs.AI cs.LG stat.MLcited by 74

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

Robust real-world learning should benefit from both demonstrations and interactions with the environment. Current approaches to learning from demonstration and reward perform supervised learning on expert demonstration data and use reinforcement learning to further improve performance based on the reward received from the environment. These tasks have divergent losses which are difficult to jointly optimize and such methods can be very sensitive to noisy demonstrations. We propose a unified reinforcement learning algorithm, Normalized Actor-Critic (NAC), that effectively normalizes the Q-function, reducing the Q-values of actions unseen in the demonstration data. NAC learns an initial policy network from demonstrations and refines the policy in the environment, surpassing the demonstrator's performance. Crucially, both learning from demonstration and interactive refinement use the same objective, unlike prior approaches that combine distinct supervised and reinforcement losses. This makes NAC robust to suboptimal demonstration data since the method is not forced to mimic all of the examples in the dataset. We show that our unified reinforcement learning algorithm can learn robustly and outperform existing baselines when evaluated on several realistic driving games.

Key facts. Cited by 74 papers in the map (data through 7 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. First posted on arXiv 14 Feb 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2018: 5; 2019: 6; 2020: 4; 2021: 19; 2022: 18; 2023: 4; 2024: 6; 2025: 10; 2026: 2.

74 papers.

Sort:

By area:

  1. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  2. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 256
  3. 2024ReFT: Reasoning with Reinforced Fine-Tuningcited by 236
  4. 2019Goal-conditioned Imitation Learningcited by 159
  5. 2018Self-Imitation Learningcited by 135
  6. 2019BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learningcited by 90
  7. 2018Vision-based Navigation with Language-based Assistance via Imitation Learning with Indirect Interventioncited by 77
  8. 2022Reincarnating Reinforcement Learning: Reusing Prior Computation to Accelerate Progresscited by 63
  9. 2021GRI: General Reinforced Imitation and its Application to Vision-Based Autonomous Drivingcited by 59
  10. 2022Reinforcement Learning with Sparse Rewards using Guidance from Offline Demonstrationcited by 54
  11. 2021The MineRL 2020 Competition on Sample Efficient Reinforcement Learning using Human Priorscited by 40
  12. 2022Aligning to Social Norms and Values in Interactive Narrativescited by 32
38 more
  1. 2019SQIL: Imitation Learning via Reinforcement Learning with Sparse Rewardscited by 28
  2. 2021PsiPhi-Learning: Reinforcement Learning with Demonstrations using Successor Features and Inverse Temporal Difference Learningcited by 23
  3. 2018Imitation Learning with Concurrent Actions in 3D Gamescited by 18
  4. 2022On Pathologies in KL-Regularized Reinforcement Learning from Expert Demonstrationscited by 18
  5. 2019Adversarial Imitation Learning from Incomplete Demonstrationscited by 16
  6. 2020Towards Target-Driven Visual Navigation in Indoor Scenes via Generative Imitation Learningcited by 14
  7. 2022Monte Carlo Augmented Actor-Critic for Sparse Reward Deep Reinforcement Learning from Suboptimal Demonstrationscited by 14
  8. 2018OIL: Observational Imitation Learningcited by 11
  9. 2024Inverse Reinforcement Learning by Estimating Expertise of Demonstratorscited by 11
  10. 2020Episodic Self-Imitation Learning with Hindsightcited by 10
  11. 2021Action Advising with Advice Imitation in Deep Reinforcement Learningcited by 10
  12. 2022Explainable Action Advising for Multi-Agent Reinforcement Learningcited by 10
  13. 2021Deep reinforcement learning for guidewire navigation in coronary artery phantomcited by 9
  14. 2024Proximal Policy Distillationcited by 9
  15. 2018LIFT: Reinforcement Learning in Computer Systems by Learning From Demonstrationscited by 8
  16. 2021Learning from an Exploring Demonstrator: Optimal Reward Estimation for Banditscited by 8
  17. 2021Convergent and Efficient Deep Q Network Algorithmcited by 8
  18. 2022Goal-Aware Generative Adversarial Imitation Learning from Imperfect Demonstration for Robotic Cloth Manipulationcited by 7
  19. 2021Generative Adversarial Reward Learning for Generalized Behavior Tendency Inferencecited by 6
  20. 2024Leveraging Sub-Optimal Data for Human-in-the-Loop Reinforcement Learningcited by 6
  21. 2019Better-than-Demonstrator Imitation Learning via Automatically-Ranked Demonstrationscited by 5
  22. 2020POPO: Pessimistic Offline Policy Optimizationcited by 5
  23. 2021Instance-Aware Predictive Navigation in Multi-Agent Environmentscited by 5
  24. 2021Learning from Ambiguous Demonstrations with Self-Explanation Guided Reinforcement Learningcited by 5
  25. 2021Learning from demonstrations with SACR2: Soft Actor-Critic with Reward Relabelingcited by 4
  26. 2021Multi-Agent Advisor Q-Learningcited by 4
  27. 2022Self-Imitation Learning from Demonstrationscited by 4
  28. 2022Reinforcement learning with Demonstrations from Mismatched Task under Sparse Rewardcited by 4
  29. 2023ACPO: A Policy Optimization Algorithm for Average MDPs with Constraintscited by 4
  30. 2022A Joint Imitation-Reinforcement Learning Framework for Reduced Baseline Regretcited by 3
  31. 2022Safety Correction from Baseline: Towards the Risk-aware Policy in Robotics via Dual-agent Reinforcement Learningcited by 3
  32. 2023ChemGymRL: An Interactive Framework for Reinforcement Learning for Digital Chemistrycited by 3
  33. 2020Generalization Guarantees for Imitation Learningcited by 2
  34. 2021Co-Imitation Learning without Expert Demonstrationcited by 2
  35. 2021GAN-Based Interactive Reinforcement Learning from Demonstration and Human Evaluative Feedbackcited by 2
  36. 2021End-to-end grasping policies for human-in-the-loop robots via deep reinforcement learningcited by 2
  37. 2025A Survey of Reinforcement Learning-Based Motion Planning for Autonomous Driving: Lessons Learned from a Driving Task Perspectivecited by 2
  38. 2025TW-CRL: Time-Weighted Contrastive Reward Learning for Efficient Inverse Reinforcement Learningcited by 2

50 most cited of 74 All 74 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2019Reward-Conditioned Policies
  2. 2020Shaping Rewards for Reinforcement Learning with Imperfect Demonstrations using Generative Models
  3. 2018Teaching Inverse Reinforcement Learners via Features and Demonstrations
  4. 2022Robust Imitation of a Few Demonstrations with a Backwards Model
  5. 2022Robust Learning from Observation with Model Misspecification
  6. 2023Demonstration-Regularized RL
  7. 2019Reinforcement Learning from Imperfect Demonstrations under Soft Expert Guidance
  8. 2021Learning from Imperfect Demonstrations from Agents with Varying Dynamics