mlsanityData to 7 Oct 2026

Reinforcement Learning from Imperfect Demonstrations under Soft Expert Guidance

arXiv 1911.07109 ↗First posted 16 Nov 2019cs.LG cs.AI cs.ROcited by 25

On the map: Offline RL and imitation learning from data → Inverse reinforcement learning

Abstract (arXiv)

In this paper, we study Reinforcement Learning from Demonstrations (RLfD) that improves the exploration efficiency of Reinforcement Learning (RL) by providing expert demonstrations. Most of existing RLfD methods require demonstrations to be perfect and sufficient, which yet is unrealistic to meet in practice. To work on imperfect demonstrations, we first define an imperfect expert setting for RLfD in a formal way, and then point out that previous methods suffer from two issues in terms of optimality and convergence, respectively. Upon the theoretical findings we have derived, we tackle these two issues by regarding the expert guidance as a soft constraint on regulating the policy exploration of the agent, which eventually leads to a constrained optimization problem. We further demonstrate that such problem is able to be addressed efficiently by performing a local linear search on its dual form. Considerable empirical evaluations on a comprehensive collection of benchmarks indicate our method attains consistent improvement over other RLfD counterparts.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data. First posted on arXiv 16 Nov 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 9; 2023: 6; 2024: 2; 2025: 3; 2026: 1.

25 papers.

Sort:

By area:

  1. 2021ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrationscited by 193
  2. 2022Reinforcement Learning with Sparse Rewards using Guidance from Offline Demonstrationcited by 54
  3. 2023GROOT: Learning to Follow Instructions by Watching Gameplay Videoscited by 42
  4. 2021Bayesian Controller Fusion: Leveraging Control Priors in Deep Reinforcement Learning for Roboticscited by 18
  5. 2022A Ranking Game for Imitation Learningcited by 18
  6. 2022Monte Carlo Augmented Actor-Critic for Sparse Reward Deep Reinforcement Learning from Suboptimal Demonstrationscited by 14
  7. 2021Multi-Agent Advisor Q-Learningcited by 4
  8. 2022Self-Imitation Learning from Demonstrationscited by 4
  9. 2023Reinforcement Learning in Robotic Motion Planning by Combined Experience-based Planning and Self-Imitation Learningcited by 4
  10. 2024Inverse Reinforcement Learning with Sub-optimal Expertscited by 4
  11. 2022Bayesian Q-learning With Imperfect Expert Demonstrationscited by 3
  12. 2022Safety Correction from Baseline: Towards the Risk-aware Policy in Robotics via Dual-agent Reinforcement Learningcited by 3
13 more
  1. 2021GAN-Based Interactive Reinforcement Learning from Demonstration and Human Evaluative Feedbackcited by 2
  2. 2023Immersive Demonstrations are the Key to Imitation Learningcited by 2
  3. 2025Sharpening the Spear: Adaptive Expert-Guided Adversarial Attack Against DRL-based Autonomous Driving Policiescited by 2
  4. 2022Auto-Encoding Adversarial Imitation Learningcited by 1
  5. 2024Synergistic Reinforcement and Imitation Learning for Vision-driven Autonomous Flight of UAV Along Rivercited by 1
  6. 2025Restoring Noisy Demonstration for Imitation Learning With Diffusion Modelscited by 1
  7. 2022Reinforcement Learning from Demonstrations by Novel Interactive Expert and Application to Automatic Berthing Control Systems for Unmanned Surface Vesselcited by 0
  8. 2022Efficient Reinforcement Learning from Demonstration Using Local Ensemble and Reparameterization with Split and Merge of Expert Policiescited by 0
  9. 2023Learning from Multiple Independent Advisors in Multi-agent Reinforcement Learningcited by 0
  10. 2023Adversarial Batch Inverse Reinforcement Learning: Learn to Reward from Imperfect Demonstration for Interactive Recommendationcited by 0
  11. 2023Policy Optimization with Smooth Guidance Learned from State-Only Demonstrationscited by 0
  12. 2025Uncertainty-Based Smooth Policy Regularisation for Reinforcement Learning with Few Demonstrationscited by 0
  13. 2026From Prior-Guided Heuristics to Deployable Agents: Accelerating Demonstration-Driven Reinforcement Learning for Deadline-Constrained Network Controlcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Reinforcement learning with Demonstrations from Mismatched Task under Sparse Reward
  2. 2021Learning from Ambiguous Demonstrations with Self-Explanation Guided Reinforcement Learning
  3. 2020Exploration-efficient Deep Reinforcement Learning with Demonstration Guidance for Robot Control
  4. 2025Generalizing Behavior via Inverse Reinforcement Learning with Closed-Form Reward Centroids
  5. 2023Is Inverse Reinforcement Learning Harder than Standard Reinforcement Learning? A Theoretical Perspective
  6. 2025Enhancing Online Reinforcement Learning with Meta-Learned Objective from Offline Data
  7. 2024The Virtues of Pessimism in Inverse Reinforcement Learning
  8. 2024Hybrid Inverse Reinforcement Learning