# Reinforcement Learning from Imperfect Demonstrations under Soft Expert Guidance

arXiv: https://arxiv.org/abs/1911.07109 · HTML: https://mlsanity.com/paper/1911.07109/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data.
- First posted on arXiv 16 Nov 2019.

## Placed in

- Offline RL and imitation learning from data / [Inverse reinforcement learning](https://mlsanity.com/offlinerl/inverse-reinforcement-learning/)

## Built on by (most cited first, 25 of 25)

- [ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrations](https://mlsanity.com/paper/2107.14483/) (2021)
- [Reinforcement Learning with Sparse Rewards using Guidance from Offline Demonstration](https://mlsanity.com/paper/2202.04628/) (2022)
- [GROOT: Learning to Follow Instructions by Watching Gameplay Videos](https://mlsanity.com/paper/2310.08235/) (2023)
- [Bayesian Controller Fusion: Leveraging Control Priors in Deep Reinforcement Learning for Robotics](https://mlsanity.com/paper/2107.09822/) (2021)
- [A Ranking Game for Imitation Learning](https://mlsanity.com/paper/2202.03481/) (2022)
- [Monte Carlo Augmented Actor-Critic for Sparse Reward Deep Reinforcement Learning from Suboptimal Demonstrations](https://mlsanity.com/paper/2210.07432/) (2022)
- [Multi-Agent Advisor Q-Learning](https://mlsanity.com/paper/2111.00345/) (2021)
- [Self-Imitation Learning from Demonstrations](https://mlsanity.com/paper/2203.10905/) (2022)
- [Reinforcement Learning in Robotic Motion Planning by Combined Experience-based Planning and Self-Imitation Learning](https://mlsanity.com/paper/2306.06754/) (2023)
- [Inverse Reinforcement Learning with Sub-optimal Experts](https://mlsanity.com/paper/2401.03857/) (2024)
- [Bayesian Q-learning With Imperfect Expert Demonstrations](https://mlsanity.com/paper/2210.01800/) (2022)
- [Safety Correction from Baseline: Towards the Risk-aware Policy in Robotics via Dual-agent Reinforcement Learning](https://mlsanity.com/paper/2212.06998/) (2022)
- [GAN-Based Interactive Reinforcement Learning from Demonstration and Human Evaluative Feedback](https://mlsanity.com/paper/2104.06600/) (2021)
- [Immersive Demonstrations are the Key to Imitation Learning](https://mlsanity.com/paper/2301.09157/) (2023)
- [Sharpening the Spear: Adaptive Expert-Guided Adversarial Attack Against DRL-based Autonomous Driving Policies](https://mlsanity.com/paper/2506.18304/) (2025)
- [Auto-Encoding Adversarial Imitation Learning](https://mlsanity.com/paper/2206.11004/) (2022)
- [Synergistic Reinforcement and Imitation Learning for Vision-driven Autonomous Flight of UAV Along River](https://mlsanity.com/paper/2401.09332/) (2024)
- [Restoring Noisy Demonstration for Imitation Learning With Diffusion Models](https://mlsanity.com/paper/2510.14467/) (2025)
- [Reinforcement Learning from Demonstrations by Novel Interactive Expert and Application to Automatic Berthing Control Systems for Unmanned Surface Vessel](https://mlsanity.com/paper/2202.11325/) (2022)
- [Efficient Reinforcement Learning from Demonstration Using Local Ensemble and Reparameterization with Split and Merge of Expert Policies](https://mlsanity.com/paper/2205.11019/) (2022)
- [Learning from Multiple Independent Advisors in Multi-agent Reinforcement Learning](https://mlsanity.com/paper/2301.11153/) (2023)
- [Adversarial Batch Inverse Reinforcement Learning: Learn to Reward from Imperfect Demonstration for Interactive Recommendation](https://mlsanity.com/paper/2310.19536/) (2023)
- [Policy Optimization with Smooth Guidance Learned from State-Only Demonstrations](https://mlsanity.com/paper/2401.00162/) (2023)
- [Uncertainty-Based Smooth Policy Regularisation for Reinforcement Learning with Few Demonstrations](https://mlsanity.com/paper/2509.15981/) (2025)
- [From Prior-Guided Heuristics to Deployable Agents: Accelerating Demonstration-Driven Reinforcement Learning for Deadline-Constrained Network Control](https://mlsanity.com/paper/2609.03590/) (2026)

## Abstract (arXiv)

In this paper, we study Reinforcement Learning from Demonstrations (RLfD) that improves the exploration efficiency of Reinforcement Learning (RL) by providing expert demonstrations. Most of existing RLfD methods require demonstrations to be perfect and sufficient, which yet is unrealistic to meet in practice. To work on imperfect demonstrations, we first define an imperfect expert setting for RLfD in a formal way, and then point out that previous methods suffer from two issues in terms of optimality and convergence, respectively. Upon the theoretical findings we have derived, we tackle these two issues by regarding the expert guidance as a soft constraint on regulating the policy exploration of the agent, which eventually leads to a constrained optimization problem. We further demonstrate that such problem is able to be addressed efficiently by performing a local linear search on its dual form. Considerable empirical evaluations on a comprehensive collection of benchmarks indicate our method attains consistent improvement over other RLfD counterparts.