Efficient Reinforcement Learning from Demonstration Using Local Ensemble and Reparameterization with Split and Merge of Expert Policies
On the map: Offline RL and imitation learning from data → Inverse reinforcement learning
Abstract (arXiv)
The current work on reinforcement learning (RL) from demonstrations often assumes the demonstrations are samples from an optimal policy, an unrealistic assumption in practice. When demonstrations are generated by sub-optimal policies or have sparse state-action pairs, policy learned from sub-optimal demonstrations may mislead an agent with incorrect or non-local action decisions. We propose a new method called Local Ensemble and Reparameterization with Split and Merge of expert policies (LEARN-SAM) to improve efficiency and make better use of the sub-optimal demonstrations. First, LEARN-SAM employs a new concept, the lambda-function, based on a discrepancy measure between the current state to demonstrated states to "localize" the weights of the expert policies during learning. Second, LEARN-SAM employs a split-and-merge (SAM) mechanism by separating the helpful parts in each expert demonstration and regrouping them into new expert policies to use the demonstrations selectively. Both the lambda-function and SAM mechanism help boost the learning speed. Theoretically, we prove the invariant property of reparameterized policy before and after the SAM mechanism, providing theoretical guarantees for the convergence of the employed policy gradient method. We demonstrate the superiority of the LEARN-SAM method and its robustness with varying demonstration quality and sparsity in six experiments on complex continuous control problems of low to high dimensions, compared to existing methods on RL from demonstration.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 17 of its 55 references that are in the map. First posted on arXiv 23 May 2022.
Builds on
17 of 55 references.
Closest ancestors · 1
Same direction, newest first
Other references · 16
- 2015Trust Region Policy Optimizationcited by 2,503
- 2016OpenAI Gymcited by 1,684
- 2016Generative Adversarial Imitation Learningcited by 1,576
- 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
- 2018Behavioral Cloning from Observationcited by 482
- 2016Benchmarking Deep Reinforcement Learning for Continuous Controlcited by 361
- 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 322
- 2017Deep Reinforcement Learning framework for Autonomous Drivingcited by 224
- 2015Maximum Entropy Deep Inverse Reinforcement Learningcited by 173
- 2020OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learningcited by 164
- 2020Parrot: Data-Driven Behavioral Priors for Reinforcement Learningcited by 136
- 2019Imitation Learning from Imperfect Demonstrationcited by 112
4 more
- 2019Unconstrained Monotonic Neural Networkscited by 91
- 2018Hierarchical Imitation and Reinforcement Learningcited by 88
- 2018Reinforcement Learning from Imperfect Demonstrationscited by 74
- 2017Pre-training Neural Networks with Human Demonstrations for Deep Reinforcement Learningcited by 15
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2018Fast Policy Learning through Imitation and Reinforcement
- 2023Demonstration-Regularized RL
- 2024Hybrid Inverse Reinforcement Learning
- 2021Efficiently Training On-Policy Actor-Critic Networks in Robotic Deep Reinforcement Learning with Demonstration-like Sampled Exploration
- 2022Data augmentation for efficient learning from parametric experts
- 2023BC-IRL: Learning Generalizable Reward Functions from Demonstrations
- 2026Coherent Off-Policy Improvement of Large Behavior Models with Learned Rewards
- 2019Integrating Behavior Cloning and Reinforcement Learning for Improved Performance in Dense and Sparse Reward Environments