# Strategy Discovery and Mixture in Lifelong Learning from Heterogeneous Demonstration

arXiv: https://arxiv.org/abs/2202.07014 · HTML: https://mlsanity.com/paper/2202.07014/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Robot learning from demonstration, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 17 of its 37 references that are in the map.
- First posted on arXiv 14 Feb 2022.

## Placed in

- Offline RL and imitation learning from data / [Robot learning from demonstration](https://mlsanity.com/offlinerl/robot-learning-from-demonstration/)

## Builds on (most cited first, 17 of 17)

- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [Soft Actor-Critic Algorithms and Applications](https://mlsanity.com/paper/1812.05905/) (2018)
- [Diversity is All You Need: Learning Skills without a Reward Function](https://mlsanity.com/paper/1802.06070/) (2018)
- [Learning Robust Rewards with Adversarial Inverse Reinforcement Learning](https://mlsanity.com/paper/1710.11248/) (2017)
- [Continual Learning for Robotics: Definition, Framework, Learning Strategies, Opportunities and Challenges](https://mlsanity.com/paper/1907.00182/) (2019)
- [Learning from Suboptimal Demonstration via Self-Supervised Reward Regression](https://mlsanity.com/paper/2010.11723/) (2020)
- [Learning Reward Functions by Integrating Human Demonstrations and Preferences](https://mlsanity.com/paper/1906.08928/) (2019)
- [Efficient Model Learning for Human-Robot Collaborative Tasks](https://mlsanity.com/paper/1405.6341/) (2014)
- [Joint Goal and Strategy Inference across Heterogeneous Demonstrators via Reward Network Distillation](https://mlsanity.com/paper/2001.00503/) (2020)
- [Interpretable and Personalized Apprenticeship Scheduling: Learning Interpretable Scheduling Policies from Heterogeneous User Demonstrations](https://mlsanity.com/paper/1906.06397/) (2019)
- [Iterated Reasoning with Mutual Information in Cooperative and Byzantine Decentralized Teaming](https://mlsanity.com/paper/2201.08484/) (2022)
- [Deep Adaptive Multi-Intention Inverse Reinforcement Learning](https://mlsanity.com/paper/2107.06692/) (2021)
- [Heterogeneous Graph Attention Networks for Learning Diverse Communication](https://mlsanity.com/paper/2108.09568/) (2021)
- [Inverse Reinforce Learning with Nonparametric Behavior Clustering](https://mlsanity.com/paper/1712.05514/) (2017)
- [Towards Sample-efficient Apprenticeship Learning from Suboptimal Demonstration](https://mlsanity.com/paper/2110.04347/) (2021)

## Built on by (most cited first, 1 of 1)

- [Robots That Take Initiative: A Framework for Building and Evaluating Proactive Robots](https://mlsanity.com/paper/2609.28910/) (2026)

## Abstract (arXiv)

Learning from Demonstration (LfD) approaches empower end-users to teach robots novel tasks via demonstrations of the desired behaviors, democratizing access to robotics. A key challenge in LfD research is that users tend to provide heterogeneous demonstrations for the same task due to various strategies and preferences. Therefore, it is essential to develop LfD algorithms that ensure textitflexibility (the robot adapts to personalized strategies), textitefficiency (the robot achieves sample-efficient adaptation), and textitscalability (robot reuses a concise set of strategies to represent a large amount of behaviors). In this paper, we propose a novel algorithm, Dynamic Multi-Strategy Reward Distillation (DMSRD), which distills common knowledge between heterogeneous demonstrations, leverages learned strategies to construct mixture policies, and continues to improve by learning from all available data. Our personalized, federated, and lifelong LfD architecture surpasses benchmarks in two continuous control problems with an average 77% improvement in policy returns and 42% improvement in log likelihood, alongside stronger task reward correlation and more precise strategy rewards.