# Robot Policy Transfer with Online Demonstrations: An Active Reinforcement Learning Approach

arXiv: https://arxiv.org/abs/2503.12993 · HTML: https://mlsanity.com/paper/2503.12993/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL for robot locomotion and manipulation, one of 15 directions in Deep reinforcement learning methods.
- Builds on 21 of its 39 references that are in the map.
- First posted on arXiv 17 Mar 2025.

## Placed in

- Deep reinforcement learning methods / [RL for robot locomotion and manipulation](https://mlsanity.com/deeprl/rl-for-robot-locomotion-and-manipulation/)

## Builds on (most cited first, 21 of 21)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [AWAC: Accelerating Online Reinforcement Learning with Offline Datasets](https://mlsanity.com/paper/2006.09359/) (2020)
- [Efficient Online Reinforcement Learning with Offline Data](https://mlsanity.com/paper/2302.02948/) (2023)
- [HG-DAgger: Interactive Imitation Learning with Human Experts](https://mlsanity.com/paper/1810.02890/) (2018)
- [Query-Efficient Imitation Learning for End-to-End Autonomous Driving](https://mlsanity.com/paper/1605.06450/) (2016)
- [Learning Reward Functions from Diverse Sources of Human Feedback: Optimally Integrating Demonstrations and Preferences](https://mlsanity.com/paper/2006.14091/) (2020)
- [panda-gym: Open-source goal-conditioned environments for robotic learning](https://mlsanity.com/paper/2106.13687/) (2021)
- [The EMPATHIC Framework for Task Learning from Implicit Human Feedback](https://mlsanity.com/paper/2009.13649/) (2020)
- [Correct Me if I am Wrong: Interactive Learning for Robotic Manipulation](https://mlsanity.com/paper/2110.03316/) (2021)
- [Efficient Sim-to-real Transfer of Contact-Rich Manipulation Skills with Online Admittance Residual Learning](https://mlsanity.com/paper/2310.10509/) (2023)
- [Cross-Domain Transfer via Semantic Skill Imitation](https://mlsanity.com/paper/2212.07407/) (2022)
- [AdaptSim: Task-Driven Simulation Adaptation for Sim-to-Real Transfer](https://mlsanity.com/paper/2302.04903/) (2023)
- [Transfer Learning in Robotics: An Upcoming Breakthrough? A Review of Promises and Challenges](https://mlsanity.com/paper/2311.18044/) (2023)
- [Active Deep Q-learning with Demonstration](https://mlsanity.com/paper/1812.02632/) (2018)
- [Taming the Panda with Python: A Powerful Duo for Seamless Robotics Programming and Integration](https://mlsanity.com/paper/2307.07633/) (2023)
- [Interactively shaping robot behaviour with unlabeled human instructions](https://mlsanity.com/paper/1902.01670/) (2019)
- [Transfering Hierarchical Structure with Dual Meta Imitation Learning](https://mlsanity.com/paper/2201.11981/) (2022)
- [A Framework for Learning from Demonstration with Minimal Human Effort](https://mlsanity.com/paper/2306.09211/) (2023)
- [Policy Stitching: Learning Transferable Robot Policies](https://mlsanity.com/paper/2309.13753/) (2023)
- [How Can Everyday Users Efficiently Teach Robots by Demonstrations?](https://mlsanity.com/paper/2310.13083/) (2023)
- ["Give Me an Example Like This": Episodic Active Reinforcement Learning from Demonstrations](https://mlsanity.com/paper/2406.03069/) (2024)

## Abstract (arXiv)

Transfer Learning (TL) is a powerful tool that enables robots to transfer learned policies across different environments, tasks, or embodiments. To further facilitate this process, efforts have been made to combine it with Learning from Demonstrations (LfD) for more flexible and efficient policy transfer. However, these approaches are almost exclusively limited to offline demonstrations collected before policy transfer starts, which may suffer from the intrinsic issue of covariance shift brought by LfD and harm the performance of policy transfer. Meanwhile, extensive work in the learning-from-scratch setting has shown that online demonstrations can effectively alleviate covariance shift and lead to better policy performance with improved sample efficiency. This work combines these insights to introduce online demonstrations into a policy transfer setting. We present Policy Transfer with Online Demonstrations, an active LfD algorithm for policy transfer that can optimize the timing and content of queries for online episodic expert demonstrations under a limited demonstration budget. We evaluate our method in eight robotic scenarios, involving policy transfer across diverse environment characteristics, task objectives, and robotic embodiments, with the aim to transfer a trained policy from a source task to a related but different target task. The results show that our method significantly outperforms all baselines in terms of average success rate and sample efficiency, compared to two canonical LfD methods with offline demonstrations and one active LfD method with online demonstrations. Additionally, we conduct preliminary sim-to-real tests of the transferred policy on three transfer scenarios in the real-world environment, demonstrating the policy effectiveness on a real robot manipulator.