# MULTIPOLAR: Multi-Source Policy Aggregation for Transfer Reinforcement Learning between Diverse Environmental Dynamics

arXiv: https://arxiv.org/abs/1909.13111 · HTML: https://mlsanity.com/paper/1909.13111/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Sep 2019.

## Built on by (most cited first, 18 of 18)

- [Residual Reinforcement Learning from Demonstrations](https://mlsanity.com/paper/2106.08050/) (2021)
- [Cross-Domain Policy Adaptation via Value-Guided Data Filtering](https://mlsanity.com/paper/2305.17625/) (2023)
- [Cross-Domain Policy Adaptation by Capturing Representation Mismatch](https://mlsanity.com/paper/2405.15369/) (2024)
- [ODRL: A Benchmark for Off-Dynamics Reinforcement Learning](https://mlsanity.com/paper/2410.20750/) (2024)
- [Residual Feedback Learning for Contact-Rich Manipulation Tasks with Uncertainty](https://mlsanity.com/paper/2106.04306/) (2021)
- [Efficient Residual Learning with Mixture-of-Experts for Universal Dexterous Grasping](https://mlsanity.com/paper/2410.02475/) (2024)
- [Procedural Content Generation: Better Benchmarks for Transfer Reinforcement Learning](https://mlsanity.com/paper/2105.14780/) (2021)
- [CUP: Critic-Guided Policy Reuse](https://mlsanity.com/paper/2210.08153/) (2022)
- [Multi-Source Transfer Learning for Deep Model-Based Reinforcement Learning](https://mlsanity.com/paper/2205.14410/) (2022)
- [Residual Policy Learning for Powertrain Control](https://mlsanity.com/paper/2212.07611/) (2022)
- [Towards Sample-Efficiency and Generalization of Transfer and Inverse Reinforcement Learning: A Comprehensive Literature Review](https://mlsanity.com/paper/2411.10268/) (2024)
- [Cultivated Wildness: Technodiversity and Wildness in Machines](https://mlsanity.com/paper/2305.02328/) (2023)
- [On the Value of Myopic Behavior in Policy Reuse](https://mlsanity.com/paper/2305.17623/) (2023)
- [IOB: Integrating Optimization Transfer and Behavior Transfer for Multi-Policy Reuse](https://mlsanity.com/paper/2308.07351/) (2023)
- [Efficient Real-World Autonomous Racing via Attenuated Residual Policy Optimization](https://mlsanity.com/paper/2603.12960/) (2026)
- [Fusing Interpretable Knowledge of Neural Network Learning Agents For Swarm-Guidance](https://mlsanity.com/paper/2204.00272/) (2022)
- [Adaptive Aggregation for Safety-Critical Control](https://mlsanity.com/paper/2302.03586/) (2023)
- [Shared learning of powertrain control policies for vehicle fleets](https://mlsanity.com/paper/2404.17892/) (2024)

## Abstract (arXiv)

Transfer reinforcement learning (RL) aims at improving the learning efficiency of an agent by exploiting knowledge from other source agents trained on relevant tasks. However, it remains challenging to transfer knowledge between different environmental dynamics without having access to the source environments. In this work, we explore a new challenge in transfer RL, where only a set of source policies collected under diverse unknown dynamics is available for learning a target task efficiently. To address this problem, the proposed approach, MULTI-source POLicy AggRegation (MULTIPOLAR), comprises two key techniques. We learn to aggregate the actions provided by the source policies adaptively to maximize the target task performance. Meanwhile, we learn an auxiliary network that predicts residuals around the aggregated actions, which ensures the target policy's expressiveness even when some of the source policies perform poorly. We demonstrated the effectiveness of MULTIPOLAR through an extensive experimental evaluation across six simulated environments ranging from classic control problems to challenging robotics simulations, under both continuous and discrete action spaces. The demo videos and code are available on the project webpage: https://omron-sinicx.github.io/multipolar/.