# Flexible Attention-Based Multi-Policy Fusion for Efficient Deep Reinforcement Learning

arXiv: https://arxiv.org/abs/2210.03729 · HTML: https://mlsanity.com/paper/2210.03729/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 48 references that are in the map.
- First posted on arXiv 7 Oct 2022.

## Builds on (most cited first, 28 of 28)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation](https://mlsanity.com/paper/1806.10293/) (2018)
- [Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations](https://mlsanity.com/paper/1709.10087/) (2017)
- [Reinforcement Learning with Deep Energy-Based Policies](https://mlsanity.com/paper/1702.08165/) (2017)
- [The Option-Critic Architecture](https://mlsanity.com/paper/1609.05140/) (2016)
- [Overcoming Exploration in Reinforcement Learning with Demonstrations](https://mlsanity.com/paper/1709.10089/) (2017)
- [Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards](https://mlsanity.com/paper/1707.08817/) (2017)
- [A Lyapunov-based Approach to Safe Reinforcement Learning](https://mlsanity.com/paper/1805.07708/) (2018)
- [End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Tasks](https://mlsanity.com/paper/1903.08792/) (2019)
- [Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation](https://mlsanity.com/paper/1604.06057/) (2016)
- [Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research](https://mlsanity.com/paper/1802.09464/) (2018)
- [Language as an Abstraction for Hierarchical Deep Reinforcement Learning](https://mlsanity.com/paper/1906.07343/) (2019)
- [Provably Efficient Safe Exploration via Primal-Dual Policy Optimization](https://mlsanity.com/paper/2003.00534/) (2020)
- [Near-Optimal Representation Learning for Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1810.01257/) (2018)
- [Reincarnating Reinforcement Learning: Reusing Prior Computation to Accelerate Progress](https://mlsanity.com/paper/2206.01626/) (2022)
- [Safe Exploration of State and Action Spaces in Reinforcement Learning](https://mlsanity.com/paper/1402.0560/) (2014)
- [Reinforced Imitation: Sample Efficient Deep Reinforcement Learning for Map-less Navigation by Leveraging Prior Demonstrations](https://mlsanity.com/paper/1805.07095/) (2018)
- [Deep Q-learning from Demonstrations](https://mlsanity.com/paper/1704.03732/) (2017)
- [Unsupervised Skill Discovery with Bottleneck Option Learning](https://mlsanity.com/paper/2106.14305/) (2021)
- [Composing Task-Agnostic Policies with Deep Reinforcement Learning](https://mlsanity.com/paper/1905.10681/) (2019)
- [Attend, Adapt and Transfer: Attentive Deep Architecture for Adaptive Transfer from multiple sources in the same domain](https://mlsanity.com/paper/1510.02879/) (2015)
- [A Low-Cost Ethics Shaping Approach for Designing Reinforcement Learning Agents](https://mlsanity.com/paper/1712.04172/) (2017)
- [Reinforcement Learning Under Moral Uncertainty](https://mlsanity.com/paper/2006.04734/) (2020)
- [KoGuN: Accelerating Deep Reinforcement Learning via Integrating Human Suboptimal Knowledge](https://mlsanity.com/paper/2002.07418/) (2020)
- [Toward Robust Long Range Policy Transfer](https://mlsanity.com/paper/2103.02957/) (2021)

## Built on by (most cited first, 5 of 5)

- [Improving Sample Efficiency of Reinforcement Learning with Background Knowledge from Large Language Models](https://mlsanity.com/paper/2407.03964/) (2024)
- [Heterogeneous Knowledge for Augmented Modular Reinforcement Learning](https://mlsanity.com/paper/2306.01158/) (2023)
- [CARoL: Context-aware Adaptation for Robot Learning](https://mlsanity.com/paper/2506.07006/) (2025)
- [SurgIRL: Towards Life-Long Learning for Surgical Automation by Incremental Reinforcement Learning](https://mlsanity.com/paper/2409.15651/) (2024)
- [Conceptual Belief-Informed Reinforcement Learning](https://mlsanity.com/paper/2410.01739/) (2024)

## Abstract (arXiv)

Reinforcement learning (RL) agents have long sought to approach the efficiency of human learning. Humans are great observers who can learn by aggregating external knowledge from various sources, including observations from others' policies of attempting a task. Prior studies in RL have incorporated external knowledge policies to help agents improve sample efficiency. However, it remains non-trivial to perform arbitrary combinations and replacements of those policies, an essential feature for generalization and transferability. In this work, we present Knowledge-Grounded RL (KGRL), an RL paradigm fusing multiple knowledge policies and aiming for human-like efficiency and flexibility. We propose a new actor architecture for KGRL, Knowledge-Inclusive Attention Network (KIAN), which allows free knowledge rearrangement due to embedding-based attentive action prediction. KIAN also addresses entropy imbalance, a problem arising in maximum entropy KGRL that hinders an agent from efficiently exploring the environment, through a new design of policy distributions. The experimental results demonstrate that KIAN outperforms alternative methods incorporating external knowledge policies and achieves efficient and flexible learning. Our implementation is available at https://github.com/Pascalson/KGRL.git