# Bayesian Controller Fusion: Leveraging Control Priors in Deep Reinforcement Learning for Robotics

arXiv: https://arxiv.org/abs/2107.09822 · HTML: https://mlsanity.com/paper/2107.09822/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 71 references that are in the map.
- First posted on arXiv 21 Jul 2021.

## Builds on (most cited first, 30 of 31)

- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [Soft Actor-Critic Algorithms and Applications](https://mlsanity.com/paper/1812.05905/) (2018)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction](https://mlsanity.com/paper/1906.00949/) (2019)
- [Learning Dexterous In-Hand Manipulation](https://mlsanity.com/paper/1808.00177/) (2018)
- [On Evaluation of Embodied Navigation Agents](https://mlsanity.com/paper/1807.06757/) (2018)
- [Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards](https://mlsanity.com/paper/1707.08817/) (2017)
- [Residual Reinforcement Learning for Robot Control](https://mlsanity.com/paper/1812.03201/) (2018)
- [Distral: Robust Multitask Reinforcement Learning](https://mlsanity.com/paper/1707.04175/) (2017)
- [Accelerating Reinforcement Learning with Learned Skill Priors](https://mlsanity.com/paper/2010.11944/) (2020)
- [How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learned](https://mlsanity.com/paper/2102.02915/) (2021)
- [Residual Policy Learning](https://mlsanity.com/paper/1812.06298/) (2018)
- [Learning to Walk via Deep Reinforcement Learning](https://mlsanity.com/paper/1812.11103/) (2018)
- [Deep Exploration via Randomized Value Functions](https://mlsanity.com/paper/1703.07608/) (2017)
- [Behavior Trees in Robotics and AI: An Introduction](https://mlsanity.com/paper/1709.00084/) (2017)
- [Deep Reinforcement Learning for Industrial Insertion Tasks with Visual Inputs and Natural Rewards](https://mlsanity.com/paper/1906.05841/) (2019)
- [Information asymmetry in KL-regularized RL](https://mlsanity.com/paper/1905.01240/) (2019)
- [Policies Modulating Trajectory Generators](https://mlsanity.com/paper/1910.02812/) (2019)
- [PyRep: Bringing V-REP to Deep Robot Learning](https://mlsanity.com/paper/1906.11176/) (2019)
- [Control Regularization for Reduced Variance Reinforcement Learning](https://mlsanity.com/paper/1905.05380/) (2019)
- [Deep Predictive Policy Training using Reinforcement Learning](https://mlsanity.com/paper/1703.00727/) (2017)
- [Reinforcement Learning from Imperfect Demonstrations under Soft Expert Guidance](https://mlsanity.com/paper/1911.07109/) (2019)
- [Guided Uncertainty-Aware Policy Optimization: Combining Learning and Model-Based Strategies for Sample-Efficient Policy Learning](https://mlsanity.com/paper/2005.10872/) (2020)
- [Learning Dexterous Manipulation from Suboptimal Experts](https://mlsanity.com/paper/2010.08587/) (2020)
- [Learning with Training Wheels: Speeding up Training with a Simple Controller for Deep Reinforcement Learning](https://mlsanity.com/paper/1812.05027/) (2018)
- [Exploiting Hierarchy for Learning and Transfer in KL-regularized RL](https://mlsanity.com/paper/1903.07438/) (2019)
- [Residual Reactive Navigation: Combining Classical and Learned Navigation Strategies For Deployment in Unknown Environments](https://mlsanity.com/paper/1909.10972/) (2019)

## Built on by (most cited first, 20 of 20)

- [Residual Skill Policies: Learning an Adaptable Skill-based Action Space for Reinforcement Learning for Robotics](https://mlsanity.com/paper/2211.02231/) (2022)
- [Robot Learning of Mobile Manipulation with Reachability Behavior Priors](https://mlsanity.com/paper/2203.04051/) (2022)
- [Physics-Regulated Deep Reinforcement Learning: Invariant Embeddings](https://mlsanity.com/paper/2305.16614/) (2023)
- [SkillS: Adaptive Skill Sequencing for Efficient Temporally-Extended Exploration](https://mlsanity.com/paper/2211.13743/) (2022)
- [Contextualized Hybrid Ensemble Q-learning: Learning Fast with Control Priors](https://mlsanity.com/paper/2406.19768/) (2024)
- [Renaissance Robot: Optimal Transport Policy Fusion for Learning Diverse Skills](https://mlsanity.com/paper/2207.00978/) (2022)
- [Simplex-enabled Safe Continual Learning Machine](https://mlsanity.com/paper/2409.05898/) (2024)
- [Verification of safety critical control policies using kernel methods](https://mlsanity.com/paper/2203.12407/) (2022)
- [Learning to Drive Using Sparse Imitation Reinforcement Learning](https://mlsanity.com/paper/2205.12128/) (2022)
- [Bayesian inference for data-efficient, explainable, and safe robotic motion planning: A review](https://mlsanity.com/paper/2307.08024/) (2023)
- [CHEQ-ing the Box: Safe Variable Impedance Learning for Robotic Polishing](https://mlsanity.com/paper/2501.07985/) (2025)
- [Combining Bayesian Inference and Reinforcement Learning for Agent Decision Making: A Review](https://mlsanity.com/paper/2505.07911/) (2025)
- [Strangeness-driven Exploration in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2212.13448/) (2022)
- [Robotic Packaging Optimization with Reinforcement Learning](https://mlsanity.com/paper/2303.14693/) (2023)
- [Learning Multimodal Confidence for Intention Recognition in Human-Robot Interaction](https://mlsanity.com/paper/2405.14116/) (2024)
- [Physics-model-guided Worst-case Sampling for Safe Reinforcement Learning](https://mlsanity.com/paper/2412.13224/) (2024)
- [Real-DRL: Teach and Learn in Reality](https://mlsanity.com/paper/2511.00112/) (2025)
- [Safe Online Learning via Smooth Safety-Structured Policy Composition](https://mlsanity.com/paper/2606.31320/) (2026)
- [Cognitively-Grounded On-Device Runtime Learning for Ground Robots in Unknown Physical Environments](https://mlsanity.com/paper/2608.28677/) (2026)
- [Adaptive Expert Guidance for Efficient On-Policy Reinforcement Learning](https://mlsanity.com/paper/2610.06019/) (2026)

## Abstract (arXiv)

We present Bayesian Controller Fusion (BCF): a hybrid control strategy that combines the strengths of traditional hand-crafted controllers and model-free deep reinforcement learning (RL). BCF thrives in the robotics domain, where reliable but suboptimal control priors exist for many tasks, but RL from scratch remains unsafe and data-inefficient. By fusing uncertainty-aware distributional outputs from each system, BCF arbitrates control between them, exploiting their respective strengths. We study BCF on two real-world robotics tasks involving navigation in a vast and long-horizon environment, and a complex reaching task that involves manipulability maximisation. For both these domains, simple handcrafted controllers exist that can solve the task at hand in a risk-averse manner but do not necessarily exhibit the optimal solution given limitations in analytical modelling, controller miscalibration and task variation. As exploration is naturally guided by the prior in the early stages of training, BCF accelerates learning, while substantially improving beyond the performance of the control prior, as the policy gains more experience. More importantly, given the risk-aversity of the control prior, BCF ensures safe exploration and deployment, where the control prior naturally dominates the action distribution in states unknown to the policy. We additionally show BCF's applicability to the zero-shot sim-to-real setting and its ability to deal with out-of-distribution states in the real world. BCF is a promising approach towards combining the complementary strengths of deep RL and traditional robotic control, surpassing what either can achieve independently. The code and supplementary video material are made publicly available at https://krishanrana.github.io/bcf.