# Efficient Bayesian Policy Reuse with a Scalable Observation Model in Deep Reinforcement Learning

arXiv: https://arxiv.org/abs/2204.07729 · HTML: https://mlsanity.com/paper/2204.07729/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 65 references that are in the map.
- First posted on arXiv 16 Apr 2022.

## Builds on (most cited first, 27 of 27)

- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Deep Reinforcement Learning that Matters](https://mlsanity.com/paper/1709.06560/) (2017)
- [Learning agile and dynamic motor skills for legged robots](https://mlsanity.com/paper/1901.08652/) (2019)
- [Benchmarking Deep Reinforcement Learning for Continuous Control](https://mlsanity.com/paper/1604.06778/) (2016)
- [Safe Model-based Reinforcement Learning with Stability Guarantees](https://mlsanity.com/paper/1705.08551/) (2017)
- [Successor Features for Transfer in Reinforcement Learning](https://mlsanity.com/paper/1606.05312/) (2016)
- [Improving Sample Efficiency in Model-Free Reinforcement Learning from Images](https://mlsanity.com/paper/1910.01741/) (2019)
- [Actor-Mimic: Deep Multitask and Transfer Reinforcement Learning](https://mlsanity.com/paper/1511.06342/) (2015)
- [Offline Meta-Reinforcement Learning with Advantage Weighting](https://mlsanity.com/paper/2008.06043/) (2020)
- [Learning Invariant Feature Spaces to Transfer Skills with Reinforcement Learning](https://mlsanity.com/paper/1703.02949/) (2017)
- [Hidden Parameter Markov Decision Processes: A Semiparametric Regression Approach for Discovering Latent Task Parametrizations](https://mlsanity.com/paper/1308.3513/) (2013)
- [Efficient Deep Reinforcement Learning with Imitative Expert Priors for Autonomous Driving](https://mlsanity.com/paper/2103.10690/) (2021)
- [Deep Reinforcement Learning based Automatic Exploration for Navigation in Unknown Environment](https://mlsanity.com/paper/2007.11808/) (2020)
- [Knowledge Transfer in Multi-Task Deep Reinforcement Learning for Continuous Control](https://mlsanity.com/paper/2010.07494/) (2020)
- [Lifelong Incremental Reinforcement Learning with Online Bayesian Inference](https://mlsanity.com/paper/2007.14196/) (2020)
- [Deep Reinforcement Learning with Quantum-inspired Experience Replay](https://mlsanity.com/paper/2101.02034/) (2021)
- [Single Episode Policy Transfer in Reinforcement Learning](https://mlsanity.com/paper/1910.07719/) (2019)
- [Context-Aware Policy Reuse](https://mlsanity.com/paper/1806.03793/) (2018)
- [Evolutionary Reinforcement Learning for Sample-Efficient Multiagent Coordination](https://mlsanity.com/paper/1906.07315/) (2019)
- [REPAINT: Knowledge Transfer in Deep Reinforcement Learning](https://mlsanity.com/paper/2011.11827/) (2020)
- [An Optimal Online Method of Selecting Source Policies for Reinforcement Learning](https://mlsanity.com/paper/1709.08201/) (2017)
- [A New Representation of Successor Features for Transfer across Dissimilar Environments](https://mlsanity.com/paper/2107.08426/) (2021)
- [Bayesian Policy Reuse](https://mlsanity.com/paper/1505.00284/) (2015)
- [Data-driven Dynamic Multi-objective Optimal Control: An Aspiration-satisfying Reinforcement Learning Approach](https://mlsanity.com/paper/2005.07118/) (2020)
- [Assured Learning-enabled Autonomy: A Metacognitive Reinforcement Learning Framework](https://mlsanity.com/paper/2103.12558/) (2021)
- [Protective Policy Transfer](https://mlsanity.com/paper/2012.06662/) (2020)

## Built on by (most cited first, 2 of 2)

- [Continual Offline Reinforcement Learning via Diffusion-based Dual Generative Replay](https://mlsanity.com/paper/2404.10662/) (2024)
- [Combining Trained Models in Reinforcement Learning](https://mlsanity.com/paper/2605.02159/) (2026)

## Abstract (arXiv)

Bayesian policy reuse (BPR) is a general policy transfer framework for selecting a source policy from an offline library by inferring the task belief based on some observation signals and a trained observation model. In this paper, we propose an improved BPR method to achieve more efficient policy transfer in deep reinforcement learning (DRL). First, most BPR algorithms use the episodic return as the observation signal that contains limited information and cannot be obtained until the end of an episode. Instead, we employ the state transition sample, which is informative and instantaneous, as the observation signal for faster and more accurate task inference. Second, BPR algorithms usually require numerous samples to estimate the probability distribution of the tabular-based observation model, which may be expensive and even infeasible to learn and maintain, especially when using the state transition sample as the signal. Hence, we propose a scalable observation model based on fitting state transition functions of source tasks from only a small number of samples, which can generalize to any signals observed in the target task. Moreover, we extend the offline-mode BPR to the continual learning setting by expanding the scalable observation model in a plug-and-play fashion, which can avoid negative transfer when faced with new unknown tasks. Experimental results show that our method can consistently facilitate faster and more efficient policy transfer.