Uncertainty-aware transfer across tasks using hybrid model-based successor feature reinforcement learning
Not on the map. Why?
Abstract (arXiv)
Sample efficiency is central to developing practical reinforcement learning (RL) for complex and large-scale decision-making problems. The ability to transfer and generalize knowledge gained from previous experiences to downstream tasks can significantly improve sample efficiency. Recent research indicates that successor feature (SF) RL algorithms enable knowledge generalization between tasks with different rewards but identical transition dynamics. It has recently been hypothesized that combining model-based (MB) methods with SF algorithms can alleviate the limitation of fixed transition dynamics. Furthermore, uncertainty-aware exploration is widely recognized as another appealing approach for improving sample efficiency. Putting together two ideas of hybrid model-based successor feature (MB-SF) and uncertainty leads to an approach to the problem of sample efficient uncertainty-aware knowledge transfer across tasks with different transition dynamics or/and reward functions. In this paper, the uncertainty of the value of each action is approximated by a Kalman filter (KF)-based multiple-model adaptive estimation. This KF-based framework treats the parameters of a model as random variables. To the best of our knowledge, this is the first attempt at formulating a hybrid MB-SF algorithm capable of generalizing knowledge across large or continuous state space tasks with various transition dynamics while requiring less computation at decision time than MB methods. The number of samples required to learn the tasks was compared to recent SF and MB baselines. The results show that our algorithm generalizes its knowledge across different transition dynamics, learns downstream tasks with significantly fewer samples than starting from scratch, and outperforms existing approaches.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 79 of its 137 references that are in the map. First posted on arXiv 16 Oct 2023.
Builds on
79 of 137 references.
References · 79
- 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
- 2018World Modelscited by 1,513
- 2016Progressive Neural Networkscited by 1,361
- 2018Learning Latent Dynamics for Planning from Pixelscited by 1,264
- 2020Mastering Atari with Discrete World Modelscited by 1,007
- 2019An Introduction to Variational Autoencoderscited by 1,000
- 2020A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challengescited by 819
- 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
- 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 774
- 2021A Survey of Uncertainty in Deep Neural Networkscited by 680
- 2019A Simple Baseline for Bayesian Uncertainty in Deep Learningcited by 560
- 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 520
38 more
- 2016Successor Features for Transfer in Reinforcement Learningcited by 340
- 2018Large-Scale Study of Curiosity-Driven Learningcited by 280
- 2020First return, then explorecited by 266
- 2020Transfer Learning in Deep Reinforcement Learning: A Surveycited by 265
- 2017Count-Based Exploration with Neural Density Modelscited by 257
- 2020Model-Based Reinforcement Learning with Value-Targeted Regressioncited by 234
- 2018Randomized Prior Functions for Deep Reinforcement Learningcited by 219
- 2019Reinforcement Learning in Feature Space: Matrix Bandit, Kernels, and Regret Boundcited by 201
- 2020Learning Near Optimal Policies with Low Inherent Bellman Errorcited by 181
- 2018Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansioncited by 168
- 2019Fast Task Inference with Variational Intrinsic Successor Featurescited by 133
- 2021Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learningcited by 123
- 2014Model-based Reinforcement Learning and the Eluder Dimensioncited by 116
- 2021Learning One Representation to Optimize All Rewardscited by 115
- 2018Count-Based Exploration with the Successor Representationcited by 112
- 2018Model-Based Reinforcement Learning via Meta-Policy Optimizationcited by 101
- 2015Gaussian Processes for Data-Efficient Learning in Robotics and Controlcited by 98
- 2020Off-Dynamics Reinforcement Learning: Training for Transfer with Domain Classifierscited by 97
- 2020Model-Based Multi-Agent RL in Zero-Sum Markov Games with Near-Optimal Sample Complexitycited by 92
- 2020Fast active learning for pure exploration in reinforcement learningcited by 92
- 2019Transfer in Deep Reinforcement Learning Using Successor Features and Generalised Policy Improvementcited by 88
- 2018The Laplacian in RL: Learning Representations with Efficient Approximationscited by 86
- 2016Deep Successor Reinforcement Learningcited by 72
- 2019Estimating Risk and Uncertainty in Deep Reinforcement Learningcited by 70
- 2017Boltzmann Exploration Done Rightcited by 69
- 2018Efficient Exploration through Bayesian Deep Q-Networkscited by 67
- 2021The Option Keyboard: Combining Skills in Reinforcement Learningcited by 67
- 2019Distributional Reinforcement Learning for Efficient Explorationcited by 60
- 2016Deep Reinforcement Learning with Successor Features for Navigation across Similar Environmentscited by 50
- 2021Practical and Rigorous Uncertainty Bounds for Gaussian Process Regressioncited by 49
- 2021AdaRL: What, Where, and How to Adapt in Transfer Reinforcement Learningcited by 48
- 2018Pseudo-Rehearsal: Achieving Deep Reinforcement Learning without Catastrophic Forgettingcited by 46
- 2020Emergent Social Learning via Multi-agent Reinforcement Learningcited by 44
- 2022A Review of Uncertainty for Deep Reinforcement Learningcited by 40
- 2022Provable Benefits of Representational Transfer in Reinforcement Learningcited by 38
- 2018Successor Uncertainties: Exploration and Uncertainty in Temporal Difference Learningcited by 35
- 2021On the Power of Multitask Representation Learning in Linear MDPcited by 30
- 2019Tutorial: Deriving the Standard Variational Autoencoder (VAE) Loss Functioncited by 26
50 most cited of 79 All 79 (JSON)
Built on by
Citing papers per year: 2024: 1.
1 paper.