Robust Knowledge Transfer in Tiered Reinforcement Learning
Not on the map. Why?
Abstract (arXiv)
In this paper, we study the Tiered Reinforcement Learning setting, a parallel transfer learning framework, where the goal is to transfer knowledge from the low-tier (source) task to the high-tier (target) task to reduce the exploration risk of the latter while solving the two tasks in parallel. Unlike previous work, we do not assume the low-tier and high-tier tasks share the same dynamics or reward functions, and focus on robust knowledge transfer without prior knowledge on the task similarity. We identify a natural and necessary condition called the ``Optimal Value Dominance'' for our objective. Under this condition, we propose novel online learning algorithms such that, for the high-tier task, it can achieve constant regret on partial states depending on the task similarity and retain near-optimal regret when the two tasks are dissimilar, while for the low-tier task, it can keep near-optimal without making sacrifice. Moreover, we further study the setting with multiple low-tier tasks, and propose a novel transfer source selection mechanism, which can ensemble the information from all low-tier tasks and allow provable benefits on a much larger state-action space.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 24 of its 38 references that are in the map. First posted on arXiv 10 Feb 2023.
Builds on
24 of 38 references.
References · 24
- 2017Minimax Regret Bounds for Reinforcement Learningcited by 500
- 2018Is Q-learning Provably Efficient?cited by 485
- 2020Is Pessimism Provably Efficient for Offline RL?cited by 362
- 2021Bellman-consistent Pessimism for Offline Reinforcement Learningcited by 267
- 2020Transfer Learning in Deep Reinforcement Learning: A Surveycited by 259
- 2017Unifying PAC and Regret: Uniform PAC Bounds for Episodic Reinforcement Learningcited by 175
- 2021Multi-Task Reinforcement Learning with Context-based Representationscited by 169
- 2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coveragecited by 160
- 2021Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learningcited by 157
- 2022Offline Reinforcement Learning with Realizability and Single-policy Concentrabilitycited by 122
- 2019Non-Asymptotic Gap-Dependent Regret Bounds for Tabular MDPscited by 104
- 2022Unpacking Reward Shaping: Understanding the Benefits of Reward Engineering on Sample Complexitycited by 59
12 more
- 2013Sample Complexity of Multi-task Reinforcement Learningcited by 58
- 2021Fine-Grained Gap-Dependent Bounds for Tabular MDPs via Adaptive Multi-Step Bootstrapcited by 48
- 2021Near-optimal Representation Learning for Linear Bandits and Linear RLcited by 45
- 2022Leveraging Offline Data in Online Reinforcement Learningcited by 42
- 2022Offline Reinforcement Learning Under Value and Density-Ratio Realizability: The Power of Gapscited by 34
- 2021Beyond Value-Function Gaps: Improved Instance-Dependent Regret Bounds for Episodic Reinforcement Learningcited by 33
- 2022Towards Deployment-Efficient Reinforcement Learning: Lower Bound and Optimalitycited by 24
- 2021Provably Efficient Multi-Task Reinforcement Learning with Model Transfercited by 16
- 2021Can Q-Learning be Improved with Advice?cited by 13
- 2020Parallel Knowledge Transfer in Multi-Agent Reinforcement Learningcited by 6
- 2022Tiered Reinforcement Learning: Pessimism in the Face of Uncertainty and Constant Regretcited by 3
- 2021The Effect of Q-function Reuse on the Total Regret of Tabular, Model-Free, Reinforcement Learningcited by 2
Built on by
Citing papers per year: 2025: 1.
1 paper.