mlsanityData to 14 Oct 2026

Generalisation in Multitask Fitted Q-Iteration and Offline Q-learning

arXiv 2512.20220 ↗First posted 23 Dec 2025cs.LGcited by 0

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

We study offline multitask reinforcement learning in settings where multiple tasks share a low-rank representation of their action-value functions. In this regime, a learner is provided with fixed datasets collected from several related tasks, without access to further online interaction, and seeks to exploit shared structure to improve statistical efficiency and generalization. We analyze a multitask variant of fitted Q-iteration that jointly learns a shared representation and task-specific value functions via Bellman error minimization on offline data. Under standard realizability and coverage assumptions commonly used in offline reinforcement learning, we establish finite-sample generalization guarantees for the learned value functions. Our analysis explicitly characterizes how pooling data across tasks improves estimation accuracy, yielding a 1/√nT dependence on the total number of samples across tasks, while retaining the usual dependence on the horizon and concentrability coefficients arising from distribution shift. In addition, we consider a downstream offline setting in which a new task shares the same underlying representation as the upstream tasks. We study how reusing the representation learned during the multitask phase affects value estimation for this new task, and show that it can reduce the effective complexity of downstream learning relative to learning from scratch. Together, our results clarify the role of shared representations in multitask offline Q-learning and provide theoretical insight into when and how multitask structure can improve generalization in model-free, value-based reinforcement learning.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 19 of its 32 references that are in the map. First posted on arXiv 23 Dec 2025.

Builds on

19 of 32 references.

References · 19

  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  2. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  3. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  4. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  5. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 287
  6. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimismcited by 279
  7. 2020FLAMBE: Structural Complexity and Representation Learning of Low Rank MDPscited by 233
  8. 2019On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shiftcited by 177
  9. 2019Provably efficient RL with Rich Observations via Latent State Decodingcited by 169
  10. 2022Adversarially Trained Actor Critic for Offline Reinforcement Learningcited by 111
  11. 2022Offline Q-Learning on Diverse Multi-Task Data Both Scales And Generalizescited by 73
  12. 2021Conservative Data Sharing for Multi-Task Offline Reinforcement Learningcited by 64
7 more
  1. 2022Provable Benefit of Multitask Representation Learning in Reinforcement Learningcited by 29
  2. 2020Kernel-Based Reinforcement Learning: A Finite-Time Analysiscited by 23
  3. 2016Learning Shared Representations in Multi-task Reinforcement Learningcited by 15
  4. 2024Offline Multitask Representation Learning for Reinforcement Learningcited by 14
  5. 2023On Representation Complexity of Model-based and Model-free Reinforcement Learningcited by 6
  6. 2023Low-Rank MDPs with Continuous Action Spacescited by 4
  7. 2025Goal-Oriented Skill Abstraction for Offline Multi-Task Reinforcement Learningcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Skills Regularized Task Decomposition for Multi-task Offline Reinforcement Learning
  2. 2024Offline Multi-task Transfer RL with Representational Penalization
  3. 2021Offline Reinforcement Learning: Fundamental Barriers for Value Function Approximation
  4. 2025In-Context Compositional Q-Learning for Offline Reinforcement Learning
  5. 2024Ensemble Successor Representations for Task Generalization in Offline-to-Online Reinforcement Learning
  6. 2024Pessimistic Value Iteration for Multi-Task Data Sharing in Offline Reinforcement Learning
  7. 2024On Sample-Efficient Offline Reinforcement Learning: Data Diversity, Posterior Sampling, and Beyond
  8. 2022Offline Reinforcement Learning at Multiple Frequencies