mlsanityData to 7 Oct 2026

Offline Multi-task Transfer RL with Representational Penalization

arXiv 2402.12570 ↗First posted 19 Feb 2024cs.LG cs.AIcited by 13

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

We study the problem of representation transfer in offline Reinforcement Learning (RL), where a learner has access to episodic data from a number of source tasks collected a priori, and aims to learn a shared representation to be used in finding a good policy for a target task. Unlike in online RL where the agent interacts with the environment while learning a policy, in the offline setting there cannot be such interactions in either the source tasks or the target task; thus multi-task offline RL can suffer from incomplete coverage. We propose an algorithm to compute pointwise uncertainty measures for the learnt representation, and establish a data-dependent upper bound for the suboptimality of the learnt policy for the target task. Our algorithm leverages the collective exploration done by source tasks to mitigate poor coverage at some points by a few tasks, thus overcoming the limitation of needing uniformly good coverage for a meaningful transfer by existing offline algorithms. We complement our theoretical results with empirical evaluation on a rich-observation MDP which requires many samples for complete coverage. Our findings illustrate the benefits of penalizing and quantifying the uncertainty in the learnt representation.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 56 of its 78 references that are in the map. First posted on arXiv 19 Feb 2024.

Builds on

56 of 78 references.

Closest ancestors · 8

Same direction, newest first

  1. 2021Representation Learning for Online and Offline RL in Low-rank MDPs9 Oct
  2. 2020Is Pessimism Provably Efficient for Offline RL?30 Dec
  3. 2020What are the Statistical Limits of Offline RL with Linear Function Approximation?22 Oct
  4. 2020On Reward-Free Reinforcement Learning with Linear Function Approximation19 Jun
  5. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  6. 2020MOPO: Model-based Offline Policy Optimization27 May
  7. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  8. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May

Other references · 48

  1. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  2. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,436
  3. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,313
  4. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  5. 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 514
  6. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 337
  7. 2016Safe, Multi-Agent, Reinforcement Learning for Autonomous Drivingcited by 301
  8. 2016Contextual Decision Processes with Low Bellman Rank are PAC-Learnablecited by 290
  9. 2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 285
  10. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 285
  11. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  12. 2020Model-Based Reinforcement Learning with Value-Targeted Regressioncited by 233
36 more
  1. 2020FLAMBE: Structural Complexity and Representation Learning of Low Rank MDPscited by 232
  2. 2018Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimationcited by 226
  3. 2020Few-Shot Learning via Learning the Representation, Provablycited by 213
  4. 2019A Theoretical Analysis of Deep Q-Learningcited by 212
  5. 2020Reward-Free Exploration for Reinforcement Learningcited by 177
  6. 2019Provably efficient RL with Rich Observations via Latent State Decodingcited by 169
  7. 2020Provable Meta-Learning of Linear Representationscited by 161
  8. 2019Neural Policy Gradient Methods: Global Optimality and Rates of Convergencecited by 157
  9. 2018More Robust Doubly Robust Off-policy Evaluationcited by 141
  10. 2020GenDICE: Generalized Offline Estimation of Stationary Valuescited by 122
  11. 2019Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Samplingcited by 118
  12. 2019Kinematic State Abstraction and Provably Efficient Rich-Observation Reinforcement Learningcited by 114
  13. 2017A Tutorial on Kernel Density Estimation and Recent Advancescited by 100
  14. 2019Sample Complexity of Reinforcement Learning using Linearly Combined Model Ensemblescited by 100
  15. 2020Reinforcement Learning via Fenchel-Rockafellar Dualitycited by 100
  16. 2020The Importance of Pessimism in Fixed-Dataset Policy Optimizationcited by 99
  17. 2020Offline Meta-Reinforcement Learning with Advantage Weightingcited by 92
  18. 2020Off-Policy Evaluation via the Regularized Lagrangiancited by 86
  19. 2021Model-free Representation Learning and Exploration in Low-rank MDPscited by 82
  20. 2022Pre-Training for Robots: Offline RL Enables Learning New Tasks from a Handful of Trialscited by 78
  21. 2019Efficiently Breaking the Curse of Horizon in Off-Policy Evaluation with Double Reinforcement Learningcited by 75
  22. 2020Batch Policy Learning in Average Reward Markov Decision Processescited by 69
  23. 2020Asymptotically Efficient Off-Policy Evaluation for Tabular Reinforcement Learningcited by 60
  24. 2020Impact of Representation Learning in Linear Banditscited by 51
  25. 2019Doubly Robust Bias Reduction in Infinite Horizon Off-Policy Estimationcited by 48
  26. 2021Near-optimal Representation Learning for Linear Bandits and Linear RLcited by 45
  27. 2022Provable Benefits of Representational Transfer in Reinforcement Learningcited by 38
  28. 2020Task-agnostic Exploration in Reinforcement Learningcited by 37
  29. 2022Provable Benefit of Multitask Representation Learning in Reinforcement Learningcited by 29
  30. 2020Knowledge Transfer in Multi-Task Deep Reinforcement Learning for Continuous Controlcited by 27
  31. 2024An Experimental Design Framework for Label-Efficient Supervised Finetuning of Large Language Modelscited by 22
  32. 2022Towards Sample-efficient Overparameterized Meta-learningcited by 21
  33. 2022Nearly Minimax Algorithms for Linear Bandits with Shared Representationcited by 16
  34. 2022Active Multi-Task Representation Learningcited by 13
  35. 2022Provable General Function Class Representation Learning in Multitask Bandits and MDPscited by 12
  36. 2021Conditional Expectation based Value Decomposition for Scalable On-Demand Ride Poolingcited by 1

Built on by

Cited by year

Citing papers per year: 2024: 9; 2025: 4.

13 papers.

Sort:

By area:

  1. 2024RLHF from Heterogeneous Feedback via Personalization and Preference Aggregationcited by 71
  2. 2025LoRe: Personalizing LLMs via Low-Rank Reward Modelingcited by 34
  3. 2024Provable Pluralistic Alignment: Multi-Party RLHF under Offline Human Feedbackcited by 31
  4. 2024ASID: Active Exploration for System Identification in Robotic Manipulationcited by 27
  5. 2024Doubly Mild Generalization for Offline Reinforcement Learningcited by 21
  6. 2024Contrastive Representation for Data Filtering in Cross-Domain Offline Reinforcement Learningcited by 18
  7. 2024Minimax Optimal and Computationally Efficient Algorithms for Distributionally Robust Offline Reinforcement Learningcited by 17
  8. 2024Offline Multitask Representation Learning for Reinforcement Learningcited by 14
  9. 2025Learning Generalizable Skills from Offline Multi-Task Data for Multi-Agent Cooperationcited by 8
  10. 2024Sample Complexity Reduction via Policy Difference Estimation in Tabular Reinforcement Learningcited by 3
  11. 2025Provable Zero-Shot Generalization in Offline Reinforcement Learningcited by 3
  12. 2024The Power of Active Multi-Task Learning in Reinforcement Learning from Human Feedbackcited by 1
1 more
  1. 2025Energy-Based Transfer for Reinforcement Learningcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2021Conservative Data Sharing for Multi-Task Offline Reinforcement Learning
  2. 2021Representation Matters: Offline Pretraining for Sequential Decision Making
  3. 2024Pessimistic Value Iteration for Multi-Task Data Sharing in Offline Reinforcement Learning
  4. 2024Ensemble Successor Representations for Task Generalization in Offline-to-Online Reinforcement Learning
  5. 2022Leveraging Offline Data in Online Reinforcement Learning
  6. 2024Skills Regularized Task Decomposition for Multi-task Offline Reinforcement Learning
  7. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism
  8. 2025Evaluation-Time Policy Switching for Offline Reinforcement Learning