mlsanityData to 14 Oct 2026

Provable Benefits of Representational Transfer in Reinforcement Learning

arXiv 2205.14571 ↗First posted 29 May 2022cs.LG cs.AIcited by 38

Not on the map. Why?

Abstract (arXiv)

We study the problem of representational transfer in RL, where an agent first pretrains in a number of source tasks to discover a shared representation, which is subsequently used to learn a good policy in a target task. We propose a new notion of task relatedness between source and target tasks, and develop a novel approach for representational transfer under this assumption. Concretely, we show that given generative access to source tasks, we can discover a representation, using which subsequent linear RL techniques quickly converge to a near-optimal policy in the target task. The sample complexity is close to knowing the ground truth features in the target task, and comparable to prior representation learning results in the source tasks. We complement our positive results with lower bounds without generative access, and validate our findings with empirical evaluation on rich observation MDPs that require deep exploration. In our experiments, we observe a speed up in learning in the target by pre-training, and also validate the need for generative access in source tasks.

Key facts. Cited by 38 papers in the map (data through 14 Oct 2026). Builds on 42 of its 65 references that are in the map. First posted on arXiv 29 May 2022.

Builds on

42 of 65 references.

References · 42

  1. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  2. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,330
  3. 2016Successor Features for Transfer in Reinforcement Learningcited by 340
  4. 2016Contextual Decision Processes with Low Bellman Rank are PAC-Learnablecited by 292
  5. 2020Transfer Learning in Deep Reinforcement Learning: A Surveycited by 265
  6. 2020Model-Based Reinforcement Learning with Value-Targeted Regressioncited by 234
  7. 2020FLAMBE: Structural Complexity and Representation Learning of Low Rank MDPscited by 233
  8. 2020Few-Shot Learning via Learning the Representation, Provablycited by 214
  9. 2019Reinforcement Learning in Feature Space: Matrix Bandit, Kernels, and Regret Boundcited by 201
  10. 2015The Benefit of Multitask Representation Learningcited by 195
  11. 2020On the Theory of Transfer Learning: The Importance of Task Diversitycited by 186
  12. 2015Actor-Mimic: Deep Multitask and Transfer Reinforcement Learningcited by 178
30 more
  1. 2021Bilinear Classes: A Structural Framework for Provable Generalization in RLcited by 178
  2. 2020Reward-Free Exploration for Reinforcement Learningcited by 177
  3. 2019Provably efficient RL with Rich Observations via Latent State Decodingcited by 169
  4. 2021Representation Learning for Online and Offline RL in Low-rank MDPscited by 129
  5. 2021Provably Efficient Reinforcement Learning with Linear Function Approximation Under Adaptivity Constraintscited by 124
  6. 2021Learning One Representation to Optimize All Rewardscited by 115
  7. 2019Kinematic State Abstraction and Provably Efficient Rich-Observation Reinforcement Learningcited by 114
  8. 2019Sample Complexity of Reinforcement Learning using Linearly Combined Model Ensemblescited by 100
  9. 2020On Reward-Free Reinforcement Learning with Linear Function Approximationcited by 89
  10. 2021Model-free Representation Learning and Exploration in Low-rank MDPscited by 82
  11. 2019Corruption-robust exploration in episodic reinforcement learningcited by 72
  12. 2021Learning Successor States and Goal-Dependent Values: A Mathematical Viewpointcited by 70
  13. 2013Sample Complexity of Multi-task Reinforcement Learningcited by 58
  14. 2021Cautiously Optimistic Policy Optimization and Exploration with Linear Function Approximationcited by 53
  15. 2020Impact of Representation Learning in Linear Banditscited by 52
  16. 2022Efficient Reinforcement Learning in Block MDPs: A Model-free Representation Learning Approachcited by 51
  17. 2022Reward-Free RL is No Harder Than Reward-Aware RL in Linear Markov Decision Processescited by 49
  18. 2020Regret Bound Balancing and Elimination for Model Selection in Bandits and RLcited by 47
  19. 2021Near-optimal Representation Learning for Linear Bandits and Linear RLcited by 45
  20. 2020Provable Representation Learning for Imitation Learning via Bi-level Optimizationcited by 41
  21. 2013Sequential Transfer in Multi-armed Bandit with Finite Set of Modelscited by 38
  22. 2020Task-agnostic Exploration in Reinforcement Learningcited by 37
  23. 2021On the Power of Multitask Representation Learning in Linear MDPcited by 30
  24. 2020Online Model Selection for Reinforcement Learning with Function Approximationcited by 28
  25. 2020Online Sparse Reinforcement Learningcited by 25
  26. 2020Sparse Feature Selection Makes Batch Reinforcement Learning More Sample Efficientcited by 24
  27. 2021A Free Lunch from the Noise: Provable and Practical Exploration for Representation Learningcited by 24
  28. 2020Lipschitz Lifelong Reinforcement Learningcited by 23
  29. 2022Nearly Minimax Algorithms for Linear Bandits with Shared Representationcited by 16
  30. 2021Provably Efficient Representation Selection in Low-rank Markov Decision Processes: From Online to Offline RLcited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 5; 2023: 13; 2024: 11; 2025: 7; 2026: 2.

38 papers.

Sort:

By area:

  1. 2022Provable Benefit of Multitask Representation Learning in Reinforcement Learningcited by 29
  2. 2023The Benefits of Being Distributional: Small-Loss Bounds for Reinforcement Learningcited by 27
  3. 2024FuRL: Visual-Language Models as Fuzzy Rewards for Reinforcement Learningcited by 26
  4. 2023Provable Reward-Agnostic Preference-Based Reinforcement Learningcited by 25
  5. 2025Bigger, Regularized, Categorical: High-Capacity Value Functions are Efficient Multi-Task Learnerscited by 23
  6. 2024Overcoming the Sim-to-Real Gap: Leveraging Simulation to Learn to Explore for Real-World RLcited by 21
  7. 2022Pretraining in Deep Reinforcement Learning: A Surveycited by 18
  8. 2024Minimax Optimal and Computationally Efficient Algorithms for Distributionally Robust Offline Reinforcement Learningcited by 18
  9. 2022Understanding the Complexity Gains of Single-Task RL with a Curriculumcited by 17
  10. 2025A Survey of State Representation Learning for Deep Reinforcement Learningcited by 17
  11. 2023Efficient Model-Free Exploration in Low-Rank MDPscited by 16
  12. 2023Improved Sample Complexity for Reward-free Reinforcement Learning under Low-rank MDPscited by 14
26 more
  1. 2024Offline Multi-task Transfer RL with Representational Penalizationcited by 14
  2. 2024Offline Multitask Representation Learning for Reinforcement Learningcited by 14
  3. 2022Behavior Prior Representation learning for Offline Reinforcement Learningcited by 12
  4. 2023Offline Data Enhanced On-Policy Policy Gradient with Provable Guaranteescited by 11
  5. 2023Provably Efficient Representation Learning with Tractable Planning in Low-Rank POMDPcited by 8
  6. 2023On the Model-Misspecification in Reinforcement Learningcited by 7
  7. 2023Multi-task Representation Learning for Pure Exploration in Linear Banditscited by 6
  8. 2024Skill Transfer and Discovery for Sim-to-Real Learning: A Representation-Based Viewpointcited by 6
  9. 2022Joint Representation Training in Sequential Tasks with Shared Structurecited by 5
  10. 2025Learning Policy Committees for Effective Personalization in MDPs with Diverse Taskscited by 4
  11. 2023Provably Efficient Offline Reinforcement Learning with Perturbed Data Sourcescited by 3
  12. 2024Efficient and Sharp Off-Policy Evaluation in Robust Markov Decision Processescited by 3
  13. 2024Transfer Learning for Latent Variable Network Modelscited by 3
  14. 2023Provably Efficient Algorithm for Nonstationary Low-Rank MDPscited by 2
  15. 2024Sample Efficient Myopic Exploration Through Multitask Reinforcement Learning with Diverse Taskscited by 2
  16. 2025Optimal Transfer Learning for Missing Not-at-Random Matrix Completioncited by 2
  17. 2025Pessimism Principle Can Be Effective: Towards a Framework for Zero-Shot Transfer Reinforcement Learningcited by 2
  18. 2023JoinGym: An Efficient Query Optimization Environment for Reinforcement Learningcited by 1
  19. 2023Uncertainty-aware transfer across tasks using hybrid model-based successor feature reinforcement learningcited by 1
  20. 2024Open Problem: Active Representation Learningcited by 1
  21. 2024The Limits of Transfer Reinforcement Learning with Latent Low-rank Structurecited by 1
  22. 2025Towards Understanding the Benefit of Multitask Representation Learning in Decision Processcited by 1
  23. 2026One-Step Bellman Alignment Enables Provably Efficient Transfer in Online RLcited by 1
  24. 2023Provable Benefits of Multi-task RL under Non-Markovian Decision Making Processescited by 0
  25. 2025Provably Efficient Reward Transfer in Reinforcement Learning with Discrete Markov Decision Processescited by 0
  26. 2026Robust Transfer Learning with Side Informationcited by 0