mlsanityData to 14 Oct 2026

Pessimism Principle Can Be Effective: Towards a Framework for Zero-Shot Transfer Reinforcement Learning

arXiv 2505.18447 ↗First posted 24 May 2025cs.LGcited by 2

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Transfer reinforcement learning aims to derive a near-optimal policy for a target environment with limited data by leveraging abundant data from related source domains. However, it faces two key challenges: the lack of performance guarantees for the transferred policy, which can lead to undesired actions, and the risk of negative transfer when multiple source domains are involved. We propose a novel framework based on the pessimism principle, which constructs and optimizes a conservative estimation of the target domain's performance. Our framework effectively addresses the two challenges by providing an optimized lower bound on target performance, ensuring safe and reliable decisions, and by exhibiting monotonic improvement with respect to the quality of the source domains, thereby avoiding negative transfer. We construct two types of conservative estimations, rigorously characterize their effectiveness, and develop efficient distributed algorithms with convergence guarantees. Our framework provides a theoretically sound and practically robust solution for transfer learning in reinforcement learning.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 67 of its 100 references that are in the map. First posted on arXiv 24 May 2025.

Builds on

67 of 100 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Hybrid Transfer Reinforcement Learning: Provable Sample Efficiency from Shifted-Dynamics Data6 Nov
  2. 2024The Limits of Transfer Reinforcement Learning with Latent Low-rank Structure28 Oct

Other references · 65

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  2. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  3. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,498
  4. 2019Federated Learning with Personalization Layerscited by 541
  5. 2021Towards Personalized Federated Learningcited by 464
  6. 2015Policy Distillationcited by 440
  7. 2020Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Surveycited by 413
  8. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 396
  9. 2020Is Pessimism Provably Efficient for Offline RL?cited by 364
  10. 2020Adaptive Personalized Federated Learningcited by 358
  11. 2020Personalized Federated Learning: A Meta-Learning Approachcited by 312
  12. 2017Distral: Robust Multitask Reinforcement Learningcited by 233
38 more
  1. 2015Contextual Markov Decision Processescited by 220
  2. 2019Action Robust Reinforcement Learning and Applications in Continuous Controlcited by 174
  3. 2021Multi-Task Reinforcement Learning with Context-based Representationscited by 174
  4. 2019Active Domain Randomizationcited by 170
  5. 2021A Survey of Zero-shot Generalisation in Deep Reinforcement Learningcited by 170
  6. 2023A Survey of Imitation Learning: Algorithms, Recent Developments, and Challengescited by 148
  7. 2020Multi-Task Reinforcement Learning with Soft Modularizationcited by 143
  8. 2021Online Robust Reinforcement Learning with Model Uncertaintycited by 96
  9. 2022Robust Reinforcement Learning using Offline Datacited by 96
  10. 2022Pessimistic Q-Learning for Offline Reinforcement Learning: Towards Optimal Sample Complexitycited by 85
  11. 2022RORL: Robust Offline Reinforcement Learning via Conservative Smoothingcited by 84
  12. 2021Sample Complexity of Robust Reinforcement Learning with a Generative Modelcited by 83
  13. 2021Understanding Domain Randomization for Sim-to-real Transfercited by 80
  14. 2022Policy Gradient Method For Robust Reinforcement Learningcited by 69
  15. 2022Federated Reinforcement Learning with Environment Heterogeneitycited by 63
  16. 2017Markov Decision Processes with Continuous Side Informationcited by 56
  17. 2023The Curious Price of Distributional Robustness in Reinforcement Learning with a Generative Modelcited by 52
  18. 2021Twice regularized MDPs and the equivalence between robustness and regularizationcited by 47
  19. 2022Contextualize Me -- The Case for Context in Reinforcement Learningcited by 46
  20. 2023Improved Sample Complexity Bounds for Distributionally Robust Reinforcement Learningcited by 44
  21. 2022Provable Benefits of Representational Transfer in Reinforcement Learningcited by 38
  22. 2022Policy Gradient in Robust MDPs with Global Convergence Guaranteecited by 38
  23. 2019DisCoRL: Continual Reinforcement Learning via Policy Distillationcited by 36
  24. 2021Deep Reinforcement Learning for the Control of Robotic Manipulation: A Focussed Mini-Reviewcited by 36
  25. 2023The Blessing of Heterogeneity in Federated Q-Learning: Linear Speedup and Beyondcited by 36
  26. 2023Natural Actor-Critic for Robust Reinforcement Learning with Function Approximationcited by 36
  27. 2020Trajectory-wise Multiple Choice Learning for Dynamics Generalization in Reinforcement Learningcited by 35
  28. 2021Survey on reinforcement learning for language processingcited by 31
  29. 2023Policy Gradient for Rectangular Robust Markov Decision Processescited by 31
  30. 2022Provable Benefit of Multitask Representation Learning in Reinforcement Learningcited by 29
  31. 2021Towards Theoretical Understandings of Robust Markov Decision Processes: Sample Complexity and Asymptoticscited by 27
  32. 2019Continual Reinforcement Learning deployed in Real-life using Policy Distillation and Sim2Real Transfercited by 26
  33. 2024Distributionally Robust Reinforcement Learning with Interactive Data Collection: Fundamental Hardness and Near-Optimal Algorithmscited by 26
  34. 2019How to pick the domain randomization parameters for sim-to-real transfer of reinforcement learning policies?cited by 25
  35. 2024Distributionally Robust Off-Dynamics Reinforcement Learning: Provable Efficiency with Linear Function Approximationcited by 25
  36. 2020Probabilistic Active Meta-Learningcited by 23
  37. 2023Model-Free Robust Average-Reward Reinforcement Learningcited by 20
  38. 2023Dynamics Generalisation in Reinforcement Learning via Adaptive Context-Aware Policiescited by 20

50 most cited of 65 All 65 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Learning Sequential Decisions from Multiple Sources via Group-Robust Markov Decision Processescited by 0
  2. 2026Formalizing Task-Space Complexity for Zero-Shot Generalizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Pessimistic Auxiliary Policy for Offline Reinforcement Learning
  2. 2024Pessimism Meets Risk: Risk-Sensitive Offline Reinforcement Learning
  3. 2025Semi-pessimistic Reinforcement Learning
  4. 2023Double Pessimism is Provably Efficient for Distributionally Robust Offline Reinforcement Learning: Generic Algorithm and Robust Partial Coverage
  5. 2022State-Aware Proximal Pessimistic Algorithms for Offline Reinforcement Learning
  6. 2022Model-Based Offline Reinforcement Learning with Pessimism-Modulated Dynamics Belief
  7. 2022The Efficacy of Pessimism in Asynchronous Q-Learning
  8. 2022Pessimism in the Face of Confounders: Provably Efficient Offline Reinforcement Learning in Partially Observable Markov Decision Processes