mlsanityData to 14 Oct 2026

Hybrid Transfer Reinforcement Learning: Provable Sample Efficiency from Shifted-Dynamics Data

arXiv 2411.03810 ↗First posted 6 Nov 2024cs.LG stat.MLcited by 8

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Online Reinforcement learning (RL) typically requires high-stakes online interaction data to learn a policy for a target task. This prompts interest in leveraging historical data to improve sample efficiency. The historical data may come from outdated or related source environments with different dynamics. It remains unclear how to effectively use such data in the target task to provably enhance learning and sample efficiency. To address this, we propose a hybrid transfer RL (HTRL) setting, where an agent learns in a target environment while accessing offline data from a source environment with shifted dynamics. We show that -- without information on the dynamics shift -- general shifted-dynamics data, even with subtle shifts, does not reduce sample complexity in the target environment. However, with prior information on the degree of the dynamics shift, we design HySRL, a transfer algorithm that achieves problem-dependent sample complexity and outperforms pure online RL. Finally, our experimental results demonstrate that HySRL surpasses state-of-the-art online RL baseline.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 73 of its 90 references that are in the map. First posted on arXiv 6 Nov 2024.

Builds on

73 of 90 references.

References · 73

  1. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  2. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  3. 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 672
  4. 2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 503
  5. 2017Minimax Regret Bounds for Reinforcement Learningcited by 503
  6. 2018Is Q-learning Provably Efficient?cited by 487
  7. 2016Learning to reinforcement learncited by 377
  8. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 339
  9. 2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 286
  10. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimismcited by 279
  11. 2019Tighter Problem-Dependent Regret Bounds in Reinforcement Learning without Domain Knowledge using Value Function Boundscited by 195
  12. 2020Reward-Free Exploration for Reinforcement Learningcited by 177
38 more
  1. 2021Multi-Task Reinforcement Learning with Context-based Representationscited by 174
  2. 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 161
  3. 2021Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learningcited by 158
  4. 2019Minimax Weight and Q-Function Learning for Off-Policy Evaluationcited by 150
  5. 2020Almost Optimal Model-Free Reinforcement Learning via Reference-Advantage Decompositioncited by 127
  6. 2020Minimax-Optimal Off-Policy Evaluation with Linear Function Approximationcited by 120
  7. 2019Double Reinforcement Learning for Efficient Off-Policy Evaluation in Markov Decision Processescited by 119
  8. 2023A Tutorial on Meta-Reinforcement Learningcited by 108
  9. 2020Episodic Reinforcement Learning in Finite MDPs: Minimax Lower Bounds Revisitedcited by 106
  10. 2020Is Reinforcement Learning More Difficult Than Bandits? A Near-optimal Algorithm Escaping the Curse of Horizoncited by 99
  11. 2020Off-Dynamics Reinforcement Learning: Training for Transfer with Domain Classifierscited by 97
  12. 2020Model-Based Multi-Agent RL in Zero-Sum Markov Games with Near-Optimal Sample Complexitycited by 92
  13. 2020Fast active learning for pure exploration in reinforcement learningcited by 92
  14. 2020On Reward-Free Reinforcement Learning with Linear Function Approximationcited by 89
  15. 2020Off-Policy Evaluation via the Regularized Lagrangiancited by 86
  16. 2021Sample Complexity of Robust Reinforcement Learning with a Generative Modelcited by 83
  17. 2021Understanding Domain Randomization for Sim-to-real Transfercited by 80
  18. 2019Provably Efficient Q-Learning with Low Switching Costcited by 78
  19. 2020Adaptive Reward-Free Explorationcited by 77
  20. 2019Variance-reduced Q-learning is minimax optimalcited by 73
  21. 2013Hidden Parameter Markov Decision Processes: A Semiparametric Regression Approach for Discovering Latent Task Parametrizationscited by 70
  22. 2021RL for Latent MDPs: Regret Guarantees and a Lower Boundcited by 70
  23. 2020Q-learning with Logarithmic Regretcited by 60
  24. 2022When to Trust Your Simulator: Dynamics-Aware Hybrid Offline-and-Online Reinforcement Learningcited by 60
  25. 2013Sample Complexity of Multi-task Reinforcement Learningcited by 58
  26. 2020Provably Efficient Reward-Agnostic Navigation with Linear Value Iterationcited by 56
  27. 2019Q-learning with UCB Exploration is Sample Efficient for Infinite-Horizon MDPcited by 55
  28. 2023The Curious Price of Distributional Robustness in Reinforcement Learning with a Generative Modelcited by 52
  29. 2022Reward-Free RL is No Harder Than Reward-Aware RL in Linear Markov Decision Processescited by 49
  30. 2021UCB Momentum Q-learning: Correcting the bias without forgettingcited by 46
  31. 2018Variance-Aware Regret Bounds for Undiscounted Reinforcement Learning in MDPscited by 43
  32. 2020Nearly Minimax Optimal Reinforcement Learning for Discounted MDPscited by 42
  33. 2022DARA: Dynamics-Aware Reward Augmentation in Offline Reinforcement Learningcited by 41
  34. 2021Nearly Horizon-Free Offline Reinforcement Learningcited by 36
  35. 2020Planning in Markov Decision Processes with Gap-Dependent Sample Complexitycited by 34
  36. 2022Distributionally Robust Offline Reinforcement Learning with Linear Function Approximationcited by 32
  37. 2018Variational Bayesian Reinforcement Learning with Regret Boundscited by 31
  38. 2022On the Statistical Efficiency of Reward-Free Exploration in Non-Linear RLcited by 27

50 most cited of 73 All 73 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 4.

8 papers.

  1. 2025Composite Flow Matching for Reinforcement Learning with Shifted-Dynamics Datacited by 9
  2. 2025Contextual Online Pricing with (Biased) Offline Datacited by 4
  3. 2025Pessimism Principle Can Be Effective: Towards a Framework for Zero-Shot Transfer Reinforcement Learningcited by 2
  4. 2025Hybrid Combinatorial Multi-armed Bandits with Probabilistically Triggered Armscited by 2
  5. 2026Provably Efficient Offline-to-Online Value Adaptation with General Function Approximationcited by 0
  6. 2026Sample-Mean Anchored Thompson Sampling for Offline-to-Online Learning with Distribution Shiftcited by 0
  7. 2026Transfer Learning Across Policy Regimes in Adaptive Multi-Agent Systemscited by 0
  8. 2026Provably Safe Sim-to-Real Transfercited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Efficient Reinforcement Learning by Guiding World Models with Non-Curated Data
  2. 2023Using Offline Data to Speed Up Reinforcement Learning in Procedurally Generated Environments
  3. 2024A Natural Extension To Online Algorithms For Hybrid RL With Limited Coverage
  4. 2025Can RLHF be More Efficient with Imperfect Reward Models? A Policy Coverage Perspective
  5. 2025Multi-Fidelity Hybrid Reinforcement Learning via Information Gain Maximization
  6. 2022Leveraging Offline Data in Online Reinforcement Learning
  7. 2025A Snapshot of Influence: A Local Data Attribution Framework for Online Reinforcement Learning
  8. 2025NeoRL-2: Near Real-World Benchmarks for Offline Reinforcement Learning with Extended Realistic Scenarios