Importance Weighted Transfer of Samples in Reinforcement Learning
On the map: Offline RL and imitation learning from data → Offline RL algorithms
Abstract (arXiv)
We consider the transfer of experience samples (i.e., tuples < s, a, s', r >) in reinforcement learning (RL), collected from a set of source tasks to improve the learning process in a given target task. Most of the related approaches focus on selecting the most relevant source samples for solving the target task, but then all the transferred samples are used without considering anymore the discrepancies between the task models. In this paper, we propose a model-based technique that automatically estimates the relevance (importance weight) of each source sample for solving the target task. In the proposed approach, all the samples are transferred and used by a batch RL algorithm to solve the target task, but their contribution to the learning process is proportional to their importance weight. By extending the results for importance weighting provided in supervised learning literature, we develop a finite-sample analysis of the proposed batch RL algorithm. Furthermore, we empirically compare the proposed algorithm to state-of-the-art approaches, showing that it achieves better learning performance and is very robust to negative transfer, even when some source tasks are significantly different from the target task.
Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. First posted on arXiv 28 May 2018.
Builds on
No reference list yet.
Built on by
Citing papers per year: 2021: 5; 2022: 3; 2023: 3; 2024: 1; 2025: 2; 2026: 2.
16 papers.
Sort:
By area:
- 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,439
- 2024RS-Agent: Automating Remote Sensing Tasks through Intelligent Agentcited by 49
- 2021AdaRL: What, Where, and How to Adapt in Transfer Reinforcement Learningcited by 48
- 2021Lifelong Robotic Reinforcement Learning by Retaining Experiencescited by 34
- 2021Fractional Transfer Learning for Deep Model-Based Reinforcement Learningcited by 6
- 2021Lifelong Hyper-Policy Optimization with Multiple Importance Sampling Regularizationcited by 5
- 2022Hybrid Transfer in Deep Reinforcement Learning for Ads Allocationcited by 5
- 2022Multi-Source Transfer Learning for Deep Model-Based Reinforcement Learningcited by 5
- 2021Seeing Differently, Acting Similarly: Heterogeneously Observable Imitation Learningcited by 4
- 2022ARLO: A Framework for Automated Reinforcement Learningcited by 3
- 2023Truncating Trajectories in Monte Carlo Reinforcement Learningcited by 1
- 2025Every Question Has Its Own Value: Reinforcement Learning with Explicit Human Valuescited by 1
4 more
- 2023Distill Knowledge in Multi-task Reinforcement Learning with Optimal-Transport Regularizationcited by 0
- 2025Zero-Shot Context Generalization in Reinforcement Learning from Few Training Contextscited by 0
- 2026Principled Fast and Meta Knowledge Learners for Continual Reinforcement Learningcited by 0
- 2026Robust Transfer Learning with Side Informationcited by 0
- 2026Robust Transfer Learning with Side Information9 Mar 2026
- 2026Principled Fast and Meta Knowledge Learners for Continual Reinforcement Learning1 Mar 2026
- 2025Every Question Has Its Own Value: Reinforcement Learning with Explicit Human Values23 Oct 2025
- 2025Zero-Shot Context Generalization in Reinforcement Learning from Few Training Contexts10 Jul 2025
- 2024RS-Agent: Automating Remote Sensing Tasks through Intelligent Agent11 Jun 2024
- 2023Distill Knowledge in Multi-task Reinforcement Learning with Optimal-Transport Regularization27 Sep 2023
- 2023The Rise and Potential of Large Language Model Based Agents: A Survey14 Sep 2023
- 2023Truncating Trajectories in Monte Carlo Reinforcement Learning7 May 2023
- 2022Multi-Source Transfer Learning for Deep Model-Based Reinforcement Learning28 May 2022
- 2022ARLO: A Framework for Automated Reinforcement Learning20 May 2022
- 2022Hybrid Transfer in Deep Reinforcement Learning for Ads Allocation2 Apr 2022
- 2021Lifelong Hyper-Policy Optimization with Multiple Importance Sampling Regularization13 Dec 2021
4 more
- 2021Lifelong Robotic Reinforcement Learning by Retaining Experiences19 Sep 2021
- 2021Fractional Transfer Learning for Deep Model-Based Reinforcement Learning14 Aug 2021
- 2021AdaRL: What, Where, and How to Adapt in Transfer Reinforcement Learning6 Jul 2021
- 2021Seeing Differently, Acting Similarly: Heterogeneously Observable Imitation Learning17 Jun 2021
- 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,439
- 2021AdaRL: What, Where, and How to Adapt in Transfer Reinforcement Learningcited by 48
- 2021Lifelong Robotic Reinforcement Learning by Retaining Experiencescited by 34
- 2021Lifelong Hyper-Policy Optimization with Multiple Importance Sampling Regularizationcited by 5
- 2022Hybrid Transfer in Deep Reinforcement Learning for Ads Allocationcited by 5
- 2022Multi-Source Transfer Learning for Deep Model-Based Reinforcement Learningcited by 5
- 2022ARLO: A Framework for Automated Reinforcement Learningcited by 3
- 2023Truncating Trajectories in Monte Carlo Reinforcement Learningcited by 1
- 2023Distill Knowledge in Multi-task Reinforcement Learning with Optimal-Transport Regularizationcited by 0
- 2026Robust Transfer Learning with Side Informationcited by 0
- 2026Robust Transfer Learning with Side Information9 Mar 2026
- 2023Distill Knowledge in Multi-task Reinforcement Learning with Optimal-Transport Regularization27 Sep 2023
- 2023Truncating Trajectories in Monte Carlo Reinforcement Learning7 May 2023
- 2022Multi-Source Transfer Learning for Deep Model-Based Reinforcement Learning28 May 2022
- 2022ARLO: A Framework for Automated Reinforcement Learning20 May 2022
- 2022Hybrid Transfer in Deep Reinforcement Learning for Ads Allocation2 Apr 2022
- 2021Lifelong Hyper-Policy Optimization with Multiple Importance Sampling Regularization13 Dec 2021
- 2021Lifelong Robotic Reinforcement Learning by Retaining Experiences19 Sep 2021
- 2021AdaRL: What, Where, and How to Adapt in Transfer Reinforcement Learning6 Jul 2021
Similar, not cited
Close in wording, no citation either way
- 2024Towards Sample-Efficiency and Generalization of Transfer and Inverse Reinforcement Learning: A Comprehensive Literature Review
- 2025When Importance Sampling Misallocates Credit: Asymmetric Ratios for Outcome-Supervised RL
- 2024Reusing Historical Trajectories in Natural Policy Gradient via Importance Sampling: Convergence and Convergence Rate
- 2024Model-Based Transfer Learning for Contextual Reinforcement Learning
- 2024Offline Multi-task Transfer RL with Representational Penalization
- 2025Deep Transfer Q-Learning for Offline Non-Stationary Reinforcement Learning
- 2025Supervised Fine Tuning on Curated Data is Reinforcement Learning (and can be improved)
- 2024Policy Gradient with Active Importance Sampling