# Importance Weighted Transfer of Samples in Reinforcement Learning

arXiv: https://arxiv.org/abs/1805.10886 · HTML: https://mlsanity.com/paper/1805.10886/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data.
- First posted on arXiv 28 May 2018.

## Placed in

- Offline RL and imitation learning from data / [Offline RL algorithms](https://mlsanity.com/offlinerl/offline-rl-algorithms/)

## Built on by (most cited first, 16 of 16)

- [The Rise and Potential of Large Language Model Based Agents: A Survey](https://mlsanity.com/paper/2309.07864/) (2023)
- [RS-Agent: Automating Remote Sensing Tasks through Intelligent Agent](https://mlsanity.com/paper/2406.07089/) (2024)
- [AdaRL: What, Where, and How to Adapt in Transfer Reinforcement Learning](https://mlsanity.com/paper/2107.02729/) (2021)
- [Lifelong Robotic Reinforcement Learning by Retaining Experiences](https://mlsanity.com/paper/2109.09180/) (2021)
- [Fractional Transfer Learning for Deep Model-Based Reinforcement Learning](https://mlsanity.com/paper/2108.06526/) (2021)
- [Lifelong Hyper-Policy Optimization with Multiple Importance Sampling Regularization](https://mlsanity.com/paper/2112.06625/) (2021)
- [Hybrid Transfer in Deep Reinforcement Learning for Ads Allocation](https://mlsanity.com/paper/2204.11589/) (2022)
- [Multi-Source Transfer Learning for Deep Model-Based Reinforcement Learning](https://mlsanity.com/paper/2205.14410/) (2022)
- [Seeing Differently, Acting Similarly: Heterogeneously Observable Imitation Learning](https://mlsanity.com/paper/2106.09256/) (2021)
- [ARLO: A Framework for Automated Reinforcement Learning](https://mlsanity.com/paper/2205.10416/) (2022)
- [Truncating Trajectories in Monte Carlo Reinforcement Learning](https://mlsanity.com/paper/2305.04361/) (2023)
- [Every Question Has Its Own Value: Reinforcement Learning with Explicit Human Values](https://mlsanity.com/paper/2510.20187/) (2025)
- [Distill Knowledge in Multi-task Reinforcement Learning with Optimal-Transport Regularization](https://mlsanity.com/paper/2309.15603/) (2023)
- [Zero-Shot Context Generalization in Reinforcement Learning from Few Training Contexts](https://mlsanity.com/paper/2507.07348/) (2025)
- [Principled Fast and Meta Knowledge Learners for Continual Reinforcement Learning](https://mlsanity.com/paper/2603.00903/) (2026)
- [Robust Transfer Learning with Side Information](https://mlsanity.com/paper/2603.07921/) (2026)

## Abstract (arXiv)

We consider the transfer of experience samples (i.e., tuples < s, a, s', r >) in reinforcement learning (RL), collected from a set of source tasks to improve the learning process in a given target task. Most of the related approaches focus on selecting the most relevant source samples for solving the target task, but then all the transferred samples are used without considering anymore the discrepancies between the task models. In this paper, we propose a model-based technique that automatically estimates the relevance (importance weight) of each source sample for solving the target task. In the proposed approach, all the samples are transferred and used by a batch RL algorithm to solve the target task, but their contribution to the learning process is proportional to their importance weight. By extending the results for importance weighting provided in supervised learning literature, we develop a finite-sample analysis of the proposed batch RL algorithm. Furthermore, we empirically compare the proposed algorithm to state-of-the-art approaches, showing that it achieves better learning performance and is very robust to negative transfer, even when some source tasks are significantly different from the target task.