# Transfer Q-learning

arXiv: https://arxiv.org/abs/2202.04709 · HTML: https://mlsanity.com/paper/2202.04709/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion and flow policies for offline RL, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 11 of its 32 references that are in the map.
- First posted on arXiv 9 Feb 2022.

## Placed in

- Offline RL and imitation learning from data / [Diffusion and flow policies for offline RL](https://mlsanity.com/offlinerl/diffusion-and-flow-policies-for-offline-rl/)

## Builds on (most cited first, 11 of 11)

- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Provably Efficient Reinforcement Learning with Linear Function Approximation](https://mlsanity.com/paper/1907.05388/) (2019)
- [Provable Meta-Learning of Linear Representations](https://mlsanity.com/paper/2002.11684/) (2020)
- [Transfer Learning for High-dimensional Linear Regression: Prediction, Estimation, and Minimax Optimality](https://mlsanity.com/paper/2006.10593/) (2020)
- [Estimating Dynamic Treatment Regimes in Mobile Health Using V-learning](https://mlsanity.com/paper/1611.03531/) (2016)
- [Transfer Learning for Nonparametric Classification: Minimax Rate and Adaptive Classifier](https://mlsanity.com/paper/1906.02903/) (2019)
- [A Reinforcement Learning Approach to Weaning of Mechanical Ventilation in Intensive Care Units](https://mlsanity.com/paper/1704.06300/) (2017)
- [Transfer Learning in Large-scale Gaussian Graphical Models with False Discovery Rate Control](https://mlsanity.com/paper/2010.11037/) (2020)
- [Reinforcement Learning for Education: Opportunities and Challenges](https://mlsanity.com/paper/2107.08828/) (2021)
- [More Efficient Policy Learning via Optimal Retargeting](https://mlsanity.com/paper/1906.08611/) (2019)
- [Robust Q-learning](https://mlsanity.com/paper/2003.12427/) (2020)

## Built on by (most cited first, 3 of 3)

- [Off-policy Evaluation in Doubly Inhomogeneous Environments](https://mlsanity.com/paper/2306.08719/) (2023)
- [Tensor-Fused Multi-View Graph Contrastive Learning](https://mlsanity.com/paper/2410.15247/) (2024)
- [Transfer Faster, Price Smarter: Minimax Dynamic Pricing under Cross-Market Preference Shift](https://mlsanity.com/paper/2505.17203/) (2025)

## Abstract (arXiv)

Time-inhomogeneous finite-horizon Markov decision processes (MDP) are frequently employed to model decision-making in dynamic treatment regimes and other statistical reinforcement learning (RL) scenarios. These fields, especially healthcare and business, often face challenges such as high-dimensional state spaces and time-inhomogeneity of the MDP process, compounded by insufficient sample availability which complicates informed decision-making. To overcome these challenges, we investigate knowledge transfer within time-inhomogeneous finite-horizon MDP by leveraging data from both a target RL task and several related source tasks. We have developed transfer learning (TL) algorithms that are adaptable for both batch and online Q-learning, integrating valuable insights from offline source studies. The proposed transfer Q-learning algorithm contains a novel em re-targeting step that enables em cross-stage transfer along multiple stages in an RL task, besides the usual em cross-task transfer for supervised learning. We establish the first theoretical justifications of TL in RL tasks by showing a faster rate of convergence of the Q^*-function estimation in the offline RL transfer, and a lower regret bound in the offline-to-online RL transfer under stage-wise reward similarity and mild design similarity across tasks. Empirical evidence from both synthetic and real datasets is presented to evaluate the proposed algorithm and support our theoretical results.