mlsanityData to 7 Oct 2026

Provable Domain Adaptation for Offline Reinforcement Learning with Limited Samples

arXiv 2408.12136 ↗First posted 22 Aug 2024cs.LG stat.MLcited by 5

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning (RL) learns effective policies from a static target dataset. The performance of state-of-the-art offline RL algorithms notwithstanding, it relies on the size of the target dataset, and it degrades if limited samples in the target dataset are available, which is often the case in real-world applications. To address this issue, domain adaptation that leverages auxiliary samples from related source datasets (such as simulators) can be beneficial. However, establishing the optimal way to trade off the limited target dataset and the large-but-biased source dataset while ensuring provably theoretical guarantees remains an open challenge. To the best of our knowledge, this paper proposes the first framework that theoretically explores the impact of the weights assigned to each dataset on the performance of offline RL. In particular, we establish performance bounds and the existence of the optimal weight, which can be computed in closed form under simplifying assumptions. We also provide algorithmic guarantees in terms of convergence to a neighborhood of the optimum. Notably, these results depend on the quality of the source dataset and the number of samples in the target dataset. Our empirical results on the well-known Procgen and MuJoCo benchmarks substantiate the theoretical contributions in this work.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 30 of its 33 references that are in the map. First posted on arXiv 22 Aug 2024.

Builds on

30 of 33 references.

Closest ancestors · 11

Same direction, newest first

  1. 2024Dataset Distillation for Offline Reinforcement Learning29 Jul
  2. 2024Improving Offline Reinforcement Learning with Inaccurate Simulators7 May
  3. 2023The Generalization Gap in Offline Reinforcement Learning10 Dec
  4. 2023Leveraging Factored Action Spaces for Efficient Offline Reinforcement Learning in Healthcare2 May
  5. 2022When to Trust Your Simulator: Dynamics-Aware Hybrid Offline-and-Online Reinforcement Learning27 Jun
  6. 2022DARA: Dynamics-Aware Reward Augmentation in Offline Reinforcement Learning13 Mar
  7. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  8. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  9. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  10. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  11. 2019Benchmarking Batch Deep Reinforcement Learning Algorithms3 Oct

Other references · 19

  1. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  2. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  3. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  4. 2015End-to-End Training of Deep Visuomotor Policiescited by 847
  5. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  6. 2017Minimax Regret Bounds for Reinforcement Learningcited by 500
  7. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 478
  8. 2016Benchmarking Deep Reinforcement Learning for Continuous Controlcited by 361
  9. 2020A Brief Review of Domain Adaptationcited by 287
  10. 2021Reinforcement learning based recommender systems: A surveycited by 187
  11. 2021A Survey of Zero-shot Generalisation in Deep Reinforcement Learningcited by 164
  12. 2018Assessing Generalization in Deep Reinforcement Learningcited by 132
7 more
  1. 2020A survey on domain adaptation theory: learning bounds and theoretical guaranteescited by 109
  2. 2020Off-Dynamics Reinforcement Learning: Training for Transfer with Domain Classifierscited by 93
  3. 2019Model-free Reinforcement Learning in Infinite-horizon Average-reward Markov Decision Processescited by 81
  4. 2017Virtual to Real Reinforcement Learning for Autonomous Drivingcited by 79
  5. 2019Diagnosing Bottlenecks in Deep Q-learning Algorithmscited by 73
  6. 2021A Survey on Reinforcement Learning for Recommender Systemscited by 17
  7. 2023Probabilistic Constraint for Safety-Critical Reinforcement Learningcited by 11

Built on by

Cited by year

Citing papers per year: 2024: 2; 2025: 3.

5 papers.

  1. 2024Hybrid Transfer Reinforcement Learning: Provable Sample Efficiency from Shifted-Dynamics Datacited by 8
  2. 2025SPiDR: A Simple Approach for Zero-Shot Safety in Sim-to-Real Transfercited by 8
  3. 2025DmC: Nearest Neighbor Guidance Diffusion Model for Offline Cross-domain Reinforcement Learningcited by 7
  4. 2025Contextual Online Pricing with (Biased) Offline Datacited by 3
  5. 2024Random Policy Enables In-Context Reinforcement Learning within Trust Horizonscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024Integrating Domain Knowledge for handling Limited Data in Offline RL
  2. 2025Efficient Cross-Domain Offline Reinforcement Learning with Dynamics- and Value-Aligned Data Filtering
  3. 2022Domain Generalization for Robust Model-Based Offline Reinforcement Learning
  4. 2026Cross-Domain Offline Policy Adaptation via Selective Transition Correction
  5. 2026Localized Dynamics-Aware Domain Adaption for Off-Dynamics Offline Reinforcement Learning
  6. 2024On Sample-Efficient Offline Reinforcement Learning: Data Diversity, Posterior Sampling, and Beyond
  7. 2025Statistical Guarantees for Offline Domain Randomization
  8. 2023Adaptive Policy Learning for Offline-to-Online Reinforcement Learning