mlsanityData to 14 Oct 2026

Offline-Online Reinforcement Learning for Linear Mixture MDPs

arXiv 2604.11994 ↗First posted 13 Apr 2026cs.LG math.OC stat.MLcited by 0

Not on the map. Why?

Abstract (arXiv)

We study offline-online reinforcement learning in linear mixture Markov decision processes (MDPs) under environment shift. In the offline phase, data are collected by an unknown behavior policy and may come from a mismatched environment, while in the online phase the learner interacts with the target environment. We propose an algorithm that adaptively leverages offline data. When the offline data are informative, either due to sufficient coverage or small environment shift, the algorithm provably improves over purely online learning. When the offline data are uninformative, it safely ignores them and matches the online-only performance. We establish regret upper bounds that explicitly characterize when offline data are beneficial, together with nearly matching lower bounds. Numerical experiments further corroborate our theoretical findings.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 18 of its 71 references that are in the map. First posted on arXiv 13 Apr 2026.

Builds on

18 of 71 references.

References · 18

  1. 2018Exploration by Random Network Distillationcited by 993
  2. 2017StarCraft II: A New Challenge for Reinforcement Learningcited by 274
  3. 2021The Statistical Complexity of Interactive Decision Makingcited by 218
  4. 2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coveragecited by 164
  5. 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 161
  6. 2022Near-optimal Offline Reinforcement Learning with Linear Representation: Leveraging Variance Information with Pessimismcited by 65
  7. 2020Meta-Reinforcement Learning Robust to Distributional Shift via Model Identification and Experience Relabelingcited by 26
  8. 2024UCB algorithms for multi-armed bandits: Precise regret and adaptive inferencecited by 19
  9. 2021Near-Optimal Reward-Free Exploration for Linear Mixture MDPs with Plug-in Solvercited by 14
  10. 2022Artificial Replay: A Meta-Algorithm for Harnessing Historical Data in Banditscited by 13
  11. 2024Online Bandits with (Biased) Offline Data: Adaptive Learning under Distribution Mismatchcited by 10
  12. 2021Sample Complexity of Offline Reinforcement Learning with Deep ReLU Networkscited by 8
6 more
  1. 2023Optimal Best-Arm Identification in Bandits with Access to Offline Datacited by 8
  2. 2022Data-pooling Reinforcement Learning for Personalized Healthcare Interventioncited by 6
  3. 2025Balancing optimism and pessimism in offline-to-online learningcited by 6
  4. 2025Contextual Online Pricing with (Biased) Offline Datacited by 4
  5. 2024Exploiting Exogenous Structure for Sample-Efficient Reinforcement Learningcited by 3
  6. 2025Reinforcement Learning in MDPs with Information-Ordered Policiescited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2022Offline Reinforcement Learning via Linear-Programming with Error-Bound Induced Constraints
  2. 2022Settling the Sample Complexity of Model-Based Offline Reinforcement Learning
  3. 2021Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learning
  4. 2025Optimal Single-Policy Sample Complexity and Transient Coverage for Average-Reward Offline RL
  5. 2021Towards Instance-Optimal Offline Reinforcement Learning with Pessimism
  6. 2021Nearly Horizon-Free Offline Reinforcement Learning
  7. 2022Nearly Minimax Optimal Offline Reinforcement Learning with Linear Function Approximation: Single-Agent MDP and Markov Game
  8. 2022Leveraging Offline Data in Online Reinforcement Learning