Information-Directed Offline-to-Online Reinforcement Learning
On the map: Offline RL and imitation learning from data → Offline RL algorithms
Abstract (arXiv)
Decision-making from offline datasets typically warm-starts a policy or score model from fixed offline data and then refines it with limited online interaction. Offline data reduces uncertainty, but it does not remove the need for exploration; it changes what remains to be explored. We formalise this residual uncertainty by the conditional mutual information I(chi;τ1:TmidDN) between a learning target chi and the online trajectories after conditioning on the offline dataset. This view leads naturally to information-directed sampling (IDS), a family parameterised by η≥ 0 that selects actions by trading off instantaneous regret against information gain. We prove a generic offline-to-online Bayesian regret bound for IDS through a ratio certificate: any information-ratio bound satisfied by a reference Thompson-sampling policy over the same randomised policy class is inherited by IDS. In a known-dynamics Bayesian linear-reward model, the conditional mutual information has a log-determinant form, and vanilla IDS (η=0) satisfies widetilde O!left(Hdminleft√ T,,T√C^daggerβ,IDS0(N,T)/Nrightright), where the coverage coefficient is tied to the visitation distribution induced by vanilla IDS itself. We also identify a warm-start regime with a dominated but informative probe in which vanilla IDS selects the probe while Thompson sampling never does, giving a constant-factor Bayesian regret separation. Controlled bandit experiments and D4RL offline-to-online RL experiments validate this mechanism: IDS is most beneficial when offline data is informative but leaves biased or low-probability residual uncertainty that targeted online actions can resolve, a regime shared by offline RL, offline black-box optimization, and Bayesian optimization.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 35 of its 46 references that are in the map. First posted on arXiv 28 May 2026.
Builds on
35 of 46 references.
Closest ancestors · 17
Same direction, newest first
- 2024Bayesian Design Principles for Offline-to-Online Reinforcement Learning31 May
- 2023PROTO: Iterative Policy Regularized Offline-to-Online Reinforcement Learning25 May
- 2023Adaptive Policy Learning for Offline-to-Online Reinforcement Learning14 Mar
- 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning9 Mar
- 2023Policy Expansion for Bridging Offline-to-Online Reinforcement Learning2 Feb
- 2022Leveraging Offline Data in Online Reinforcement Learning9 Nov
- 2022The Role of Coverage in Online Reinforcement Learning9 Oct
- 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
- 2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coverage13 Jul
- 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble1 Jul
- 2021Bellman-consistent Pessimism for Offline Reinforcement Learning13 Jun
- 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
5 more
- 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism22 Mar
- 2020Is Pessimism Provably Efficient for Offline RL?30 Dec
- 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
- 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
- 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
Other references · 18
- 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
- 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
- 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 520
- 2023Efficient Online Reinforcement Learning with Offline Datacited by 363
- 2013Learning to Optimize Via Posterior Samplingcited by 256
- 2013(More) Efficient Reinforcement Learning via Posterior Samplingcited by 233
- 2021The Statistical Complexity of Interactive Decision Makingcited by 218
- 2014An Information-Theoretic Analysis of Thompson Samplingcited by 178
- 2014Learning to Optimize via Information-Directed Samplingcited by 168
- 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 161
- 2018Information Directed Sampling and Bandits with Heteroscedastic Noisecited by 76
- 2021Reinforcement Learning, Bit by Bitcited by 59
6 more
- 2022Regret Bounds for Information-Directed Reinforcement Learningcited by 24
- 2024CoMAL: Collaborative Multi-Agent Large Language Models for Mixed-Autonomy Trafficcited by 22
- 2025LiloDriver: A Lifelong Learning Framework for Closed-loop Motion Planning in Long-tail Autonomous Driving Scenarioscited by 7
- 2025FUAS-Agents: Autonomous Multi-Modal LLM Agents for Treatment Planning in Focused Ultrasound Ablation Surgerycited by 5
- 2026LangMARL: Natural Language Multi-Agent Reinforcement Learningcited by 4
- 2024Towards Fast Safe Online Reinforcement Learning via Policy Finetuningcited by 3
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Efficient and Uncertainty-Aware Diffusion Framework for Offline-to-Online Reinforcement Learning
- 2026Offline-Online Reinforcement Learning for Linear Mixture MDPs
- 2026From Static Policies to Adaptive Priors in Offline Reinforcement Learning
- 2024Diffusion Policies for Risk-Averse Behavior Modeling in Offline Reinforcement Learning
- 2021Revisiting Design Choices in Offline Model-Based Reinforcement Learning
- 2026Conservative Equilibrium Discovery in Offline Game-Theoretic Multiagent Reinforcement Learning
- 2022Model-Based Offline Reinforcement Learning with Pessimism-Modulated Dynamics Belief
- 2025Unsupervised Data Generation for Offline Reinforcement Learning: A Perspective from Model