mlsanityData to 7 Oct 2026

Action-Free Offline-to-Online RL via Discretised State Policies

arXiv 2602.00629 ↗First posted 31 Jan 2026stat.ML cs.AI cs.LGcited by 0

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Most existing offline RL methods presume the availability of action labels within the dataset, but in many practical scenarios, actions may be missing due to privacy, storage, or sensor limitations. We formalise the setting of action-free offline-to-online RL, where agents must learn from datasets consisting solely of (s,r,s') tuples and later leverage this knowledge during online interaction. To address this challenge, we propose learning state policies that recommend desirable next-state transitions rather than actions. Our contributions are twofold. First, we introduce a simple yet novel state discretisation transformation and propose Offline State-Only DecQN (algo), a value-based algorithm designed to pre-train state policies from action-free data. algo integrates the transformation to scale efficiently to high-dimensional problems while avoiding instability and overfitting associated with continuous state prediction. Second, we propose a novel mechanism for guided online learning that leverages these pre-trained state policies to accelerate the learning of online agents. Together, these components establish a scalable and practical framework for leveraging action-free datasets to accelerate online RL. Empirical results across diverse benchmarks demonstrate that our approach improves convergence speed and asymptotic performance, while analyses reveal that discretisation and regularisation are critical to its effectiveness.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 31 of its 40 references that are in the map. First posted on arXiv 31 Jan 2026.

Builds on

31 of 40 references.

Closest ancestors · 13

Same direction, newest first

  1. 2024An Investigation of Offline Reinforcement Learning in Factorisable Action Spaces17 Nov
  2. 2024State-Constrained Offline Reinforcement Learning23 May
  3. 2023Leveraging Factored Action Spaces for Efficient Offline Reinforcement Learning in Healthcare2 May
  4. 2023Balancing policy constraint and ensemble size in uncertainty-based offline reinforcement learning26 Mar
  5. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning9 Mar
  6. 2023Guiding Online Reinforcement Learning with Action-Free Offline Pretraining30 Jan
  7. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning23 Feb
  8. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  9. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  10. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  11. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  12. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
1 more
  1. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 18

  1. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  2. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  3. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  4. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  5. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  6. 2020Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 475
  7. 2019Reinforcement Learning in Healthcare: A Surveycited by 308
  8. 2024Stop Regressing: Training Value Functions via Classification for Scalable Deep RLcited by 136
  9. 2022Reinforcement Learning with Action-Free Pre-Training from Videoscited by 107
  10. 2023Action-Quantized Offline Reinforcement Learning for Robotic Skill Learningcited by 34
  11. 2022Solving Continuous Control via Q-learningcited by 21
  12. 2023When should we prefer Decision Transformers for Offline Reinforcement Learning?cited by 20
6 more
  1. 2023Learning from Visual Observation via Offline Pretrained State-to-Go Transformercited by 14
  2. 2020Estimating Q(s,s') with Deep Deterministic Dynamics Gradientscited by 12
  3. 2024Pre-trained Visual Dynamics Representations for Efficient Policy Learningcited by 8
  4. 2024REValueD: Regularised Ensemble Value-Decomposition for Factorisable Markov Decision Processescited by 6
  5. 2024Growing Q-Networks: Solving Continuous Control Tasks with Adaptive Control Resolutioncited by 6
  6. 2024NOLO: Navigate Only Look Oncecited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Offline Data Enhanced On-Policy Policy Gradient with Provable Guarantees
  2. 2024Unsupervised-to-Online Reinforcement Learning
  3. 2023Offline Retraining for Online RL: Decoupled Policy Learning to Mitigate Exploration Bias
  4. 2022Leveraging Offline Data in Online Reinforcement Learning
  5. 2025Offline vs. Online Learning in Model-based RL: Lessons for Data Collection Strategies
  6. 2024Augmenting Offline RL with Unlabeled Data
  7. 2026Active Offline-to-Online Reinforcement Learning
  8. 2022Semi-Supervised Offline Reinforcement Learning with Action-Free Trajectories