mlsanityData to 14 Oct 2026

Robust Policy Expansion for Offline-to-Online RL under Diverse Data Corruption

arXiv 2509.24748 ↗First posted 29 Sep 2025cs.LG cs.AIcited by 3

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Pretraining a policy on offline data followed by fine-tuning through online interactions, known as Offline-to-Online Reinforcement Learning (O2O RL), has emerged as a promising paradigm for real-world RL deployment. However, both offline datasets and online interactions in practical environments are often noisy or even maliciously corrupted, severely degrading the performance of O2O RL. Existing works primarily focus on mitigating the conservatism of offline policies via online exploration, while the robustness of O2O RL under data corruption, including states, actions, rewards, and dynamics, is still unexplored. In this work, we observe that data corruption induces heavy-tailed behavior in the policy, thereby substantially degrading the efficiency of online exploration. To address this issue, we incorporate Inverse Probability Weighted (IPW) into the online exploration policy to alleviate heavy-tailedness, and propose a novel, simple yet effective method termed RPEX: Robust Policy EXpansion. Extensive experimental results on D4RL datasets demonstrate that RPEX achieves SOTA O2O performance across a wide range of data corruption scenarios. Code is available at hrefhttps://github.com/felix-thu/RPEXhttps://github.com/felix-thu/RPEX.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 40 of its 53 references that are in the map. First posted on arXiv 29 Sep 2025.

Builds on

40 of 53 references.

Closest ancestors · 16

Same direction, newest first

  1. 2024Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Data10 Dec
  2. 2024AlignIQL: Policy Alignment in Implicit Q-Learning through Constrained Optimization28 May
  3. 2023Towards Robust Offline Reinforcement Learning under Diverse Data Corruption19 Oct
  4. 2023ENOTO: Improving Offline-to-Online Reinforcement Learning with Q-Ensembles12 Jun
  5. 2023Double Pessimism is Provably Efficient for Distributionally Robust Offline Reinforcement Learning: Generic Algorithm and Robust Partial Coverage16 May
  6. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning9 Mar
  7. 2022Distributionally Robust Model-Based Offline Reinforcement Learning with Near-Optimal Sample Complexity11 Aug
  8. 2022RORL: Robust Offline Reinforcement Learning via Conservative Smoothing6 Jun
  9. 2021Offline Reinforcement Learning with Value-based Episodic Memory19 Oct
  10. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  11. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  12. 2021Corruption-Robust Offline Reinforcement Learning11 Jun
4 more
  1. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
  2. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  3. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  4. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr

Other references · 24

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2015Continuous control with deep reinforcement learningcited by 3,407
  4. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  5. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  6. 2019Class-Balanced Loss Based on Effective Number of Samplescited by 1,245
  7. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  8. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  9. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,061
  10. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  11. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  12. 2022Is Conditional Generative Modeling all you need for Decision-Making?cited by 516
12 more
  1. 2023Efficient Online Reinforcement Learning with Offline Datacited by 363
  2. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policiescited by 325
  3. 2023AdaptDiffuser: Diffusion Models as Adaptive Self-evolving Plannerscited by 149
  4. 2022Robust Reinforcement Learning using Offline Datacited by 96
  5. 2018Reinforcement Learning with Perturbed Rewardscited by 79
  6. 2023The Curious Price of Distributional Robustness in Reinforcement Learning with a Generative Modelcited by 52
  7. 2021On Proximal Policy Optimization's Heavy-tailed Gradientscited by 31
  8. 2024A Systematic Review on Long-Tailed Learningcited by 28
  9. 2024Energy-Guided Diffusion Sampling for Offline-to-Online Reinforcement Learningcited by 17
  10. 2024Towards Robust Model-Based Reinforcement Learning Against Adversarial Corruptioncited by 12
  11. 2021Adaptive normalization for IPW estimationcited by 7
  12. 2024Solving Continual Offline RL through Selective Weights Activation on Aligned Spacescited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025Adversarial Fine-tuning in Offline-to-Online Reinforcement Learning for Robust Robot Controlcited by 0
  2. 2025From Static Constraints to Dynamic Adaptation: Sample-Level Constraint Relaxation for Offline-to-Online Reinforcement Learningcited by 0
  3. 2026CaM-Wolf: Causal-Aware Multimodal Agents for Social Deduction Gamescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Offline Retraining for Online RL: Decoupled Policy Learning to Mitigate Exploration Bias
  2. 2023Towards Robust Offline-to-Online Reinforcement Learning via Uncertainty and Smoothness
  3. 2023Planning to Go Out-of-Distribution in Offline-to-Online Reinforcement Learning
  4. 2024Towards Fast Safe Online Reinforcement Learning via Policy Finetuning
  5. 2025Enhancing Robustness of Offline Reinforcement Learning Under Data Corruption via Sharpness-Aware Minimization
  6. 2024Tackling Data Corruption in Offline Reinforcement Learning via Sequence Modeling
  7. 2026Flow Matching with Injected Noise for Offline-to-Online Reinforcement Learning
  8. 2026Active Offline-to-Online Reinforcement Learning