mlsanityData to 7 Oct 2026

Adversarial Fine-tuning in Offline-to-Online Reinforcement Learning for Robust Robot Control

arXiv 2510.13358 ↗First posted 15 Oct 2025cs.RO cs.AIcited by 0

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning enables sample-efficient policy acquisition without risky online interaction, yet policies trained on static datasets remain brittle under action-space perturbations such as actuator faults. This study introduces an offline-to-online framework that trains policies on clean data and then performs adversarial fine-tuning, where perturbations are injected into executed actions to induce compensatory behavior and improve resilience. A performance-aware curriculum further adjusts the perturbation probability during training via an exponential-moving-average signal, balancing robustness and stability throughout the learning process. Experiments on continuous-control locomotion tasks demonstrate that the proposed method consistently improves robustness over offline-only baselines and converges faster than training from scratch. Matching the fine-tuning and evaluation conditions yields the strongest robustness to action-space perturbations, while the adaptive curriculum strategy mitigates the degradation of nominal performance observed with the linear curriculum strategy. Overall, the results show that adversarial fine-tuning enables adaptive and robust control under uncertain environments, bridging the gap between offline efficiency and online adaptability.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 45 of its 53 references that are in the map. First posted on arXiv 15 Oct 2025.

Builds on

45 of 53 references.

Closest ancestors · 18

Same direction, newest first

  1. 2025Robust Policy Expansion for Offline-to-Online RL under Diverse Data Corruption29 Sep
  2. 2024Robustness Evaluation of Offline Reinforcement Learning for Robot Control Against Action Perturbations25 Dec
  3. 2024Towards Robust Policy: Enhancing Offline Reinforcement Learning with Adversarial Attacks and Defenses18 May
  4. 2023MICRO: Model-Based Offline Reinforcement Learning with a Conservative Bellman Operator7 Dec
  5. 2023Train Once, Get a Family: State-Adaptive Balances for Offline-to-Online Reinforcement Learning27 Oct
  6. 2023Adaptive Policy Learning for Offline-to-Online Reinforcement Learning14 Mar
  7. 2023Policy Expansion for Bridging Offline-to-Online Reinforcement Learning2 Feb
  8. 2022Adaptive Behavior Cloning Regularization for Stable Offline-to-Online Reinforcement Learning25 Oct
  9. 2022RORL: Robust Offline Reinforcement Learning via Conservative Smoothing6 Jun
  10. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems2 Mar
  11. 2021d3rlpy: An Offline Deep Reinforcement Learning Library6 Nov
  12. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
6 more
  1. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble1 Jul
  2. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  3. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
  4. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  5. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  6. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr

Other references · 27

  1. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  4. 2016OpenAI Gymcited by 1,684
  5. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
  6. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  7. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  8. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  9. 2017Robust Adversarial Reinforcement Learningcited by 439
  10. 2020Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observationscited by 235
  11. 2018Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experiencecited by 203
  12. 2019Action Robust Reinforcement Learning and Applications in Continuous Controlcited by 167
15 more
  1. 2021Robust Reinforcement Learning on State Observations with Learned Optimal Adversarycited by 133
  2. 2021Who Is the Strongest Enemy? Towards Optimal and Efficient Evasion Attacks in Deep RLcited by 54
  3. 2022PD-MORL: Preference-Driven Multi-Objective Reinforcement Learning Algorithmcited by 51
  4. 2021DeepThermal: Combustion Optimization for Thermal Power Generating Units Using Offline Reinforcement Learningcited by 47
  5. 2021Robust Policy Gradient against Strong Data Corruptioncited by 30
  6. 2023Benchmarking Offline Reinforcement Learning on Real-Robot Hardwarecited by 23
  7. 2022Offline Reinforcement Learning for Safer Blood Glucose Control in People with Type 1 Diabetescited by 22
  8. 2019Robust Model-free Reinforcement Learning with Multi-objective Bayesian Optimizationcited by 16
  9. 2024Robust Deep Reinforcement Learning with Adaptive Adversarial Perturbations in Action Spacecited by 9
  10. 2022Cyclic Policy Distillation: Sample-Efficient Sim-to-Real Reinforcement Learning with Domain Randomizationcited by 7
  11. 2022Adversarial joint attacks on legged robotscited by 4
  12. 2024A Robust Defense against Adversarial Attacks on Deep Learning-based Malware Detectors via (De)Randomized Smoothingcited by 4
  13. 2023Toward Evaluating Robustness of Reinforcement Learning with Adversarial Policycited by 3
  14. 2022DEFT: Diverse Ensembles for Fast Transfer in Reinforcement Learningcited by 2
  15. 2025Action Robust Reinforcement Learning via Optimal Adversary Aware Policy Optimizationcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Finetuning Offline World Models in the Real World
  2. 2023Finetuning from Offline Reinforcement Learning: Challenges, Trade-offs and Practical Solutions
  3. 2022Latent-Variable Advantage-Weighted Policy Optimization for Offline RL
  4. 2024MOTO: Offline Pre-training to Online Fine-tuning for Model-based Robot Learning
  5. 2025The Three Regimes of Offline-to-Online Reinforcement Learning
  6. 2024Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Data
  7. 2023Towards Robust Offline-to-Online Reinforcement Learning via Uncertainty and Smoothness
  8. 2026Active Offline-to-Online Reinforcement Learning