mlsanityData to 7 Oct 2026

Failure-Aware RL: Reliable Offline-to-Online Reinforcement Learning with Self-Recovery for Real-World Manipulation

arXiv 2601.07821 ↗First posted 12 Jan 2026cs.RO cs.AI cs.LGcited by 6

On the map: Offline RL and imitation learning from data → Safe offline RL

Abstract (arXiv)

Post-training algorithms based on deep reinforcement learning can push the limits of robotic models for specific objectives, such as generalizability, accuracy, and robustness. However, Intervention-requiring Failures (IR Failures) (e.g., a robot spilling water or breaking fragile glass) during real-world exploration happen inevitably, hindering the practical deployment of such a paradigm. To tackle this, we introduce Failure-Aware Offline-to-Online Reinforcement Learning (FARL), a new paradigm minimizing failures during real-world reinforcement learning. We create FailureBench, a benchmark that incorporates common failure scenarios requiring human intervention, and propose an algorithm that integrates a world-model-based safety critic and a recovery policy trained offline to prevent failures during online exploration. Extensive simulation and real-world experiments demonstrate the effectiveness of FARL in significantly reducing IR Failures while improving performance and generalization during online reinforcement learning post-training. FARL reduces IR Failures by 73.1% while elevating performance by 11.3% on average during real-world RL post-training. Videos and code are available at https://failure-aware-rl.github.io.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in Safe offline RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 47 of its 50 references that are in the map. First posted on arXiv 12 Jan 2026.

Builds on

47 of 50 references.

References · 47

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  3. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  4. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  5. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  6. 2017Constrained Policy Optimizationcited by 928
  7. 2019Behavior Regularized Offline Reinforcement Learningcited by 654
  8. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
  9. 2018Reward Constrained Policy Optimizationcited by 397
  10. 2024Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robotscited by 373
  11. 2019Control Barrier Functions: Theory and Applicationscited by 332
  12. 2018A Lyapunov-based Approach to Safe Reinforcement Learningcited by 278
35 more
  1. 2019End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Taskscited by 256
  2. 2018Safe Exploration in Continuous Action Spacescited by 237
  3. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 230
  4. 2014Robots that can adapt like animalscited by 203
  5. 2020Recovery RL: Safe Reinforcement Learning with Learned Recovery Zonescited by 156
  6. 2021Offline RL Without Off-Policy Evaluationcited by 150
  7. 2019Lyapunov-based Safe Policy Optimization for Continuous Controlcited by 140
  8. 2015Model Predictive Path Integral Control using Covariance Variable Importance Samplingcited by 122
  9. 2020Learning to be Safe: Deep RL with a Safety Criticcited by 109
  10. 2018A predictive safety filter for learning-based control of constrained nonlinear dynamical systemscited by 91
  11. 2021Reset-Free Reinforcement Learning via Multi-Task Learning: Learning Dexterous Manipulation Behaviors without Human Interventioncited by 90
  12. 2017Leave no Trace: Learning to Reset for Safe and Autonomous Reinforcement Learningcited by 86
  13. 2018Accelerated Primal-Dual Policy Optimization for Safe Reinforcement Learningcited by 61
  14. 2022Penalized Proximal Policy Optimization for Safe Reinforcement Learningcited by 60
  15. 2024Agile But Safe: Learning Collision-Free High-Speed Legged Locomotioncited by 53
  16. 2023SafeDreamer: Safe Reinforcement Learning with World Modelscited by 50
  17. 2025RL-100: Performant Robotic Manipulation with Real-World Reinforcement Learningcited by 47
  18. 2023DrM: Mastering Visual Reinforcement Learning through Dormant Ratio Minimizationcited by 45
  19. 2023Behavior Proximal Policy Optimizationcited by 43
  20. 2022Efficient Adversarial Training without Attacking: Worst-Case-Aware Robust Reinforcement Learningcited by 42
  21. 2024A Survey of Constraint Formulations in Safe Reinforcement Learningcited by 42
  22. 2022On the Robustness of Safe Reinforcement Learning under Observational Perturbationscited by 38
  23. 2021Autonomous Reinforcement Learning: Formalism and Benchmarkingcited by 33
  24. 2023PROTO: Iterative Policy Regularized Offline-to-Online Reinforcement Learningcited by 28
  25. 2023RL-ViGen: A Reinforcement Learning Benchmark for Visual Generalizationcited by 27
  26. 2023Uni-O4: Unifying Online and Offline Deep Reinforcement Learning with Multi-Step On-Policy Optimizationcited by 23
  27. 2024Robots Pre-train Robots: Manipulation-Centric Robotic Representation from Large-Scale Robot Datasetscited by 23
  28. 2023Seizing Serendipity: Exploiting the Value of Past Success in Off-Policy Actor-Criticcited by 20
  29. 2024ACE : Off-Policy Actor-Critic with Causality-Aware Entropy Regularizationcited by 16
  30. 2020Chance-Constrained Trajectory Optimization for Safe Exploration and Learning of Nonlinear Systemscited by 15
  31. 2024Learning Visual Quadrupedal Loco-Manipulation from Demonstrationscited by 15
  32. 2024Safe Reinforcement Learning on the Constraint Manifold: Theory and Applicationscited by 12
  33. 2025HERMES: Human-to-Robot Embodied Learning from Multi-Source Motion Data for Mobile Dexterous Manipulationcited by 10
  34. 2023Safe Deep Policy Adaptationcited by 9
  35. 2024Beyond Worst-case Attacks: Robust RL with Adaptive Defense via Non-dominated Policiescited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026UniIntervene: Agentic Intervention for Efficient Real-World Reinforcement Learningcited by 4
  2. 2026FAR: Failure-Aware Retry for Test-Time Recovery and Continual Policy Improvementcited by 4
  3. 2026AEGIS: A Backup Reflex for Physical AIcited by 2
  4. 2026Learning Robust Execution in Robotic Manipulation with Agentic Reinforcement Learningcited by 2
  5. 2026DreamAvoid: Critical-Phase Test-Time Dreaming to Avoid Failures in VLA Policiescited by 1
  6. 2026Rule-based High-Level Coaching for Goal-Conditioned Reinforcement Learning in Search-and-Rescue UAV Missions Under Limited-Simulation Trainingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Learning from Demonstration with Failure Awareness for Safe Robot Navigation
  2. 2026F4R: Failure-Driven Recognition, Reconstruction, Refinement, and Redeployment for Continual Robot Self-Improvement
  3. 2026Failing Forward: Adaptive Failure-Informed Learning for Vision-Language-Action Models
  4. 2025Feasibility-Aware Pessimistic Estimation: Toward Long-Horizon Safety in Offline RL
  5. 2026RedFlow: Redirect Failure into Action-Level Corrections for Flow-matching VLA Policy
  6. 2025Uncertainty-Aware Robotic World Model Makes Offline Model-Based Reinforcement Learning Work on Real Robots
  7. 2026Robo-ValueRL: Reliable Value Estimation for Offline-to-Online Reinforcement Learning
  8. 2024FOSP: Fine-tuning Offline Safe Policy through World Models