mlsanityData to 7 Oct 2026

Robo-ValueRL: Reliable Value Estimation for Offline-to-Online Reinforcement Learning

arXiv 2607.09866 ↗First posted 10 Jul 2026cs.RO cs.AIcited by 0

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline-to-online reinforcement learning is promising for generalizable robotic manipulation, yet its full-stack complexity obscures reproduction and diagnosis. Within such systems, value estimation plays a central role in prioritizing heterogeneous data for policy improvement. Despite its importance, the central question remains underexplored: how value-function reliability shapes policy optimization in offline-to-online reinforcement learning. To answer this question, we propose Robo-ValueRL, a unified framework that enables reliable value estimation and systematically traces its downstream effects on policy pretraining and online improvement. Concretely, Robo-ValueRL learns a history-conditioned value estimator and evaluates its reliability through global-progress and local-preference metrics. These resulting value estimates are propagated into quality-conditioned consistency-policy pretraining and a residual adaptation module on online rollouts, providing a unified testbed for analyzing how value reliability shapes downstream policy performance. Across 240 hours of offline demonstrations and over 3,000 online rollout trajectories, our extensive experiments show that downstream performance is strongly associated with value reliability. Reliable value functions provide better action-quality estimates, allowing value-guided offline RL to scale more effectively than quality-agnostic behavior cloning, and stabilize online improvement by prioritizing high-quality rollout data. Integrating reliable value guidance through offline pretraining with online improvement, our system achieves 86% success on millimeter-level precise chip insertion and 84% on generalizable block disassembly. We hope these findings highlight the importance of value-guided data utilization for effective policy improvement from heterogeneous robotic experience.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 48 of its 56 references that are in the map. First posted on arXiv 10 Jul 2026.

Builds on

48 of 56 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023Finetuning from Offline Reinforcement Learning: Challenges, Trade-offs and Practical Solutions30 Mar
  2. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning9 Mar
  3. 2023Policy Expansion for Bridging Offline-to-Online Reinforcement Learning2 Feb
  4. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems2 Mar
  5. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  6. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble1 Jul
  7. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun

Other references · 41

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  3. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  4. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  5. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,297
  6. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  7. 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
  8. 2025GR00T N1: An Open Foundation Model for Generalist Humanoid Robotscited by 1,167
  9. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  10. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 849
  11. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  12. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
29 more
  1. 2017Deep Reinforcement Learning that Matterscited by 727
  2. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
  3. 2024PaliGemma: A versatile 3B VLM for transfercited by 679
  4. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  5. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 384
  6. 2016Benchmarking Deep Reinforcement Learning for Continuous Controlcited by 361
  7. 2025π*0.6: a VLA That Learns From Experiencecited by 280
  8. 2024Consistency Policy: Accelerated Visuomotor Policies via Consistency Distillationcited by 159
  9. 2024Precise and Dexterous Robotic Manipulation via Human-in-the-Loop Reinforcement Learningcited by 153
  10. 2024RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedbackcited by 147
  11. 2025SimpleVLA-RL: Scaling VLA Training via Reinforcement Learningcited by 141
  12. 2025ConRFT: A Reinforced Fine-tuning Method for VLA Models via Consistency Policycited by 139
  13. 2024Stop Regressing: Training Value Functions via Classification for Scalable Deep RLcited by 136
  14. 2023Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functionscited by 129
  15. 2024Vision Language Models are In-Context Value Learnerscited by 84
  16. 2022Pre-Training for Robots: Offline RL Enables Learning New Tasks from a Handful of Trialscited by 78
  17. 2025A Vision-Language-Action-Critic Model for Robotic Real-World Reinforcement Learningcited by 66
  18. 2023Robot Fine-Tuning Made Easy: Pre-Training Rewards and Policies for Autonomous Real-World Reinforcement Learningcited by 49
  19. 2025GR-RL: Going Dexterous and Precise for Long-Horizon Robotic Manipulationcited by 41
  20. 2024Rank2Reward: Learning Shaped Reward Functions from Passive Videocited by 30
  21. 2021Learning from Imperfect Demonstrations from Agents with Varying Dynamicscited by 26
  22. 2025Curating Demonstrations using Online Experiencecited by 16
  23. 2025Phoenix: A Motion-based Self-Reflection Framework for Fine-grained Robotic Action Correctioncited by 16
  24. 2025XR-1: Towards Versatile Vision-Language-Action Models via Learning Unified Vision-Motion Representationscited by 15
  25. 2025Human-assisted Robotic Policy Refinement via Action Preference Optimizationcited by 13
  26. 2026Learning While Deploying: Fleet-Scale Reinforcement Learning for Generalist Robot Policiescited by 12
  27. 2026ViVa: A Video-Generative Value Model for Robot Reinforcement Learningcited by 8
  28. 2026World Value Models for Robotic Manipulationcited by 5
  29. 2026RoboGene: Boosting VLA Pre-training via Diversity-Driven Agentic Framework for Real-World Task Generationcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning
  2. 2024Is Value Learning Really the Main Bottleneck in Offline RL?
  3. 2021NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning
  4. 2026Policy Plasticity Matters in Offline-to-Online Reinforcement Learning: Refitting Offline Policies for Online Adaptation
  5. 2025Expressive Value Learning for Scalable Offline Reinforcement Learning
  6. 2022RORL: Robust Offline Reinforcement Learning via Conservative Smoothing
  7. 2025Efficient Reinforcement Learning by Guiding World Models with Non-Curated Data
  8. 2025Uncertainty-Aware Robotic World Model Makes Offline Model-Based Reinforcement Learning Work on Real Robots