mlsanityData to 7 Oct 2026

Robot Fine-Tuning Made Easy: Pre-Training Rewards and Policies for Autonomous Real-World Reinforcement Learning

arXiv 2310.15145 ↗First posted 23 Oct 2023cs.RO cs.AI cs.LGcited by 49

Not on the map. Why?

Abstract (arXiv)

The pre-train and fine-tune paradigm in machine learning has had dramatic success in a wide range of domains because the use of existing data or pre-trained models on the internet enables quick and easy learning of new tasks. We aim to enable this paradigm in robotic reinforcement learning, allowing a robot to learn a new task with little human effort by leveraging data and models from the Internet. However, reinforcement learning often requires significant human effort in the form of manual reward specification or environment resets, even if the policy is pre-trained. We introduce RoboFuME, a reset-free fine-tuning system that pre-trains a multi-task manipulation policy from diverse datasets of prior experiences and self-improves online to learn a target task with minimal human intervention. Our insights are to utilize calibrated offline reinforcement learning techniques to ensure efficient online fine-tuning of a pre-trained policy in the presence of distribution shifts and leverage pre-trained vision language models (VLMs) to build a robust reward classifier for autonomously providing reward signals during the online fine-tuning process. In a diverse set of five real robot manipulation tasks, we show that our method can incorporate data from an existing robot dataset collected at a different institution and improve on a target task within as little as 3 hours of autonomous real-world experience. We also demonstrate in simulation experiments that our method outperforms prior works that use different RL algorithms or different approaches for predicting rewards. Project website: https://robofume.github.io

Key facts. Cited by 49 papers in the map (data through 7 Oct 2026). Builds on 47 of its 55 references that are in the map. First posted on arXiv 23 Oct 2023.

Builds on

47 of 55 references.

References · 47

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  4. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  5. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  6. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  7. 2016Generative Adversarial Imitation Learningcited by 1,576
  8. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  9. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  10. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  11. 2019Behavior Regularized Offline Reinforcement Learningcited by 654
  12. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
35 more
  1. 2023BridgeData V2: A Dataset for Robot Learning at Scalecited by 634
  2. 2020MOPO: Model-based Offline Policy Optimizationcited by 630
  3. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
  4. 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 453
  5. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 400
  6. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 384
  7. 2023Language to Rewards for Robotic Skill Synthesiscited by 307
  8. 2021Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasetscited by 248
  9. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 230
  10. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemblecited by 211
  11. 2023LIV: Language-Image Representations and Rewards for Robotic Controlcited by 180
  12. 2023Vision-Language Models as Success Detectorscited by 138
  13. 2020PLAS: Latent Action Space for Offline Reinforcement Learningcited by 133
  14. 2019End-to-End Robotic Reinforcement Learning without Reward Engineeringcited by 115
  15. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learningcited by 112
  16. 2020The Ingredients of Real-World Robotic Reinforcement Learningcited by 111
  17. 2020Learning to Walk in the Real World with Minimal Human Effortcited by 103
  18. 2020Offline Reinforcement Learning from Images with Latent Space Modelscited by 94
  19. 2021Reset-Free Reinforcement Learning via Multi-Task Learning: Learning Dexterous Manipulation Behaviors without Human Interventioncited by 90
  20. 2017Leave no Trace: Learning to Reset for Safe and Autonomous Reinforcement Learningcited by 86
  21. 2022Pre-Training for Robots: Offline RL Enables Learning New Tasks from a Handful of Trialscited by 78
  22. 2022Supported Policy Optimization for Offline Reinforcement Learningcited by 73
  23. 2018Variational Inverse Control with Events: A General Framework for Data-Driven Reward Definitioncited by 66
  24. 2016Model-Free Imitation Learning with Policy Optimizationcited by 52
  25. 2020Reset-Free Lifelong Learning with Skill-Space Planningcited by 38
  26. 2021Autonomous Reinforcement Learning via Subgoal Curriculacited by 32
  27. 2017Deep Predictive Policy Training using Reinforcement Learningcited by 30
  28. 2021Fully Autonomous Real-World Reinforcement Learning with Applications to Mobile Manipulationcited by 30
  29. 2023Self-Improving Robots: End-to-End Autonomous Visuomotor Reinforcement Learningcited by 29
  30. 2022Don't Start From Scratch: Leveraging Prior Data to Automate Robotic Reinforcement Learningcited by 28
  31. 2022A State-Distribution Matching Approach to Non-Episodic Reinforcement Learningcited by 21
  32. 2022Improving TD3-BC: Relaxed Policy Constraint for Offline Learning and Stable Online Fine-Tuningcited by 20
  33. 2022Dexterous Manipulation from Images: Autonomous Real-World RL via Substep Guidancecited by 20
  34. 2022Demonstration-Bootstrapped Autonomous Practicing via Multi-Task Reinforcement Learningcited by 15
  35. 2020High Acceleration Reinforcement Learning for Real-World Juggling with Binary Rewardscited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 11; 2025: 17; 2026: 21.

49 papers.

Sort:

By area:

  1. 2024Robotic Control via Embodied Chain-of-Thought Reasoningcited by 340
  2. 2025π*0.6: a VLA That Learns From Experiencecited by 280
  3. 2024Diffusion Policy Policy Optimizationcited by 255
  4. 2024Reconciling Reality through Simulation: A Real-to-Sim-to-Real Approach for Robust Manipulationcited by 135
  5. 2024Unpacking Failure Modes of Generative Policies: Runtime Monitoring of Consistency and Progresscited by 63
  6. 2024Policy Decorator: Model-Agnostic Online Refinement for Large Policy Modelcited by 60
  7. 2025Residual Off-Policy RL for Finetuning Behavior Cloning Policiescited by 42
  8. 2024Autonomous Improvement of Instruction Following Skills via Foundation Modelscited by 32
  9. 2025DiWA: Diffusion Policy Adaptation with World Modelscited by 28
  10. 2024Affordance-Guided Reinforcement Learning via Visual Promptingcited by 20
  11. 2025Robust Finetuning of Vision-Language-Action Robot Policies via Parameter Mergingcited by 19
  12. 2025DeepThinkVLA: Enhancing Reasoning Capability of Vision-Language-Action Modelscited by 17
37 more
  1. 2026BPP: Long-Context Robot Imitation Learning by Focusing on Key History Framescited by 17
  2. 2025A Taxonomy for Evaluating Generalist Robot Manipulation Policiescited by 15
  3. 2025Object Detection with Multimodal Large Vision-Language Models: An In-depth Reviewcited by 14
  4. 2025Distributionally Robust Reinforcement Learning with Human Feedbackcited by 12
  5. 2025A Survey on Imitation Learning for Contact-Rich Tasks in Roboticscited by 12
  6. 2024Robot Learning with Super-Linear Scalingcited by 10
  7. 2025OpenGVL -- Benchmarking Visual Temporal Progress for Data Curationcited by 9
  8. 2026From Prior to Pro: Efficient Skill Mastery via Distribution Contractive RL Finetuningcited by 8
  9. 2024On-Robot Reinforcement Learning with Goal-Contrastive Rewardscited by 7
  10. 2025TeViR: Text-to-Video Reward with Diffusion Models for Efficient Reinforcement Learningcited by 7
  11. 2025SLAC: Safe and Efficient Real-Robot Reinforcement Learning via Unsupervised Simulation Pre-Trainingcited by 7
  12. 2025Scaffolding Dexterous Manipulation with Vision-Language Modelscited by 7
  13. 2026ALOE: Action-Level Off-Policy Evaluation for Vision-Language-Action Model Post-Trainingcited by 7
  14. 2024The Duality of Generative AI and Reinforcement Learning in Robotics: A Reviewcited by 5
  15. 2025LuciBot: Automated Robot Policy Learning from Generated Videoscited by 4
  16. 2026TapSampling: Inference-Time Sampling with a Task-Progress-Understanding Verifier for Robotic Manipulationcited by 4
  17. 2026Beyond Imitation: Self-Improving Robot Policies via Off-Policy Q-Planningcited by 4
  18. 2025CIVIL: Causal and Intuitive Visual Imitation Learningcited by 3
  19. 2026TMRL: Diffusion Timestep-Modulated Pretraining Enables Exploration for Efficient Policy Finetuningcited by 3
  20. 2026Look Before You Leap: Distilling Tree Search into Action Evaluation for Frozen VLA Modelscited by 3
  21. 2026Progress Reward Modeling for Robotic Learning: A Comprehensive Surveycited by 3
  22. 2025Humans Coexist, So Must Embodied Artificial Agentscited by 2
  23. 2026Rewarding DINO: Predicting Dense Rewards with Vision Foundation Modelscited by 2
  24. 2026Scaling Sim-to-Real VLA Reinforcement Learning with Generative 3D Worldscited by 2
  25. 2026SARM2: Multi-Task Stage Aware Reward Modeling for Self Improving Robotic Manipulationcited by 2
  26. 2024LoKO: Low-Rank Kalman Optimizer for Online Fine-Tuning of Large Modelscited by 1
  27. 2025Towards Autonomous Reinforcement Learning for Real-World Robotic Manipulation with Large Language Modelscited by 1
  28. 2026HAF: Adapting Generalist VLAs to Humanoid Whole-Body Loco-manipulation via Hierarchical Action Flow and Spectral Latent RLcited by 1
  29. 2026Plan2Cleanse: Test-Time Backdoor Defense via Monte-Carlo Planning in Deep Reinforcement Learningcited by 0
  30. 2026Adaptive Smooth Tchebycheff Attention for Multi-Objective Policy Optimizationcited by 0
  31. 2026Learning Process Rewards via Success Visitation Matching for Efficient RLcited by 0
  32. 2026HELP: Human-Efficient Large-Scale Robot Post-Training with Rollout Segmentationcited by 0
  33. 2026Robo-ValueRL: Reliable Value Estimation for Offline-to-Online Reinforcement Learningcited by 0
  34. 2026What Language Does and What the Evidence Supports: A Functional Role Taxonomy and Evidence Audit of Language Grounding in Embodied Agentscited by 0
  35. 2026SpeedTuning: Speeding Up Policy Execution with Lightweight Reinforcement Learningcited by 0
  36. 2026SeeQ: Training Generalist Value Functions for Long-Horizon Robotic Manipulationcited by 0
  37. 2026Find Something You Can't Do: Agentic Real-World Reinforcement Learning for Self-Improving VLA Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024MOTO: Offline Pre-training to Online Fine-tuning for Model-based Robot Learning
  2. 2025Self-Improving Embodied Foundation Models
  3. 2026Large Reward Models: Generalizable Online Robot Reward Generation with Vision-Language Models
  4. 2025Dual-Actor Fine-Tuning of VLA Models: A Talk-and-Tweak Human-in-the-Loop Approach
  5. 2025RoboGPT-R1: Enhancing Robot Task Planning with Reinforcement Learning
  6. 2024EXTRACT: Efficient Policy Learning by Extracting Transferable Robot Skills from Offline Data
  7. 2022Planning to Practice: Efficient Online Fine-Tuning by Composing Goals in Latent Space
  8. 2025ReWiND: Language-Guided Rewards Teach Robot Policies without New Demonstrations