# Never Stop Learning: The Effectiveness of Fine-Tuning in Robotic Reinforcement Learning

arXiv: https://arxiv.org/abs/2004.10190 · HTML: https://mlsanity.com/paper/2004.10190/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Apr 2020.

## Built on by (most cited first, 23 of 23)

- [COG: Connecting New Skills to Past Experience with Offline Reinforcement Learning](https://mlsanity.com/paper/2010.14500/) (2020)
- [A Workflow for Offline Model-Free Robotic Reinforcement Learning](https://mlsanity.com/paper/2109.10813/) (2021)
- [Continual Learning for Real-World Autonomous Systems: Algorithms, Challenges and Frameworks](https://mlsanity.com/paper/2105.12374/) (2021)
- [Finetuning Offline World Models in the Real World](https://mlsanity.com/paper/2310.16029/) (2023)
- [Lifelong Robotic Reinforcement Learning by Retaining Experiences](https://mlsanity.com/paper/2109.09180/) (2021)
- [Don't Start From Scratch: Leveraging Prior Data to Automate Robotic Reinforcement Learning](https://mlsanity.com/paper/2207.04703/) (2022)
- [MoDem-V2: Visuo-Motor World Models for Real-World Robot Manipulation](https://mlsanity.com/paper/2309.14236/) (2023)
- [Bayesian Meta-Learning for Few-Shot Policy Adaptation Across Robotic Platforms](https://mlsanity.com/paper/2103.03697/) (2021)
- [Play it by Ear: Learning Skills amidst Occlusion through Audio-Visual Imitation Learning](https://mlsanity.com/paper/2205.14850/) (2022)
- [Overcoming the Sim-to-Real Gap: Leveraging Simulation to Learn to Explore for Real-World RL](https://mlsanity.com/paper/2410.20254/) (2024)
- [Learning to Extrapolate: A Transductive Approach](https://mlsanity.com/paper/2304.14329/) (2023)
- [Fast TRAC: A Parameter-Free Optimizer for Lifelong Reinforcement Learning](https://mlsanity.com/paper/2405.16642/) (2024)
- [On the Feasibility of Cross-Task Transfer with Model-Based Reinforcement Learning](https://mlsanity.com/paper/2210.10763/) (2022)
- [A Recipe for Unbounded Data Augmentation in Visual Reinforcement Learning](https://mlsanity.com/paper/2405.17416/) (2024)
- [Policy Transfer across Visual and Dynamics Domain Gaps via Iterative Grounding](https://mlsanity.com/paper/2107.00339/) (2021)
- [Learning Domain Invariant Representations in Goal-conditioned Block MDPs](https://mlsanity.com/paper/2110.14248/) (2021)
- [Policy Adaptation from Foundation Model Feedback](https://mlsanity.com/paper/2212.07398/) (2022)
- [Adaptive Visual Imitation Learning for Robotic Assisted Feeding Across Varied Bowl Configurations and Food Types](https://mlsanity.com/paper/2403.12891/) (2024)
- [Efficient Policy Adaptation with Contrastive Prompt Ensemble for Embodied Agents](https://mlsanity.com/paper/2412.11484/) (2024)
- [Contrastive Example-Based Control](https://mlsanity.com/paper/2307.13101/) (2023)
- [Machine Learning Meets Advanced Robotic Manipulation](https://mlsanity.com/paper/2309.12560/) (2023)
- [Task-agnostic Lifelong Robot Learning with Retrieval-based Weighted Local Adaptation](https://mlsanity.com/paper/2410.02995/) (2024)
- [Next-Future: Sample-Efficient Policy Learning for Robotic-Arm Tasks](https://mlsanity.com/paper/2504.11247/) (2025)

## Abstract (arXiv)

One of the great promises of robot learning systems is that they will be able to learn from their mistakes and continuously adapt to ever-changing environments. Despite this potential, most of the robot learning systems today are deployed as a fixed policy and they are not being adapted after their deployment. Can we efficiently adapt previously learned behaviors to new environments, objects and percepts in the real world? In this paper, we present a method and empirical evidence towards a robot learning framework that facilitates continuous adaption. In particular, we demonstrate how to adapt vision-based robotic manipulation policies to new variations by fine-tuning via off-policy reinforcement learning, including changes in background, object shape and appearance, lighting conditions, and robot morphology. Further, this adaptation uses less than 0.2% of the data necessary to learn the task from scratch. We find that our approach of adapting pre-trained policies leads to substantial performance gains over the course of fine-tuning, and that pre-training via RL is essential: training from scratch or adapting from supervised ImageNet features are both unsuccessful with such small amounts of data. We also find that these positive results hold in a limited continual learning setting, in which we repeatedly fine-tune a single lineage of policies using data from a succession of new tasks. Our empirical conclusions are consistently supported by experiments on simulated manipulation tasks, and by 52 unique fine-tuning experiments on a real robotic grasping system pre-trained on 580,000 grasps.