mlsanityData to 7 Oct 2026

MOTO: Offline Pre-training to Online Fine-tuning for Model-based Robot Learning

arXiv 2401.03306 ↗First posted 6 Jan 2024cs.LG cs.AI cs.ROcited by 13

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

We study the problem of offline pre-training and online fine-tuning for reinforcement learning from high-dimensional observations in the context of realistic robot tasks. Recent offline model-free approaches successfully use online fine-tuning to either improve the performance of the agent over the data collection policy or adapt to novel tasks. At the same time, model-based RL algorithms have achieved significant progress in sample efficiency and the complexity of the tasks they can solve, yet remain under-utilized in the fine-tuning setting. In this work, we argue that existing model-based offline RL methods are not suitable for offline-to-online fine-tuning in high-dimensional domains due to issues with distribution shifts, off-dynamics data, and non-stationary rewards. We propose an on-policy model-based method that can efficiently reuse prior data through model-based value expansion and policy regularization, while preventing model exploitation by controlling epistemic uncertainty. We find that our approach successfully solves tasks from the MetaWorld benchmark, as well as the Franka Kitchen robot manipulation environment completely from images. To the best of our knowledge, MOTO is the first method to solve this environment from pixels.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 50 of its 56 references that are in the map. First posted on arXiv 6 Jan 2024.

Builds on

50 of 56 references.

Closest ancestors · 12

Same direction, newest first

  1. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  2. 2021Conservative Data Sharing for Multi-Task Offline Reinforcement Learning16 Sep
  3. 2021Behavioral Priors and Dynamics Models: Improving Performance and Domain Transfer in Offline RL16 Jun
  4. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  5. 2021Representation Matters: Offline Pretraining for Sequential Decision Making11 Feb
  6. 2020Offline Reinforcement Learning from Images with Latent Space Models21 Dec
  7. 2020Overcoming Model Bias for Robust Offline Deep Reinforcement Learning12 Aug
  8. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  9. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  10. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  11. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  12. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning19 Feb

Other references · 38

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2017CARLA: An Open Urban Driving Simulatorcited by 2,431
  3. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  4. 2016Generative Adversarial Imitation Learningcited by 1,576
  5. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  6. 2018World Modelscited by 1,444
  7. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  8. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  9. 2020Mastering Atari with Discrete World Modelscited by 990
  10. 2022A Generalist Agentcited by 802
  11. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  12. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
26 more
  1. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  2. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learningcited by 380
  3. 2020Planning to Explore via Self-Supervised World Modelscited by 365
  4. 2015Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Imagescited by 266
  5. 2021FIERY: Future Instance Prediction in Bird's-Eye View from Surround Monocular Camerascited by 239
  6. 2019Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Modelcited by 228
  7. 2018Model-Ensemble Trust-Region Policy Optimizationcited by 214
  8. 2019Deep Dynamics Models for Learning Dexterous Manipulationcited by 200
  9. 2018Distributed Distributional Deterministic Policy Gradientscited by 195
  10. 2019Tighter Problem-Dependent Regret Bounds in Reinforcement Learning without Domain Knowledge using Value Function Boundscited by 194
  11. 2018Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansioncited by 167
  12. 2018Model-Based Value Estimation for Efficient Model-Free Reinforcement Learningcited by 152
  13. 2015Deep Spatial Autoencoders for Visuomotor Learningcited by 149
  14. 2022Model-Based Imitation Learning for Urban Drivingcited by 149
  15. 2018Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guaranteescited by 138
  16. 2020Model-Based Offline Planningcited by 126
  17. 2022Imitation Is Not Enough: Robustifying Imitation with Reinforcement Learning for Challenging Driving Scenarioscited by 124
  18. 2020Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimizationcited by 120
  19. 2018Model-Based Reinforcement Learning via Meta-Policy Optimizationcited by 99
  20. 2020Model-Augmented Actor-Critic: Backpropagating through Pathscited by 61
  21. 2022StretchBEV: Stretching Future Instance Prediction Spatially and Temporallycited by 47
  22. 2020On the model-based stochastic value gradient for continuous reinforcement learningcited by 46
  23. 2022Hierarchical Model-Based Imitation Learning for Planning in Autonomous Drivingcited by 43
  24. 2021Visual Adversarial Imitation Learning using Variational Modelscited by 42
  25. 2021Mitigating Covariate Shift in Imitation Learning via Offline Data Without Great Coveragecited by 26
  26. 2022Discriminator-Guided Model-Based Offline Imitation Learningcited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 6; 2026: 1.

13 papers.

Sort:

By area:

  1. 2025ConRFT: A Reinforced Fine-tuning Method for VLA Models via Consistency Policycited by 139
  2. 2024Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Datacited by 58
  3. 2024Energy-Guided Diffusion Sampling for Offline-to-Online Reinforcement Learningcited by 17
  4. 2025A Survey on Imitation Learning for Contact-Rich Tasks in Roboticscited by 12
  5. 2024Efficient Imitation Learning with Conservative World Modelscited by 11
  6. 2024The Evolution of Reinforcement Learning in Quantitative Finance: A Surveycited by 11
  7. 2024SELFI: Autonomous Self-Improvement with Reinforcement Learning for Social Navigationcited by 8
  8. 2025Temporal Distance-aware Transition Augmentation for Offline Model-based Reinforcement Learningcited by 5
  9. 2024A General Control-Theoretic Approach for Reinforcement Learning: Theory and Algorithmscited by 1
  10. 2025Efficient Reinforcement Learning by Guiding World Models with Non-Curated Datacited by 1
  11. 2025Pretraining a Shared Q-Network for Data-Efficient Offline Reinforcement Learningcited by 1
  12. 2025Behavior-Adaptive Q-Learning: A Unifying Framework for Offline-to-Online RLcited by 1
1 more
  1. 2026ProDVI: Programmatic Dynamics Priors for Value Network Initializationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Finetuning Offline World Models in the Real World
  2. 2021A Workflow for Offline Model-Free Robotic Reinforcement Learning
  3. 2025Uncertainty-Aware Robotic World Model Makes Offline Model-Based Reinforcement Learning Work on Real Robots
  4. 2023Robot Fine-Tuning Made Easy: Pre-Training Rewards and Policies for Autonomous Real-World Reinforcement Learning
  5. 2023Small Dataset, Big Gains: Enhancing Reinforcement Learning by Offline Pre-Training with Model Based Augmentation
  6. 2020MOPO: Model-based Offline Policy Optimization
  7. 2024FOSP: Fine-tuning Offline Safe Policy through World Models
  8. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble