mlsanityData to 7 Oct 2026

Finetuning Offline World Models in the Real World

arXiv 2310.16029 ↗First posted 24 Oct 2023cs.LG cs.AI cs.CVcited by 39

On the map: World models → Latent dynamics models and model-based RL

Abstract (arXiv)

Reinforcement Learning (RL) is notoriously data-inefficient, which makes training on a real robot difficult. While model-based RL algorithms (world models) improve data-efficiency to some extent, they still require hours or days of interaction to learn skills. Recently, offline RL has been proposed as a framework for training RL policies on pre-existing datasets without any online interaction. However, constraining an algorithm to a fixed dataset induces a state-action distribution shift between training and inference, and limits its applicability to new tasks. In this work, we seek to get the best of both worlds: we consider the problem of pretraining a world model with offline data collected on a real robot, and then finetuning the model on online data collected by planning with the learned model. To mitigate extrapolation errors during online interaction, we propose to regularize the planner at test-time by balancing estimated returns and (epistemic) model uncertainty. We evaluate our method on a variety of visuo-motor control tasks in simulation and on a real robot, and find that our method enables few-shot finetuning to seen and unseen tasks even when offline data is limited. Videos, code, and data are available at https://yunhaifeng.com/FOWM .

Key facts. Cited by 39 papers in the map (data through 7 Oct 2026). Placed in Latent dynamics models and model-based RL, one of 11 directions in World models. Builds on 48 of its 53 references that are in the map. First posted on arXiv 24 Oct 2023.

Builds on

48 of 53 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022DayDreamer: World Models for Physical Robot Learning28 Jun
  2. 2018Recurrent World Models Facilitate Policy Evolution4 Sep

Other references · 46

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  3. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  4. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  5. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
  6. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  7. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  8. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  9. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  10. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  11. 2015Prioritized Experience Replaycited by 1,070
  12. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
34 more
  1. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  2. 2022A Generalist Agentcited by 802
  3. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  4. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  5. 2019Behavior Regularized Offline Reinforcement Learningcited by 654
  6. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  7. 2018Learning Dexterous In-Hand Manipulationcited by 643
  8. 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 616
  9. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 582
  10. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  11. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 517
  12. 2021RMA: Rapid Motor Adaptation for Legged Robotscited by 430
  13. 2020Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Surveycited by 388
  14. 2022Temporal Difference Learning for Model Predictive Controlcited by 375
  15. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 338
  16. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 278
  17. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 230
  18. 2021MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scalecited by 226
  19. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemblecited by 211
  20. 2017Asymmetric Actor Critic for Image-Based Robot Learningcited by 202
  21. 2015Model Predictive Path Integral Control using Covariance Variable Importance Samplingcited by 122
  22. 2020Self-Supervised Policy Adaptation during Deploymentcited by 121
  23. 2021Stabilizing Deep Q-Learning with ConvNets and Vision Transformers under Data Augmentationcited by 101
  24. 2022Don't Change the Algorithm, Change the Data: Exploratory Data for Offline Reinforcement Learningcited by 93
  25. 2022Pre-Training for Robots: Offline RL Enables Learning New Tasks from a Handful of Trialscited by 78
  26. 2022MoDem: Accelerating Visual Model-Based Reinforcement Learning with Demonstrationscited by 59
  27. 2022VRL3: A Data-Driven Framework for Visual Deep Reinforcement Learningcited by 51
  28. 2022Look Closer: Bridging Egocentric and Third-Person Views with Transformers for Robotic Manipulationcited by 49
  29. 2022Visual Reinforcement Learning with Self-Supervised 3D Representationscited by 45
  30. 2021Adaptive-Control-Oriented Meta-Learning for Nonlinear Systemscited by 38
  31. 2019An Optimistic Perspective on Offline Reinforcement Learningcited by 37
  32. 2022Disentangling Epistemic and Aleatoric Uncertainty in Reinforcement Learningcited by 27
  33. 2020Never Stop Learning: The Effectiveness of Fine-Tuning in Robotic Reinforcement Learningcited by 23
  34. 2022On the Feasibility of Cross-Task Transfer with Model-Based Reinforcement Learningcited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 15; 2025: 10; 2026: 13.

39 papers.

Sort:

By area:

  1. 2024RDT-1B: a Diffusion Foundation Model for Bimanual Manipulationcited by 670
  2. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 482
  3. 20243D-VLA: A 3D Vision-Language-Action Generative World Modelcited by 320
  4. 2024ManiSkill3: GPU Parallelized Robotics Simulation and Rendering for Generalizable Embodied AIcited by 216
  5. 2024SpatialBot: Precise Spatial Understanding with Vision Language Modelscited by 146
  6. 2024iVideoGPT: Interactive VideoGPTs are Scalable World Modelscited by 136
  7. 2024Reconciling Reality through Simulation: A Real-to-Sim-to-Real Approach for Robust Manipulationcited by 135
  8. 2025A Survey: Learning Embodied Intelligence from Physical Simulators and World Modelscited by 50
  9. 2025Robotic World Model: A Neural Network Simulator for Robust Policy Optimization in Roboticscited by 47
  10. 2024Hierarchical World Models as Visual Whole-Body Humanoid Controllerscited by 24
  11. 2024ActSafe: Active Exploration with Safety Constraints for Reinforcement Learningcited by 22
  12. 2024Contrastive Representation for Data Filtering in Cross-Domain Offline Reinforcement Learningcited by 18
27 more
  1. 2025XR-1: Towards Versatile Vision-Language-Action Models via Learning Unified Vision-Motion Representationscited by 15
  2. 2026World Action Verifier: Self-Improving World Models via Forward-Inverse Asymmetrycited by 13
  3. 2025A Survey on Imitation Learning for Contact-Rich Tasks in Roboticscited by 12
  4. 2024Efficient Imitation Learning with Conservative World Modelscited by 11
  5. 2025Learning Massively Multitask World Models for Continuous Controlcited by 11
  6. 2025DMWM: Dual-Mind World Model with Long-Term Imaginationcited by 10
  7. 2026Feedback World Model Enables Precise Guidance of Diffusion Policycited by 9
  8. 2024The Surprising Ineffectiveness of Pre-Trained Visual Representations for Model-Based Reinforcement Learningcited by 7
  9. 2024Bisimulation metric for Model Predictive Controlcited by 6
  10. 2026Overcoming Statistical Bias in Action-Controllable World Modelscited by 4
  11. 2024FOSP: Fine-tuning Offline Safe Policy through World Modelscited by 3
  12. 2026Morphology-Conditioned World Model for Cross-Embodiment Quadrupedal Locomotioncited by 3
  13. 2026Composition of Memory Experts for Diffusion World Modelscited by 3
  14. 2024A Simulation Benchmark for Autonomous Racing with Large-Scale Human Datacited by 2
  15. 2026What Matters for Simulation to Online Reinforcement Learning on Real Robotscited by 2
  16. 2024A General Control-Theoretic Approach for Reinforcement Learning: Theory and Algorithmscited by 1
  17. 2025Pretraining a Shared Q-Network for Data-Efficient Offline Reinforcement Learningcited by 1
  18. 2025Belief-Based Offline Reinforcement Learning for Delay-Robust Policy Optimizationcited by 1
  19. 2025A Recipe for Efficient Sim-to-Real Transfer in Manipulation with Online Imitation-Pretrained World Modelscited by 1
  20. 2025Real-World Reinforcement Learning of Active Perception Behaviorscited by 1
  21. 2026Dream-MPC: Gradient-Based Model Predictive Control with Latent Imaginationcited by 1
  22. 2026Measuring the Value of World-Model Updates: A Counterfactual Utility Protocol for Continual Adaptationcited by 1
  23. 2026Knowledge-Guided Manipulation Using Multi-Task Reinforcement Learningcited by 0
  24. 2026Learning Task-Invariant Properties via Dreamer: Enabling Efficient Policy Transfer for Quadruped Robotscited by 0
  25. 2026Reward-Free Continual Adaptation for Resilient Space Robotscited by 0
  26. 2026Generalizable Robotic Insertion with World Modelscited by 0
  27. 2026JEPA-TTT: Persistent Test-Time Training of Latent World Models for Planning under Dynamics Shiftscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024MOTO: Offline Pre-training to Online Fine-tuning for Model-based Robot Learning
  2. 2023Finetuning from Offline Reinforcement Learning: Challenges, Trade-offs and Practical Solutions
  3. 2025Uncertainty-Aware Robotic World Model Makes Offline Model-Based Reinforcement Learning Work on Real Robots
  4. 2025Efficient Reinforcement Learning by Guiding World Models with Non-Curated Data
  5. 2021A Workflow for Offline Model-Free Robotic Reinforcement Learning
  6. 2023Small Dataset, Big Gains: Enhancing Reinforcement Learning by Offline Pre-Training with Model Based Augmentation
  7. 2025Offline vs. Online Learning in Model-based RL: Lessons for Data Collection Strategies
  8. 2022Leveraging Offline Data in Online Reinforcement Learning