mlsanityData to 7 Oct 2026

MoDem: Accelerating Visual Model-Based Reinforcement Learning with Demonstrations

arXiv 2212.05698 ↗First posted 12 Dec 2022cs.LG cs.AI cs.ROcited by 59

Not on the map. Why?

Abstract (arXiv)

Poor sample efficiency continues to be the primary challenge for deployment of deep Reinforcement Learning (RL) algorithms for real-world applications, and in particular for visuo-motor control. Model-based RL has the potential to be highly sample efficient by concurrently learning a world model and using synthetic rollouts for planning and policy improvement. However, in practice, sample-efficient learning with model-based RL is bottlenecked by the exploration challenge. In this work, we find that leveraging just a handful of demonstrations can dramatically improve the sample-efficiency of model-based RL. Simply appending demonstrations to the interaction dataset, however, does not suffice. We identify key ingredients for leveraging demonstrations in model learning -- policy pretraining, targeted exploration, and oversampling of demonstration data -- which forms the three phases of our model-based RL framework. We empirically study three complex visuo-motor control domains and find that our method is 150%-250% more successful in completing sparse reward tasks compared to prior approaches in the low data regime (100K interaction steps, 5 demonstrations). Code and videos are available at: https://nicklashansen.github.io/modemrl

Key facts. Cited by 59 papers in the map (data through 7 Oct 2026). Builds on 50 of its 70 references that are in the map. First posted on arXiv 12 Dec 2022.

Builds on

50 of 70 references.

References · 50

  1. 2015Continuous control with deep reinforcement learningcited by 3,279
  2. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  3. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  4. 2016Concrete Problems in AI Safetycited by 1,582
  5. 2016Generative Adversarial Imitation Learningcited by 1,576
  6. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  7. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  8. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  9. 2016End to End Learning for Self-Driving Carscited by 1,000
  10. 2020Mastering Atari with Discrete World Modelscited by 990
  11. 2018Exploration by Random Network Distillationcited by 966
  12. 2018DeepMind Control Suitecited by 935
38 more
  1. 2015End-to-End Training of Deep Visuomotor Policiescited by 847
  2. 2022A Generalist Agentcited by 802
  3. 2018Recurrent World Models Facilitate Policy Evolutioncited by 761
  4. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  5. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
  6. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  7. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  8. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  9. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  10. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 517
  11. 2018Behavioral Cloning from Observationcited by 482
  12. 2020Reinforcement Learning with Augmented Datacited by 452
  13. 2022Temporal Difference Learning for Model Predictive Controlcited by 375
  14. 2022DayDreamer: World Models for Physical Robot Learningcited by 372
  15. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 359
  16. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 338
  17. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 322
  18. 2017Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperationcited by 286
  19. 2022Masked Visual Pre-training for Motor Controlcited by 246
  20. 2017One-Shot Imitation Learningcited by 232
  21. 2018Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Controlcited by 220
  22. 2019Go-Explore: a New Approach for Hard-Exploration Problemscited by 207
  23. 2022Masked World Models for Visual Controlcited by 169
  24. 2022The Unsurprising Effectiveness of Pre-Trained Vision Models for Controlcited by 155
  25. 2021The Surprising Effectiveness of Representation Learning for Visual Imitationcited by 149
  26. 2015Model Predictive Path Integral Control using Covariance Variable Importance Samplingcited by 122
  27. 2019End-to-End Robotic Reinforcement Learning without Reward Engineeringcited by 115
  28. 2020Visual Imitation Made Easycited by 111
  29. 2019Grasping in the Wild:Learning 6DoF Closed-Loop Grasping from Low-Cost Demonstrationscited by 104
  30. 2021XIRL: Cross-embodiment Inverse Reinforcement Learningcited by 101
  31. 2021Stabilizing Deep Q-Learning with ConvNets and Vision Transformers under Data Augmentationcited by 101
  32. 2020State-Only Imitation Learning for Dexterous Manipulationcited by 90
  33. 2021RRL: Resnet as representation for Reinforcement Learningcited by 89
  34. 2018Multiple Interactions Made Easy (MIME): Large Scale Demonstrations Data for Imitationcited by 76
  35. 2022VRL3: A Data-Driven Framework for Visual Deep Reinforcement Learningcited by 51
  36. 2022Graph Inverse Reinforcement Learning from Diverse Videoscited by 45
  37. 2022On the Feasibility of Cross-Task Transfer with Model-Based Reinforcement Learningcited by 15
  38. 2018SOLAR: Deep Structured Representations for Model-Based Reinforcement Learningcited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 19; 2024: 19; 2025: 12; 2026: 9.

59 papers.

Sort:

By area:

  1. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 482
  2. 2023Efficient Online Reinforcement Learning with Offline Datacited by 332
  3. 2023Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?cited by 215
  4. 2023Structured World Models from Human Videoscited by 144
  5. 20243D Diffusion Policy: Generalizable Visuomotor Policy Learning via Simple 3D Representationscited by 115
  6. 2024Learning and Leveraging World Models in Visual Representation Learningcited by 71
  7. 2023Imitating Task and Motion Planning with Visuomotor Transformerscited by 63
  8. 2023TorchRL: A data-driven decision-making library for PyTorchcited by 59
  9. 2023Imitation Bootstrapped Reinforcement Learningcited by 54
  10. 2023Masked Trajectory Models for Prediction, Representation, and Controlcited by 40
  11. 2023Finetuning Offline World Models in the Real Worldcited by 39
  12. 2023Model-Based Runtime Monitoring with Interactive Imitation Learningcited by 37
38 more
  1. 2024ViViDex: Learning Vision-based Dexterous Manipulation from Human Videoscited by 33
  2. 2024Diffusion World Model: Future Modeling Beyond Step-by-Step Rollout for Offline Reinforcement Learningcited by 31
  3. 2025EXPO: Stable Reinforcement Learning with Expressive Policiescited by 30
  4. 2023MoDem-V2: Visuo-Motor World Models for Real-World Robot Manipulationcited by 27
  5. 2024Hierarchical World Models as Visual Whole-Body Humanoid Controllerscited by 24
  6. 2025AdaWM: Adaptive World Model based Planning for Autonomous Drivingcited by 23
  7. 2024RH20T-P: A Primitive-Level Robotic Dataset Towards Composable Generalization Agentscited by 20
  8. 2024Learning Latent Dynamic Robust Representations for World Modelscited by 20
  9. 2025Physics-Driven Data Generation for Contact-Rich Manipulation via Trajectory Optimizationcited by 20
  10. 2023RePo: Resilient Model-Based Reinforcement Learning by Regularizing Posterior Predictabilitycited by 17
  11. 2024So You Think You Can Scale Up Autonomous Robot Data Collection?cited by 17
  12. 2023Blending Imitation and Reinforcement Learning for Robust Policy Improvementcited by 16
  13. 2023MoVie: Visual Model-Based Policy Adaptation for View Generalizationcited by 15
  14. 2024Continuous Control with Coarse-to-fine Reinforcement Learningcited by 15
  15. 2024Point Cloud Models Improve Visual Robustness in Robotic Learnerscited by 13
  16. 2024Reverse Forward Curriculum Learning for Extreme Sample and Demonstration Efficiency in Reinforcement Learningcited by 12
  17. 2025Reinforcement Learning via Implicit Imitation Guidancecited by 12
  18. 2024Efficient Imitation Learning with Conservative World Modelscited by 11
  19. 2025Learning Massively Multitask World Models for Continuous Controlcited by 11
  20. 2023Goal-Conditioned Predictive Coding for Offline Reinforcement Learningcited by 10
  21. 2025Multi-Stage Manipulation with Demonstration-Augmented Reward, Policy, and World Model Learningcited by 10
  22. 2024World Models for General Surgical Graspingcited by 9
  23. 2025Reinforcement Learning for Large Model: A Surveycited by 8
  24. 2024The Surprising Ineffectiveness of Pre-Trained Visual Representations for Model-Based Reinforcement Learningcited by 7
  25. 2025SOMBRL: Scalable and Optimistic Model-Based RLcited by 7
  26. 2024DEER: A Delay-Resilient Framework for Reinforcement Learning with Variable Delayscited by 6
  27. 2024Learning World Models for Unconstrained Goal Navigationcited by 6
  28. 2025TDMPBC: Self-Imitative Reinforcement Learning for Humanoid Robot Controlcited by 6
  29. 2023Bridging Imitation and Online Reinforcement Learning: An Optimistic Talecited by 5
  30. 2023Efficient Online Learning with Offline Datasets for Infinite Horizon MDPs: A Bayesian Approachcited by 3
  31. 2024Competence-Aware AI Agents with Metacognition for Unknown Situations and Environments (MUSE)cited by 3
  32. 2025Sample-efficient and Scalable Exploration in Continuous-Time RLcited by 3
  33. 2023Continual Visual Reinforcement Learning with A Life-Long World Modelcited by 2
  34. 2023H-InDex: Visual Reinforcement Learning with Hand-Informed Representations for Dexterous Manipulationcited by 1
  35. 2024Exploring the Edges of Latent State Clusters for Goal-Conditioned Reinforcement Learningcited by 1
  36. 2026Learning from Demonstrations via Capability-Aware Goal Samplingcited by 1
  37. 2026Model-Based Reinforcement Learning for Control under Time-Varying Dynamicscited by 1
  38. 2024ATraDiff: Accelerating Online Reinforcement Learning with Imaginary Trajectoriescited by 0

50 most cited of 59 All 59 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Sample Efficient Reinforcement Learning via Large Vision Language Model Distillation
  2. 2018Reinforcement and Imitation Learning for Diverse Visuomotor Skills
  3. 2025Beyond Human Demonstrations: Diffusion-Based Reinforcement Learning to Generate Data for VLA Training
  4. 2025Accelerating Model-Based Reinforcement Learning with State-Space World Models
  5. 2026Rainbow-DemoRL: Combining Improvements in Demonstration-Augmented Reinforcement Learning
  6. 2024PWM: Policy Learning with Multi-Task World Models
  7. 2021Efficiently Training On-Policy Actor-Critic Networks in Robotic Deep Reinforcement Learning with Demonstration-like Sampled Exploration
  8. 2020Exploration-efficient Deep Reinforcement Learning with Demonstration Guidance for Robot Control