mlsanityData to 14 Oct 2026

The Surprising Ineffectiveness of Pre-Trained Visual Representations for Model-Based Reinforcement Learning

arXiv 2411.10175 ↗First posted 15 Nov 2024cs.LG cs.AI cs.CVcited by 7

On the map: Deep reinforcement learning methods → RL for robot locomotion and manipulation

Abstract (arXiv)

Visual Reinforcement Learning (RL) methods often require extensive amounts of data. As opposed to model-free RL, model-based RL (MBRL) offers a potential solution with efficient data utilization through planning. Additionally, RL lacks generalization capabilities for real-world tasks. Prior work has shown that incorporating pre-trained visual representations (PVRs) enhances sample efficiency and generalization. While PVRs have been extensively studied in the context of model-free RL, their potential in MBRL remains largely unexplored. In this paper, we benchmark a set of PVRs on challenging control tasks in a model-based RL setting. We investigate the data efficiency, generalization capabilities, and the impact of different properties of PVRs on the performance of model-based agents. Our results, perhaps surprisingly, reveal that for MBRL current PVRs are not more sample efficient than learning representations from scratch, and that they do not generalize better to out-of-distribution (OOD) settings. To explain this, we analyze the quality of the trained dynamics model. Furthermore, we show that data diversity and network architecture are the most important contributors to OOD generalization performance.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in RL for robot locomotion and manipulation, one of 15 directions in Deep reinforcement learning methods. Builds on 70 of its 81 references that are in the map. First posted on arXiv 15 Nov 2024.

Builds on

70 of 81 references.

References · 70

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  5. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  6. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  7. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  8. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  9. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  10. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
  11. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
  12. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
38 more
  1. 2018World Modelscited by 1,513
  2. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,353
  3. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,264
  4. 2023Mastering Diverse Domains through World Modelscited by 1,253
  5. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,175
  6. 2020Mastering Atari with Discrete World Modelscited by 1,007
  7. 2018DeepMind Control Suitecited by 956
  8. 2022A Generalist Agentcited by 808
  9. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  10. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 774
  11. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 760
  12. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 694
  13. 2020MOPO: Model-based Offline Policy Optimizationcited by 636
  14. 2018Taskonomy: Disentangling Task Transfer Learningcited by 634
  15. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 622
  16. 2020SAPIEN: A SimulAted Part-based Interactive ENvironmentcited by 595
  17. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 590
  18. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 585
  19. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 551
  20. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 512
  21. 2020Reinforcement Learning with Augmented Datacited by 468
  22. 2022Temporal Difference Learning for Model Predictive Controlcited by 397
  23. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 395
  24. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 372
  25. 2021COMBO: Conservative Offline Model-Based Policy Optimizationcited by 370
  26. 2020Planning to Explore via Self-Supervised World Modelscited by 368
  27. 2023ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skillscited by 327
  28. 2023Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Taskscited by 327
  29. 2015Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Imagescited by 274
  30. 2022Masked Visual Pre-training for Motor Controlcited by 248
  31. 2022Real-World Robot Learning with Masked Visual Pre-trainingcited by 245
  32. 2023Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?cited by 218
  33. 2019Deep Dynamics Models for Learning Dexterous Manipulationcited by 206
  34. 2018Revisiting Oxford and Paris: Large-Scale Image Retrieval Benchmarkingcited by 191
  35. 2021Simple but Effective: CLIP Embeddings for Embodied AIcited by 191
  36. 2023Foundation Models for Decision Making: Problems, Methods, and Opportunitiescited by 164
  37. 2022The Unsurprising Effectiveness of Pre-Trained Vision Models for Controlcited by 155
  38. 2020Generalization in Reinforcement Learning by Soft Data Augmentationcited by 137

50 most cited of 70 All 70 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 1.

7 papers.

  1. 2025TD-JEPA: Latent-predictive Representations for Zero-Shot Reinforcement Learningcited by 22
  2. 2025A Survey of State Representation Learning for Deep Reinforcement Learningcited by 17
  3. 2025Thousand-Brains Systems: Sensorimotor Intelligence for Rapid, Robust Learning and Inferencecited by 4
  4. 2025The Thousand Brains Theory 2.0: An Extension for the Long-Range Connections of the Neocortical Heterarchycited by 2
  5. 2025Combining Pre-Trained Models for Enhanced Feature Representation in Reinforcement Learningcited by 2
  6. 2025Pre-trained Visual Representations Generalize Where it Matters in Model-Based Reinforcement Learningcited by 0
  7. 2026Back to Parsimonious Latents: Learning Task-Centric World Models from Visual Foundationscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Pretrained Visual Representations in Reinforcement Learning
  2. 2024Pre-trained Visual Dynamics Representations for Efficient Policy Learning
  3. 2026Observation-Grounded Self-Predictive Reinforcement Learning for Visual Continuous Control
  4. 2026VIGOR: Zero-Shot Visual Generalization via Latent-Space Consistency in Model-Based Reinforcement Learning
  5. 2022Challenges and Opportunities in Offline Reinforcement Learning from Visual Observations
  6. 2025Reinforcement Learning for Large Model: A Survey
  7. 2026V-Simba: Unleashing the Architectural Potential of RL in Visual Continuous Control
  8. 2024R3L: Relative Representations for Reinforcement Learning