# Pre-trained Visual Representations Generalize Where it Matters in Model-Based Reinforcement Learning

arXiv: https://arxiv.org/abs/2509.12531 · HTML: https://mlsanity.com/paper/2509.12531/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL for robot locomotion and manipulation, one of 15 directions in Deep reinforcement learning methods.
- Builds on 32 of its 37 references that are in the map.
- First posted on arXiv 16 Sep 2025.

## Placed in

- Deep reinforcement learning methods / [RL for robot locomotion and manipulation](https://mlsanity.com/deeprl/rl-for-robot-locomotion-and-manipulation/)

## Builds on (most cited first, 30 of 32)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Mastering Diverse Domains through World Models](https://mlsanity.com/paper/2301.04104/) (2023)
- [End-to-End Training of Deep Visuomotor Policies](https://mlsanity.com/paper/1504.00702/) (2015)
- [Solving Rubik's Cube with a Robot Hand](https://mlsanity.com/paper/1910.07113/) (2019)
- [R3M: A Universal Visual Representation for Robot Manipulation](https://mlsanity.com/paper/2203.12601/) (2022)
- [CLIPort: What and Where Pathways for Robotic Manipulation](https://mlsanity.com/paper/2109.12098/) (2021)
- [Extract Free Dense Labels from CLIP](https://mlsanity.com/paper/2112.01071/) (2021)
- [TD-MPC2: Scalable, Robust World Models for Continuous Control](https://mlsanity.com/paper/2310.16828/) (2023)
- [DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning](https://mlsanity.com/paper/2411.04983/) (2024)
- [Quantifying Generalization in Reinforcement Learning](https://mlsanity.com/paper/1812.02341/) (2018)
- [Masked Visual Pre-training for Motor Control](https://mlsanity.com/paper/2203.06173/) (2022)
- [ManiSkill3: GPU Parallelized Robotics Simulation and Rendering for Generalizable Embodied AI](https://mlsanity.com/paper/2410.00425/) (2024)
- [Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?](https://mlsanity.com/paper/2303.18240/) (2023)
- [The Unsurprising Effectiveness of Pre-Trained Vision Models for Control](https://mlsanity.com/paper/2203.03580/) (2022)
- [What Would Elsa Do? Freezing Layers During Transformer Fine-Tuning](https://mlsanity.com/paper/1911.03090/) (2019)
- [Offline Visual Representation Learning for Embodied Navigation](https://mlsanity.com/paper/2204.13226/) (2022)
- [Improved OOD Generalization via Adversarial Training and Pre-training](https://mlsanity.com/paper/2105.11144/) (2021)
- [Pre-Trained Image Encoder for Generalizable Visual Reinforcement Learning](https://mlsanity.com/paper/2212.08860/) (2022)
- [An Unbiased Look at Datasets for Visuo-Motor Pre-Training](https://mlsanity.com/paper/2310.09289/) (2023)
- [What Makes Pre-Trained Visual Representations Successful for Robust Manipulation?](https://mlsanity.com/paper/2312.12444/) (2023)
- [Robust Policies via Mid-Level Visual Representations: An Experimental Study in Manipulation and Navigation](https://mlsanity.com/paper/2011.06698/) (2020)
- [RL-ViGen: A Reinforcement Learning Benchmark for Visual Generalization](https://mlsanity.com/paper/2307.10224/) (2023)
- [HFT: Half Fine-Tuning for Large Language Models](https://mlsanity.com/paper/2404.18466/) (2024)
- [The Benefits of Model-Based Generalization in Reinforcement Learning](https://mlsanity.com/paper/2211.02222/) (2022)
- [Partial Fine-Tuning: A Successor to Full Fine-Tuning for Vision Transformers](https://mlsanity.com/paper/2312.15681/) (2023)
- [The Surprising Ineffectiveness of Pre-Trained Visual Representations for Model-Based Reinforcement Learning](https://mlsanity.com/paper/2411.10175/) (2024)
- [A Data-Centric Revisit of Pre-Trained Vision Models for Robot Learning](https://mlsanity.com/paper/2503.06960/) (2025)

## Abstract (arXiv)

In visuomotor policy learning, the control policy for the robotic agent is derived directly from visual inputs. The typical approach, where a policy and vision encoder are trained jointly from scratch, generalizes poorly to novel visual scene changes. Using pre-trained vision models (PVMs) to inform a policy network improves robustness in model-free reinforcement learning (MFRL). Recent developments in Model-based reinforcement learning (MBRL) suggest that MBRL is more sample-efficient than MFRL. However, counterintuitively, existing work has found PVMs to be ineffective in MBRL. Here, we investigate PVM's effectiveness in MBRL, specifically on generalization under visual domain shifts. We show that, in scenarios with severe shifts, PVMs perform much better than a baseline model trained from scratch. We further investigate the effects of varying levels of fine-tuning of PVMs. Our results show that partial fine-tuning can maintain the highest average task performance under the most extreme distribution shifts. Our results demonstrate that PVMs are highly successful in promoting robustness in visual policy learning, providing compelling evidence for their wider adoption in model-based robotic learning applications.