mlsanityData to 7 Oct 2026

What do we learn from a large-scale study of pre-trained visual representations in sim and real environments?

arXiv 2310.02219 ↗First posted 3 Oct 2023cs.RO cs.AI cs.CVcited by 11

Not on the map. Why?

Abstract (arXiv)

We present a large empirical investigation on the use of pre-trained visual representations (PVRs) for training downstream policies that execute real-world tasks. Our study involves five different PVRs, each trained for five distinct manipulation or indoor navigation tasks. We performed this evaluation using three different robots and two different policy learning paradigms. From this effort, we can arrive at three insights: 1) the performance trends of PVRs in the simulation are generally indicative of their trends in the real world, 2) the use of PVRs enables a first-of-its-kind result with indoor ImageNav (zero-shot transfer to a held-out scene in the real world), and 3) the benefits from variations in PVRs, primarily data-augmentation and fine-tuning, also transfer to the real-world performance. See project website for additional details and visuals.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 24 of its 33 references that are in the map. First posted on arXiv 3 Oct 2023.

Builds on

24 of 33 references.

References · 24

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  4. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  5. 2021Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AIcited by 579
  6. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 384
  7. 2016Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learningcited by 351
  8. 2017Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metricscited by 259
  9. 2022Real-World Robot Learning with Masked Visual Pre-trainingcited by 241
  10. 2023Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?cited by 215
  11. 2021Contact-GraspNet: Efficient 6-DoF Grasp Generation in Cluttered Scenescited by 213
  12. 2021Simple but Effective: CLIP Embeddings for Embodied AIcited by 191
12 more
  1. 2018Sim-to-Real via Sim-to-Sim: Data-efficient Robotic Grasping via Randomized-to-Canonical Adaptation Networkscited by 171
  2. 2023RoboAgent: Generalization and Efficiency in Robot Manipulation via Semantic Augmentations and Action Chunkingcited by 167
  3. 2022The Unsurprising Effectiveness of Pre-Trained Vision Models for Controlcited by 155
  4. 2022Navigating to Objects in the Real Worldcited by 104
  5. 2020RL-CycleGAN: Reinforcement Learning Aware Simulation-To-Realcited by 89
  6. 2022Offline Visual Representation Learning for Embodied Navigationcited by 74
  7. 2020RetinaGAN: An Object-aware Approach to Sim-to-Real Transfercited by 68
  8. 2023OVRL-V2: A simple state-of-art baseline for ImageNav and ObjectNavcited by 68
  9. 2022Zero Experience Required: Plug & Play Modular Transfer Learning for Semantic Visual Navigationcited by 66
  10. 2023Navigating to Objects Specified by Imagescited by 50
  11. 2020Sim2Real Transfer for Reinforcement Learning without Dynamics Randomizationcited by 43
  12. 2020TriFinger: An Open-Source Robot for Learning Dexteritycited by 31

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 10.

11 papers.

  1. 2025Embodied AI Agents: Modeling the Worldcited by 57
  2. 2024Towards Generalist Robot Learning from Internet Video: A Surveycited by 40
  3. 2025The Reality Gap in Robotics: Challenges, Solutions, and Best Practicescited by 22
  4. 2025TD-JEPA: Latent-predictive Representations for Zero-Shot Reinforcement Learningcited by 17
  5. 2025SIMS-V: Simulated Instruction-Tuning for Spatial Video Understandingcited by 16
  6. 2025Real-is-Sim: Bridging the Sim-to-Real Gap with a Dynamic Digital Twincited by 11
  7. 2025EmbodiedSplat: Personalized Real-to-Sim-to-Real Navigation with Gaussian Splats from a Mobile Devicecited by 11
  8. 2025Accelerating Model-Based Reinforcement Learning with State-Space World Modelscited by 10
  9. 2025Bridging the Sim2Real Gap: Vision Encoder Pre-Training for Visuomotor Policy Transfercited by 2
  10. 2025Attentive Feature Aggregation or: How Policies Learn to Stop Worrying about Robustness and Attend to Task-Relevant Visual Cuescited by 2
  11. 2025Synthetic vs. Real Training Data for Visual Navigationcited by 1