mlsanityData to 7 Oct 2026

Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?

arXiv 2303.18240 ↗First posted 31 Mar 2023cs.CV cs.AI cs.LGcited by 215

Not on the map. Why?

Cited by 30 in Vision-language-action models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present the largest and most comprehensive empirical study of pre-trained visual representations (PVRs) or visual 'foundation models' for Embodied AI. First, we curate CortexBench, consisting of 17 different tasks spanning locomotion, navigation, dexterous, and mobile manipulation. Next, we systematically evaluate existing PVRs and find that none are universally dominant. To study the effect of pre-training data size and diversity, we combine over 4,000 hours of egocentric videos from 7 different sources (over 4.3M images) and ImageNet to train different-sized vision transformers using Masked Auto-Encoding (MAE) on slices of this data. Contrary to inferences from prior work, we find that scaling dataset size and diversity does not improve performance universally (but does so on average). Our largest model, named VC-1, outperforms all prior PVRs on average but does not universally dominate either. Next, we show that task- or domain-specific adaptation of VC-1 leads to substantial gains, with VC-1 (adapted) achieving competitive or superior performance than the best known results on all of the benchmarks in CortexBench. Finally, we present real-world hardware experiments, in which VC-1 and VC-1 (adapted) outperform the strongest pre-existing PVR. Overall, this paper presents no new techniques but a rigorous systematic evaluation, a broad set of findings about PVRs (that in some cases, refute those made in narrow domains in prior work), and open-sourced code and models (that required over 10,000 GPU-hours to train) for the benefit of the research community.

Key facts. Cited by 215 papers in the map (data through 7 Oct 2026). Builds on 60 of its 83 references that are in the map. First posted on arXiv 31 Mar 2023.

Builds on

60 of 83 references.

References · 60

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  5. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  6. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  7. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  8. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  9. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  10. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  11. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  12. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
38 more
  1. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  2. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  3. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  4. 2021Scaling Vision Transformerscited by 993
  5. 2018DeepMind Control Suitecited by 935
  6. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 789
  7. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 722
  8. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  9. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  10. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  11. 2021Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AIcited by 579
  12. 2021Habitat 2.0: Training Home Assistants to Rearrange their Habitatcited by 501
  13. 2018Gibson Env: Real-World Perception for Embodied Agentscited by 477
  14. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 384
  15. 2022Last Layer Re-Training is Sufficient for Robustness to Spurious Correlationscited by 379
  16. 2022ProcTHOR: Large-Scale Embodied AI Using Procedural Generationcited by 369
  17. 2019DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Framescited by 352
  18. 2016Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learningcited by 351
  19. 2020Understanding Human Hands in Contact at Internet Scalecited by 281
  20. 2022Real-World Robot Learning with Masked Visual Pre-trainingcited by 241
  21. 2020ObjectNav Revisited: On Evaluation of Embodied Agents Navigating to Objectscited by 240
  22. 2022Surgical Fine-Tuning Improves Adaptation to Distribution Shiftscited by 205
  23. 2021Self-supervised Pretraining of Visual Features in the Wildcited by 197
  24. 2021Simple but Effective: CLIP Embeddings for Embodied AIcited by 191
  25. 2023Language-Driven Representation Learning for Roboticscited by 163
  26. 2022The Unsurprising Effectiveness of Pre-Trained Vision Models for Controlcited by 155
  27. 2023Open-World Object Manipulation using Pre-trained Vision-Language Modelscited by 155
  28. 2021The Surprising Effectiveness of Representation Learning for Visual Imitationcited by 149
  29. 2022Finetune like you pretrain: Improved finetuning of zero-shot vision modelscited by 149
  30. 2022Habitat-Web: Learning Embodied Object-Search Strategies from Human Demonstrations at Scalecited by 116
  31. 2022Habitat-Matterport 3D Semantics Datasetcited by 109
  32. 2022Efficient Self-supervised Learning with Contextualized Target Representations for Vision, Speech and Languagecited by 108
  33. 2023PIRLNav: Pretraining with Imitation and RL Finetuning for ObjectNavcited by 91
  34. 2021Divide and Contrast: Self-supervised Learning from Uncurated Datacited by 85
  35. 2018Robot Learning in Homes: Improving Generalization and Reducing Dataset Biascited by 75
  36. 2022Offline Visual Representation Learning for Embodied Navigationcited by 74
  37. 2022CACTI: A Framework for Scalable Multi-Task Multi-Scene Visual Imitation Learningcited by 70
  38. 2023OVRL-V2: A simple state-of-art baseline for ImageNav and ObjectNavcited by 68

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 36; 2024: 67; 2025: 67; 2026: 45.

215 papers.

Sort:

By area:

  1. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  2. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,653
  3. 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
  4. 2025Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Successcited by 844
  5. 2025CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Modelscited by 440
  6. 2023An Embodied Generalist Agent in 3D Worldcited by 355
  7. 2024Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representationscited by 279
  8. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  9. 2023ViNT: A Foundation Model for Visual Navigationcited by 210
  10. 2023LIV: Language-Image Representations and Rewards for Robotic Controlcited by 180
  11. 2023RoboAgent: Generalization and Efficiency in Robot Manipulation via Semantic Augmentations and Action Chunkingcited by 167
  12. 2024HumanPlus: Humanoid Shadowing and Imitation from Humanscited by 164
38 more
  1. 2024Scaling Proprioceptive-Visual Learning with Heterogeneous Pre-trained Transformerscited by 145
  2. 2024Gen2Act: Human Video Generation in Novel Scenarios enables Generalizable Robot Manipulationcited by 142
  3. 2024Multimodal Diffusion Transformer: Learning Versatile Behavior from Multimodal Goalscited by 131
  4. 2024Flow as the Cross-Domain Manipulation Interfacecited by 126
  5. 2024Track2Act: Predicting Point Tracks from Internet Videos enables Generalizable Robot Manipulationcited by 116
  6. 2025EgoVLA: Learning Vision-Language-Action Models from Egocentric Human Videoscited by 116
  7. 2025GR-3 Technical Reportcited by 110
  8. 2023RoboCat: A Self-Improving Generalist Agent for Robotic Manipulationcited by 108
  9. 2023On Bringing Robots Homecited by 101
  10. 2023Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysiscited by 99
  11. 2023Large Language Models as Generalizable Policies for Embodied Taskscited by 86
  12. 2024General Flow as Foundation Affordance for Scalable Robot Learningcited by 80
  13. 2024Unifying 3D Vision-Language Understanding via Promptable Queriescited by 70
  14. 2024IGOR: Image-GOal Representations are the Atomic Control Units for Foundation Models in Embodied AIcited by 68
  15. 2024In-Context Imitation Learning via Next-Token Predictioncited by 65
  16. 2024Prediction with Action: Visual Policy Learning via Joint Denoising Processcited by 64
  17. 2024Embodied Understanding of Driving Scenarioscited by 63
  18. 2024Real-World Robot Applications of Foundation Models: A Reviewcited by 60
  19. 2025Embodied AI Agents: Modeling the Worldcited by 57
  20. 2024Theia: Distilling Diverse Vision Foundation Models for Robot Learningcited by 56
  21. 2023QUAR-VLA: Vision-Language-Action Model for Quadruped Robotscited by 55
  22. 2024PoliFormer: Scaling On-Policy RL with Transformers Results in Masterful Navigatorscited by 55
  23. 2025Training Strategies for Efficient Embodied Reasoningcited by 53
  24. 2023Towards Generalizable Zero-Shot Manipulation via Translating Human Interaction Planscited by 51
  25. 2023TAIL: Task-specific Adapters for Imitation Learning with Large Pretrained Modelscited by 48
  26. 2023SPOC: Imitating Shortest Paths in Simulation Enables Effective Navigation and Manipulation in the Real Worldcited by 47
  27. 2025RL-100: Performant Robotic Manipulation with Real-World Reinforcement Learningcited by 47
  28. 2025Pure Vision Language Action (VLA) Models: A Comprehensive Surveycited by 46
  29. 2023Pre-training Contextualized World Models with In-the-wild Videos for Reinforcement Learningcited by 45
  30. 2024Point Cloud Matters: Rethinking the Impact of Different Observation Spaces on Robot Learningcited by 45
  31. 2024Lift3D Foundation Policy: Lifting 2D Large-Scale Pretrained Models for Robust 3D Robotic Manipulationcited by 45
  32. 2024DynaMo: In-Domain Dynamics Pretraining for Visuo-Motor Controlcited by 43
  33. 2023An Unbiased Look at Datasets for Visuo-Motor Pre-Trainingcited by 41
  34. 2024HRP: Human Affordances for Robotic Pre-Trainingcited by 40
  35. 2024GR-MG: Leveraging Partially Annotated Data via Multi-Modal Goal-Conditioned Policycited by 39
  36. 2023What Makes Pre-Trained Visual Representations Successful for Robust Manipulation?cited by 38
  37. 2024Hand-Object Interaction Pretraining from Videoscited by 38
  38. 2025Emergence of Human to Robot Transfer in Vision-Language-Action Modelscited by 36

50 most cited of 215 All 215 (JSON)