mlsanityData to 14 Oct 2026

Real-World Robot Learning with Masked Visual Pre-training

arXiv 2210.03109 ↗First posted 6 Oct 2022cs.RO cs.CV cs.LGcited by 245

Not on the map. Why?

Cited by 31 in Offline RL and imitation learning from data, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In this work, we explore self-supervised visual pre-training on images from diverse, in-the-wild videos for real-world robotic tasks. Like prior work, our visual representations are pre-trained via a masked autoencoder (MAE), frozen, and then passed into a learnable control module. Unlike prior work, we show that the pre-trained representations are effective across a range of real-world robotic tasks and embodiments. We find that our encoder consistently outperforms CLIP (up to 75%), supervised ImageNet pre-training (up to 81%), and training from scratch (up to 81%). Finally, we train a 307M parameter vision transformer on a massive collection of 4.5M images from the Internet and egocentric videos, and demonstrate clearly the benefits of scaling visual pre-training for robot learning.

Key facts. Cited by 245 papers in the map (data through 14 Oct 2026). Builds on 57 of its 66 references that are in the map. First posted on arXiv 6 Oct 2022.

Builds on

57 of 66 references.

References · 57

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  5. 2020Language Models are Few-Shot Learnerscited by 32,419
  6. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  7. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  8. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  9. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  10. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  11. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  12. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
38 more
  1. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,162
  2. 2016WaveNet: A Generative Model for Raw Audiocited by 1,722
  3. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,591
  4. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  5. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,322
  6. 2016Colorful Image Colorizationcited by 1,114
  7. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,103
  8. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,066
  9. 2021Scaling Vision Transformerscited by 999
  10. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 998
  11. 2015End-to-End Training of Deep Visuomotor Policiescited by 875
  12. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 795
  13. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 743
  14. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 694
  15. 2018Learning Dexterous In-Hand Manipulationcited by 687
  16. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 622
  17. 2017Self-Normalizing Neural Networkscited by 598
  18. 2020Jukebox: A Generative Model for Musiccited by 542
  19. 2020Rescaling Egocentric Visioncited by 540
  20. 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 459
  21. 2022Masked Autoencoders As Spatiotemporal Learnerscited by 429
  22. 2021Early Convolutions Help Transformers See Bettercited by 396
  23. 2020Transporter Networks: Rearranging the Visual World for Robotic Manipulationcited by 344
  24. 2016Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collectioncited by 316
  25. 2022MultiMAE: Multi-modal Multi-task Masked Autoencoderscited by 291
  26. 2020Understanding Human Hands in Contact at Internet Scalecited by 281
  27. 2022Masked Visual Pre-training for Motor Controlcited by 248
  28. 2020What Should Not Be Contrastive in Contrastive Learningcited by 226
  29. 2015Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hourscited by 209
  30. 2016Learning to Poke by Poking: Experiential Learning of Intuitive Physicscited by 154
  31. 2021The Surprising Effectiveness of Representation Learning for Visual Imitationcited by 154
  32. 2015Deep Spatial Autoencoders for Visuomotor Learningcited by 153
  33. 2021Benchmarking Detection Transfer Learning with Vision Transformerscited by 131
  34. 2018Learning Synergies between Pushing and Grasping with Self-supervised Deep Reinforcement Learningcited by 128
  35. 2021Coarse-to-Fine Q-attention: Efficient Learning for Visual Robotic Manipulation via Discretisationcited by 122
  36. 2019Self-Supervised Correspondence in Visuomotor Policy Learningcited by 107
  37. 2021Q-attention: Enabling Efficient Learning for Vision-based Robotic Manipulationcited by 85
  38. 2022Dexterous Imitation Made Easy: A Learning-Based Framework for Efficient Dexterous Manipulationcited by 78

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 8; 2023: 71; 2024: 76; 2025: 55; 2026: 35.

245 papers.

Sort:

By area:

  1. 2024RDT-1B: a Diffusion Foundation Model for Bimanual Manipulationcited by 691
  2. 2024Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperationcited by 391
  3. 2023Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulationcited by 364
  4. 2023Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Taskscited by 327
  5. 2024GR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge for Robot Manipulationcited by 296
  6. 2024Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representationscited by 289
  7. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 279
  8. 2023Affordances from Human Videos as a Versatile Representation for Roboticscited by 226
  9. 2023ViNT: A Foundation Model for Visual Navigationcited by 222
  10. 2023Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?cited by 218
  11. 2025EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Videocited by 211
  12. 2023Large Language Models as General Pattern Machinescited by 197
38 more
  1. 2025DexVLA: Vision-Language Model with Plug-In Diffusion Expert for General Robot Controlcited by 195
  2. 2024HumanPlus: Humanoid Shadowing and Imitation from Humanscited by 185
  3. 2025HybridVLA: Collaborative Diffusion and Autoregression in a Unified Vision-Language-Action Modelcited by 181
  4. 2024EgoMimic: Scaling Imitation Learning via Egocentric Videocited by 175
  5. 2022GNM: A General Navigation Model to Drive Any Robotcited by 171
  6. 2022CroCo v2: Improved Cross-view Completion Pre-training for Stereo Matching and Optical Flowcited by 166
  7. 2023Language-Driven Representation Learning for Roboticscited by 165
  8. 2023Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planningcited by 161
  9. 2024Predictive Inverse Dynamics Models are Scalable Learners for Robotic Manipulationcited by 161
  10. 2023Open-World Object Manipulation using Pre-trained Vision-Language Modelscited by 156
  11. 2023AnyTeleop: A General Vision-Based Dexterous Robot Arm-Hand Teleoperation Systemcited by 152
  12. 2024GenAD: Generalized Predictive Model for Autonomous Drivingcited by 150
  13. 2024Scaling Proprioceptive-Visual Learning with Heterogeneous Pre-trained Transformerscited by 148
  14. 2023Structured World Models from Human Videoscited by 147
  15. 2024Multimodal Diffusion Transformer: Learning Versatile Behavior from Multimodal Goalscited by 134
  16. 2024THE COLOSSEUM: A Benchmark for Evaluating Generalization for Robotic Manipulationcited by 133
  17. 2023Instruct2Act: Mapping Multi-modality Instructions to Robotic Actions with Large Language Modelcited by 116
  18. 2023RoboCat: A Self-Improving Generalist Agent for Robotic Manipulationcited by 111
  19. 2024ManiGaussian: Dynamic Gaussian Splatting for Multi-task Robotic Manipulationcited by 108
  20. 2023UniDexGrasp++: Improving Dexterous Grasping Policy Learning via Geometry-aware Curriculum and Iterative Generalist-Specialist Learningcited by 107
  21. 2023Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysiscited by 103
  22. 2025Being-H0: Vision-Language-Action Pretraining from Large-Scale Human Videoscited by 102
  23. 2023On Bringing Robots Homecited by 101
  24. 2023GNFactor: Multi-Task Real Robot Learning with Generalizable Neural Feature Fieldscited by 99
  25. 2023General In-Hand Object Rotation with Vision and Touchcited by 99
  26. 2022Decorate the Newcomers: Visual Domain Prompt for Continual Test Time Adaptationcited by 98
  27. 2024What Matters in Building Vision-Language-Action Models for Generalist Robotscited by 89
  28. 2024Dreamitate: Real-World Visuomotor Policy Learning via Video Generationcited by 87
  29. 2023An Outlook into the Future of Egocentric Visioncited by 84
  30. 2023Multi-View Masked World Models for Visual Robotic Manipulationcited by 80
  31. 2022Pre-Training for Robots: Offline RL Enables Learning New Tasks from a Handful of Trialscited by 78
  32. 2023Teach a Robot to FISH: Versatile Imitation from One Minute of Demonstrationscited by 75
  33. 2024IGOR: Image-GOal Representations are the Atomic Control Units for Foundation Models in Embodied AIcited by 72
  34. 2023OVRL-V2: A simple state-of-art baseline for ImageNav and ObjectNavcited by 70
  35. 2024In-Context Imitation Learning via Next-Token Predictioncited by 70
  36. 2024When Do We Not Need Larger Vision Models?cited by 67
  37. 2023Dexterity from Touch: Self-Supervised Pre-Training of Tactile Representations with Robotic Playcited by 62
  38. 2024Real-World Robot Applications of Foundation Models: A Reviewcited by 61

50 most cited of 245 All 245 (JSON)