mlsanityData to 7 Oct 2026

Masked Visual Pre-training for Motor Control

arXiv 2203.06173 ↗First posted 11 Mar 2022cs.CV cs.LG cs.ROcited by 246

Not on the map. Why?

Cited by 34 in Offline RL and imitation learning from data, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

This paper shows that self-supervised visual pre-training from real-world images is effective for learning motor control tasks from pixels. We first train the visual representations by masked modeling of natural images. We then freeze the visual encoder and train neural network controllers on top with reinforcement learning. We do not perform any task-specific fine-tuning of the encoder; the same visual representations are used for all motor control tasks. To the best of our knowledge, this is the first self-supervised model to exploit real-world images at scale for motor control. To accelerate progress in learning from pixels, we contribute a benchmark suite of hand-designed tasks varying in movements, scenes, and robots. Without relying on labels, state-estimation, or expert demonstrations, we consistently outperform supervised encoders by up to 80% absolute success rate, sometimes even matching the oracle state performance. We also find that in-the-wild images, e.g., from YouTube or Egocentric videos, lead to better visual representations for various manipulation tasks than ImageNet images.

Key facts. Cited by 246 papers in the map (data through 7 Oct 2026). Builds on 68 of its 81 references that are in the map. First posted on arXiv 11 Mar 2022.

Builds on

68 of 81 references.

References · 68

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2017Proximal Policy Optimization Algorithmscited by 14,441
  7. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  8. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  9. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  10. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  11. 2017Mask R-CNNcited by 7,259
  12. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
38 more
  1. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,160
  2. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  3. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  4. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  5. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,586
  6. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  7. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  8. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,424
  9. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,379
  10. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  11. 2016Colorful Image Colorizationcited by 1,113
  12. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  13. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,062
  14. 2018DeepMind Control Suitecited by 935
  15. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 882
  16. 2015End-to-End Training of Deep Visuomotor Policiescited by 847
  17. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 789
  18. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  19. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  20. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  21. 2019Solving Rubik's Cube with a Robot Handcited by 699
  22. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  23. 2019Data-Efficient Image Recognition with Contrastive Predictive Codingcited by 672
  24. 2019RLBench: The Robot Learning Benchmark & Learning Environmentcited by 669
  25. 2018Learning Dexterous In-Hand Manipulationcited by 643
  26. 2017Self-Normalizing Neural Networkscited by 591
  27. 2020Rescaling Egocentric Visioncited by 529
  28. 2020robosuite: A Modular Simulation Framework and Benchmark for Robot Learningcited by 484
  29. 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 453
  30. 2020Reinforcement Learning with Augmented Datacited by 452
  31. 2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 400
  32. 2021Early Convolutions Help Transformers See Bettercited by 395
  33. 2021Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learningcited by 395
  34. 2019Learning Deep Transformer Models for Machine Translationcited by 309
  35. 2020Understanding Human Hands in Contact at Internet Scalecited by 281
  36. 2019Improving Sample Efficiency in Model-Free Reinforcement Learning from Imagescited by 268
  37. 2020Space-Time Correspondence as a Contrastive Random Walkcited by 239
  38. 2020What Should Not Be Contrastive in Contrastive Learningcited by 224

50 most cited of 68 All 68 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 38; 2023: 69; 2024: 64; 2025: 44; 2026: 31.

246 papers.

Sort:

By area:

  1. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  2. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,653
  3. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  4. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  5. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 384
  6. 2024DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planningcited by 357
  7. 2023Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulationcited by 353
  8. 2022Scaling Language-Image Pre-training via Maskingcited by 317
  9. 2024GR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge for Robot Manipulationcited by 290
  10. 2022MultiMAE: Multi-modal Multi-task Masked Autoencoderscited by 287
  11. 2023A Cookbook of Self-Supervised Learningcited by 248
  12. 2022Real-World Robot Learning with Masked Visual Pre-trainingcited by 241
38 more
  1. 2023Affordances from Human Videos as a Versatile Representation for Roboticscited by 221
  2. 2023MimicPlay: Long-Horizon Imitation Learning by Watching Human Playcited by 214
  3. 2023LIV: Language-Image Representations and Rewards for Robotic Controlcited by 180
  4. 2022Masked World Models for Visual Controlcited by 169
  5. 2024HumanPlus: Humanoid Shadowing and Imitation from Humanscited by 164
  6. 2023Language-Driven Representation Learning for Roboticscited by 163
  7. 2023Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planningcited by 161
  8. 2023Foundation Models for Decision Making: Problems, Methods, and Opportunitiescited by 160
  9. 2024Predictive Inverse Dynamics Models are Scalable Learners for Robotic Manipulationcited by 159
  10. 2024DexCap: Scalable and Portable Mocap Data Collection System for Dexterous Manipulationcited by 149
  11. 2023Structured World Models from Human Videoscited by 144
  12. 2024Gen2Act: Human Video Generation in Novel Scenarios enables Generalizable Robot Manipulationcited by 142
  13. 2024Multimodal Diffusion Transformer: Learning Versatile Behavior from Multimodal Goalscited by 131
  14. 2026π0.7: a Steerable Generalist Robotic Foundation Model with Emergent Capabilitiescited by 127
  15. 2024Flow as the Cross-Domain Manipulation Interfacecited by 126
  16. 2024Track2Act: Predicting Point Tracks from Internet Videos enables Generalizable Robot Manipulationcited by 116
  17. 2022Towards Human-Level Bimanual Dexterous Manipulation with Reinforcement Learningcited by 113
  18. 2022MaskViT: Masked Visual Pre-Training for Video Predictioncited by 112
  19. 2025GR-3 Technical Reportcited by 110
  20. 2023RoboCat: A Self-Improving Generalist Agent for Robotic Manipulationcited by 108
  21. 2022Reinforcement Learning with Action-Free Pre-Training from Videoscited by 107
  22. 2026StarVLA: A Lego-like Codebase for Vision-Language-Action Model Developingcited by 103
  23. 2022VideoDex: Learning Dexterity from Internet Videoscited by 101
  24. 2023On Bringing Robots Homecited by 101
  25. 2024Foundation Policies with Hilbert Representationscited by 89
  26. 2023General In-Hand Object Rotation with Vision and Touchcited by 87
  27. 2024General Flow as Foundation Affordance for Scalable Robot Learningcited by 80
  28. 2022Pre-Training for Robots: Offline RL Enables Learning New Tasks from a Handful of Trialscited by 78
  29. 2023Multi-View Masked World Models for Visual Robotic Manipulationcited by 78
  30. 2022Learning and Retrieval from Prior Data for Skill-based Imitation Learningcited by 77
  31. 2025BridgeVLA: Input-Output Alignment for Efficient 3D Manipulation Learning with Vision-Language Modelscited by 77
  32. 2023Teach a Robot to FISH: Versatile Imitation from One Minute of Demonstrationscited by 74
  33. 2023XSkill: Cross Embodiment Skill Discoverycited by 74
  34. 2022Offline Q-Learning on Diverse Multi-Task Data Both Scales And Generalizescited by 72
  35. 2023OVRL-V2: A simple state-of-art baseline for ImageNav and ObjectNavcited by 68
  36. 2024IGOR: Image-GOal Representations are the Atomic Control Units for Foundation Models in Embodied AIcited by 68
  37. 2024In-Context Imitation Learning via Next-Token Predictioncited by 65
  38. 2023Reinforcement Learning from Passive Data via Latent Intentionscited by 64

50 most cited of 246 All 246 (JSON)