mlsanityData to 7 Oct 2026

R3M: A Universal Visual Representation for Robot Manipulation

arXiv 2203.12601 ↗First posted 23 Mar 2022cs.RO cs.AI cs.CVcited by 678

Not on the map. Why?

Cited by 98 in Offline RL and imitation learning from data, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We study how visual representations pre-trained on diverse human video data can enable data-efficient learning of downstream robotic manipulation tasks. Concretely, we pre-train a visual representation using the Ego4D human video dataset using a combination of time-contrastive learning, video-language alignment, and an L1 penalty to encourage sparse and compact representations. The resulting representation, R3M, can be used as a frozen perception module for downstream policy learning. Across a suite of 12 simulated robot manipulation tasks, we find that R3M improves task success by over 20% compared to training from scratch and by over 10% compared to state-of-the-art visual representations like CLIP and MoCo. Furthermore, R3M enables a Franka Emika Panda arm to learn a range of manipulation tasks in a real, cluttered apartment given just 20 demonstrations. Code and pre-trained models are available at https://tinyurl.com/robotr3m.

Key facts. Cited by 678 papers in the map (data through 7 Oct 2026). Builds on 57 of its 74 references that are in the map. First posted on arXiv 23 Mar 2022.

Builds on

57 of 74 references.

References · 57

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  7. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  8. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  9. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  10. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  11. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  12. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
38 more
  1. 2015End-to-End Training of Deep Visuomotor Policiescited by 847
  2. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 789
  3. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  4. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  5. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  6. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 596
  7. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  8. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 517
  9. 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 453
  10. 2020Reinforcement Learning with Augmented Datacited by 452
  11. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 424
  12. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learningcited by 380
  13. 2019Learning Correspondence from the Cycle-Consistency of Timecited by 269
  14. 2019RoboNet: Large-Scale Multi-Robot Learningcited by 268
  15. 2021Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasetscited by 248
  16. 2022Masked Visual Pre-training for Motor Controlcited by 246
  17. 2020Space-Time Correspondence as a Contrastive Random Walkcited by 239
  18. 2015Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hourscited by 208
  19. 2019DeepMDP: Learning Continuous Latent Space Models for Representation Learningcited by 204
  20. 2021Simple but Effective: CLIP Embeddings for Embodied AIcited by 191
  21. 2017Imitation from Observation: Learning to Imitate Behaviors from Raw Video via Context Translationcited by 167
  22. 2022The Unsurprising Effectiveness of Pre-Trained Vision Models for Controlcited by 155
  23. 2021The Surprising Effectiveness of Representation Learning for Visual Imitationcited by 149
  24. 2018One-Shot Imitation from Observing Humans via Domain-Adaptive Meta-Learningcited by 146
  25. 2020Language-Conditioned Imitation Learning for Robot Manipulation Taskscited by 136
  26. 2021Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotationcited by 117
  27. 2021Learning Generalizable Robotic Reward Functions from "In-The-Wild" Human Videoscited by 116
  28. 2020Visual Imitation Made Easycited by 111
  29. 2019AVID: Learning Multi-Stage Tasks via Pixel-Level Translation of Human Videoscited by 105
  30. 2021XIRL: Cross-embodiment Inverse Reinforcement Learningcited by 101
  31. 2021RRL: Resnet as representation for Reinforcement Learningcited by 89
  32. 2021Learning by Watching: Physical Imitation of Manipulation Skills from Human Videoscited by 87
  33. 2020Reinforcement Learning with Videos: Combining Offline Observations with Interactioncited by 80
  34. 2022Vision Models Are More Robust And Fair When Pretrained On Uncurated Images Without Supervisioncited by 80
  35. 2018Multiple Interactions Made Easy (MIME): Large Scale Demonstrations Data for Imitationcited by 76
  36. 2019Third-Person Visual Imitation Learning via Decoupled Hierarchical Controllercited by 73
  37. 2016Unsupervised Perceptual Rewards for Imitation Learningcited by 64
  38. 2020Model-Based Inverse Reinforcement Learning from Visual Demonstrationscited by 52

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 40; 2023: 131; 2024: 178; 2025: 172; 2026: 157.

678 papers.

Sort:

By area:

  1. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  2. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  3. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  4. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,653
  5. 2025GR00T N1: An Open Foundation Model for Generalist Humanoid Robotscited by 1,167
  6. 2025Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Successcited by 844
  7. 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 738
  8. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  9. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
  10. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 469
  11. 2024CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulationcited by 411
  12. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
38 more
  1. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 384
  2. 2024Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperationcited by 377
  3. 2022DayDreamer: World Models for Physical Robot Learningcited by 372
  4. 2024DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planningcited by 357
  5. 2023Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulationcited by 353
  6. 2023Vision-Language Foundation Models as Effective Robot Imitatorscited by 344
  7. 2024Latent Action Pretraining from Videoscited by 302
  8. 2024GR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge for Robot Manipulationcited by 290
  9. 2024Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representationscited by 279
  10. 2023EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thoughtcited by 276
  11. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  12. 2023A Cookbook of Self-Supervised Learningcited by 248
  13. 2022Real-World Robot Learning with Masked Visual Pre-trainingcited by 241
  14. 2023Affordances from Human Videos as a Versatile Representation for Roboticscited by 221
  15. 2023Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?cited by 215
  16. 2023MimicPlay: Long-Horizon Imitation Learning by Watching Human Playcited by 214
  17. 2023Any-point Trajectory Modeling for Policy Learningcited by 207
  18. 2024RT-H: Action Hierarchies Using Languagecited by 204
  19. 2023Large Language Models as General Pattern Machinescited by 196
  20. 2025EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Videocited by 191
  21. 2023LIV: Language-Image Representations and Rewards for Robotic Controlcited by 180
  22. 2024Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AIcited by 175
  23. 2022Masked World Models for Visual Controlcited by 169
  24. 2025DreamGen: Unlocking Generalization in Robot Learning through Video World Modelscited by 168
  25. 2023RoboAgent: Generalization and Efficiency in Robot Manipulation via Semantic Augmentations and Action Chunkingcited by 167
  26. 2024RoboMIND: Benchmark on Multi-embodiment Intelligence Normative Data for Robot Manipulationcited by 166
  27. 2024HumanPlus: Humanoid Shadowing and Imitation from Humanscited by 164
  28. 2023Language-Driven Representation Learning for Roboticscited by 163
  29. 2022GNM: A General Navigation Model to Drive Any Robotcited by 161
  30. 2023Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planningcited by 161
  31. 2024EgoMimic: Scaling Imitation Learning via Egocentric Videocited by 160
  32. 2023Learning to Act from Actionless Videos through Dense Correspondencescited by 159
  33. 2024Predictive Inverse Dynamics Models are Scalable Learners for Robotic Manipulationcited by 159
  34. 2023Open-World Object Manipulation using Pre-trained Vision-Language Modelscited by 155
  35. 2023Text2Reward: Reward Shaping with Language Models for Reinforcement Learningcited by 153
  36. 2024DexCap: Scalable and Portable Mocap Data Collection System for Dexterous Manipulationcited by 149
  37. 2024Scaling Proprioceptive-Visual Learning with Heterogeneous Pre-trained Transformerscited by 145
  38. 2023Structured World Models from Human Videoscited by 144

50 most cited of 678 All 678 (JSON)