mlsanityData to 7 Oct 2026

Robot Learning with Sensorimotor Pre-training

arXiv 2306.10007 ↗First posted 16 Jun 2023cs.RO cs.CV cs.LGcited by 51

Not on the map. Why?

Abstract (arXiv)

We present a self-supervised sensorimotor pre-training approach for robotics. Our model, called RPT, is a Transformer that operates on sequences of sensorimotor tokens. Given a sequence of camera images, proprioceptive robot states, and actions, we encode the sequence into tokens, mask out a subset, and train a model to predict the missing content from the rest. We hypothesize that if a robot can predict the masked-out content it will have acquired a good model of the physical world that can enable it to act. RPT is designed to operate on latent visual representations which makes prediction tractable, enables scaling to larger models, and allows fast inference on a real robot. To evaluate our approach, we collected a dataset of 20,000 real-world trajectories over 9 months using a combination of motion planning and grasping algorithms. We find that sensorimotor pre-training consistently outperforms training from scratch, has favorable scaling properties, and enables transfer across different tasks, environments, and robots.

Key facts. Cited by 51 papers in the map (data through 7 Oct 2026). Builds on 29 of its 35 references that are in the map. First posted on arXiv 16 Jun 2023.

Builds on

29 of 35 references.

References · 29

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  6. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  7. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,160
  8. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  9. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  10. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  11. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  12. 2022A Generalist Agentcited by 802
17 more
  1. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  2. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  3. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
  4. 2020Rescaling Egocentric Visioncited by 529
  5. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 517
  6. 2020Understanding Human Hands in Contact at Internet Scalecited by 281
  7. 2021Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasetscited by 248
  8. 2022Real-World Robot Learning with Masked Visual Pre-trainingcited by 241
  9. 2021MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scalecited by 226
  10. 2015Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hourscited by 208
  11. 2019Self-Supervised Correspondence in Visuomotor Policy Learningcited by 106
  12. 2022From Play to Policy: Conditional Behavior Generation from Uncurated Robot Datacited by 103
  13. 2022Pre-Training for Robots: Offline RL Enables Learning New Tasks from a Handful of Trialscited by 78
  14. 2023Multi-View Masked World Models for Visual Robotic Manipulationcited by 78
  15. 2020Learning Visual Robotic Control Efficiently with Contrastive Pre-training and Data Augmentationcited by 20
  16. 2020Exploratory Grasping: Asymptotically Optimal Algorithms for Grasping Challenging Polyhedral Objectscited by 9
  17. 2021LEGS: Learning Efficient Grasp Sets for Exploratory Graspingcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 22; 2025: 17; 2026: 10.

51 papers.

Sort:

By area:

  1. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  2. 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
  3. 2024Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperationcited by 377
  4. 2023Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulationcited by 353
  5. 2024GR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge for Robot Manipulationcited by 290
  6. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  7. 2025Unified Video Action Modelcited by 195
  8. 2024Scaling Proprioceptive-Visual Learning with Heterogeneous Pre-trained Transformerscited by 145
  9. 2025HAMSTER: Hierarchical Action Models For Open-World Robot Manipulationcited by 125
  10. 2024LLARVA: Vision-Action Instruction Tuning Enhances Robot Learningcited by 87
  11. 2024A Touch, Vision, and Language Dataset for Multimodal Alignmentcited by 68
  12. 2024When Do We Not Need Larger Vision Models?cited by 65
38 more
  1. 2024In-Context Imitation Learning via Next-Token Predictioncited by 65
  2. 2024Humanoid Locomotion as Next Token Predictioncited by 63
  3. 2025Embodied AI Agents: Modeling the Worldcited by 57
  4. 2024Sparse Diffusion Policy: A Sparse, Reusable, and Flexible Policy for Robot Learningcited by 56
  5. 20254D-VLA: Spatiotemporal Vision-Language-Action Pretraining with Cross-Scene Calibrationcited by 55
  6. 2026Robometer: Scaling General-Purpose Robotic Reward Models via Trajectory Comparisonscited by 51
  7. 2024DynaMo: In-Domain Dynamics Pretraining for Visuo-Motor Controlcited by 43
  8. 2024Hand-Object Interaction Pretraining from Videoscited by 38
  9. 2024Learning to Manipulate Anywhere: A Visual Generalizable Framework For Reinforcement Learningcited by 36
  10. 2025LLaDA-VLA: Vision Language Diffusion Action Modelscited by 30
  11. 2025Learning Long-Context Diffusion Policies via Past-Token Predictioncited by 28
  12. 2024Learning from Massive Human Videos for Universal Humanoid Pose Controlcited by 27
  13. 2024SUGAR: Pre-training 3D Visual Representations for Roboticscited by 26
  14. 2024CARP: Visuomotor Policy Learning via Coarse-to-Fine Autoregressive Predictioncited by 26
  15. 2025Latent Diffusion Planning for Imitation Learningcited by 26
  16. 2025World Models for Learning Dexterous Hand-Object Interactions from Human Videoscited by 26
  17. 2024In-Context Learning Enables Robot Action Prediction in LLMscited by 25
  18. 2024GenDP: 3D Semantic Fields for Category-Level Generalizable Diffusion Policycited by 25
  19. 2024Robots Pre-train Robots: Manipulation-Centric Robotic Representation from Large-Scale Robot Datasetscited by 23
  20. 2025An Empirical Study of Autoregressive Pre-training from Videoscited by 21
  21. 2023Mastering Robot Manipulation with Multimodal Prompts through Pretraining and Multi-task Fine-tuningcited by 15
  22. 2025XR-1: Towards Versatile Vision-Language-Action Models via Learning Unified Vision-Motion Representationscited by 15
  23. 2025MAPLE: Encoding Dexterous Robotic Manipulation Priors Learned From Egocentric Videoscited by 12
  24. 2026VLANeXt: Recipes for Building Strong VLA Modelscited by 12
  25. 2025StaMo: Unsupervised Learning of Generalizable Robot Motion from Compact State Representationcited by 10
  26. 20254D Visual Pre-training for Robot Learningcited by 8
  27. 2024Decision Transformer as a Foundation Model for Partially Observable Continuous Controlcited by 5
  28. 2025ROSA: Harnessing Robot States for Vision-Language and Action Alignmentcited by 5
  29. 2026GeneralVLA: Generalizable Vision-Language-Action Models with Knowledge-Guided Trajectory Planningcited by 5
  30. 2026From Abstraction to Instantiation: Learning Behavioral Representation for Vision-Language-Action Modelcited by 5
  31. 2025Open-Source Multimodal Moxin Models with Moxin-VLM and Moxin-VLAcited by 4
  32. 2026Generalizable VLA Finetuning via Representation Anchoring and Language-Action Alignmentcited by 4
  33. 2026Capturing Visual Environment Structure Correlates with Control Performancecited by 2
  34. 2026TEMPO: Learning Temporal Context for Dynamic Robot Manipulationcited by 1
  35. 2025On the Generalisation of Koopman Representations for Chaotic System Controlcited by 0
  36. 2025Learning to Grasp Anything by Playing with Random Toyscited by 0
  37. 2026Enhancing Linguistic Generalization of VLA: Fine-Tuning OpenVLA via Synthetic Instruction Augmentationcited by 0
  38. 2026Learning Structural Latent Points for Efficient Visual Representations in Robotic Manipulationcited by 0

50 most cited of 51 All 51 (JSON)