mlsanityData to 7 Oct 2026

RT-1: Robotics Transformer for Real-World Control at Scale

arXiv 2212.06817 ↗First posted 13 Dec 2022cs.RO cs.AI cs.CLcited by 2,007

Not on the map. Why?

Cited by 705 in Vision-language-action models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

By transferring knowledge from large, diverse, task-agnostic datasets, modern machine learning models can solve specific downstream tasks either zero-shot or with small task-specific datasets to a high level of performance. While this capability has been demonstrated in other fields such as computer vision, natural language processing or speech recognition, it remains to be shown in robotics, where the generalization capabilities of the models are particularly critical due to the difficulty of collecting real-world robotic data. We argue that one of the keys to the success of such general robotic models lies with open-ended task-agnostic training, combined with high-capacity architectures that can absorb all of the diverse, robotic data. In this paper, we present a model class, dubbed Robotics Transformer, that exhibits promising scalable model properties. We verify our conclusions in a study of different model classes and their ability to generalize as a function of the data size, model size, and data diversity based on a large-scale data collection on real robots performing real-world tasks. The project's website and videos can be found at robotics-transformer1.github.io

Key facts. Cited by 2,007 papers in the map (data through 7 Oct 2026). Builds on 52 of its 72 references that are in the map. First posted on arXiv 13 Dec 2022.

Builds on

52 of 72 references.

References · 52

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  4. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  5. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  6. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  7. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  8. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  9. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  10. 2022A Generalist Agentcited by 802
  11. 2018Image Transformercited by 770
  12. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
38 more
  1. 2019RLBench: The Robot Learning Benchmark & Learning Environmentcited by 669
  2. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  3. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
  4. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 517
  5. 2018Universal Sentence Encodercited by 485
  6. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  7. 2022Behavior Transformers: Cloning k modes with one stonecited by 299
  8. 2017Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperationcited by 286
  9. 2019RoboNet: Large-Scale Multi-Robot Learningcited by 268
  10. 2021Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasetscited by 248
  11. 2021MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scalecited by 226
  12. 2022Multi-Game Decision Transformerscited by 212
  13. 2015Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hourscited by 208
  14. 2020Language Conditioned Imitation Learning over Unstructured Datacited by 208
  15. 2022Open-vocabulary Queryable Scene Representations for Real World Planningcited by 159
  16. 2021Episodic Transformer for Vision-and-Language Navigationcited by 155
  17. 2020Language-Conditioned Imitation Learning for Robot Manipulation Taskscited by 136
  18. 2016Learning Modular Neural Network Policies for Multi-Task and Multi-Robot Transfercited by 124
  19. 2021Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotationcited by 117
  20. 2013Deep Learning for Detecting Robotic Graspscited by 114
  21. 2019Scene Memory Transformer for Embodied Agents in Long-Horizon Taskscited by 106
  22. 2021ManipulaTHOR: A Framework for Visual Object Manipulationcited by 101
  23. 2022MetaMorph: Learning Universal Controllers with Transformerscited by 78
  24. 2018Multiple Interactions Made Easy (MIME): Large Scale Demonstrations Data for Imitationcited by 76
  25. 2018Robot Learning in Homes: Improving Generalization and Reducing Dataset Biascited by 75
  26. 2020RetinaGAN: An Object-aware Approach to Sim-to-Real Transfercited by 68
  27. 2021Hierarchical Task Learning from Language Instructions with Unified Transformers and Self-Monitoringcited by 53
  28. 2019HRL4IN: Hierarchical Reinforcement Learning for Interactive Navigation with Mobile Manipulatorscited by 52
  29. 2015Listen, Attend, and Walk: Neural Mapping of Navigational Instructions to Action Sequencescited by 50
  30. 2020ReLMoGen: Leveraging Motion Generation in Reinforcement Learning for Mobile Manipulationcited by 47
  31. 2019Off-Policy Evaluation via Off-Policy Classificationcited by 32
  32. 2017Learning a visuomotor controller for real world robotic grasping using simulated depth imagescited by 29
  33. 2020Thinking While Moving: Deep Reinforcement Learning with Concurrent Controlcited by 29
  34. 2020Sub-Goal Trees -- a Framework for Goal-Based Reinforcement Learningcited by 27
  35. 2021Learning Kinematic Feasibility for Mobile Manipulation through Deep Reinforcement Learningcited by 24
  36. 2019Decoupling feature extraction from policy learning: assessing benefits of state representation learning in goal based roboticscited by 21
  37. 2020Scalable Multi-Task Imitation Learning with Autonomous Improvementcited by 21
  38. 2019Motion Reasoning for Goal-Based Imitation Learningcited by 6

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 219; 2024: 364; 2025: 623; 2026: 800.

2,007 papers.

Sort:

By area:

  1. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  2. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  3. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  4. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,653
  5. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,490
  6. 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
  7. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  8. 2025GR00T N1: An Open Foundation Model for Generalist Humanoid Robotscited by 1,167
  9. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134
  10. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  11. 2024Genie: Generative Interactive Environmentscited by 706
  12. 2024RDT-1B: a Diffusion Foundation Model for Bimanual Manipulationcited by 670
38 more
  1. 2025FAST: Efficient Action Tokenization for Vision-Language-Action Modelscited by 643
  2. 2023BridgeData V2: A Dataset for Robot Learning at Scalecited by 634
  3. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spacescited by 580
  4. 2023Learning Universal Policies via Text-Guided Video Generationcited by 506
  5. 2025SpatialVLA: Exploring Spatial Representations for Visual-Language-Action Modelcited by 502
  6. 2024Evaluating Real-World Robot Manipulation Policies in Simulationcited by 484
  7. 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Modelscited by 482
  8. 2025RoboTwin 2.0: A Scalable Data Generator and Benchmark with Strong Domain Randomization for Robust Bimanual Robotic Manipulationcited by 477
  9. 2023Learning Interactive Real-World Simulatorscited by 440
  10. 2025CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Modelscited by 440
  11. 2023Cognitive Architectures for Language Agentscited by 422
  12. 2024CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulationcited by 411
  13. 2026World Action Models are Zero-shot Policiescited by 411
  14. 2023AppAgent: Multimodal Agents as Smartphone Userscited by 407
  15. 2025Agent Laboratory: Using LLM Agents as Research Assistantscited by 405
  16. 2023Augmented Language Models: a Surveycited by 398
  17. 2024RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robotscited by 391
  18. 2025AgiBot World Colosseo: A Large-scale Manipulation Platform for Scalable and Intelligent Embodied Systemscited by 390
  19. 2025UniVLA: Learning to Act Anywhere with Task-centric Latent Actionscited by 386
  20. 2024DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planningcited by 357
  21. 2023An Embodied Generalist Agent in 3D Worldcited by 355
  22. 2023Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulationcited by 353
  23. 2025WorldVLA: Towards Autoregressive Action World Modelcited by 351
  24. 2023Vision-Language Foundation Models as Effective Robot Imitatorscited by 344
  25. 2024Robotic Control via Embodied Chain-of-Thought Reasoningcited by 340
  26. 2024TinyVLA: Towards Fast, Data-Efficient Vision-Language-Action Models for Robotic Manipulationcited by 331
  27. 20243D-VLA: A 3D Vision-Language-Action Generative World Modelcited by 320
  28. 2023MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrationscited by 308
  29. 2023Language to Rewards for Robotic Skill Synthesiscited by 307
  30. 2023Photorealistic Video Generation with Diffusion Modelscited by 304
  31. 2024LongVILA: Scaling Long-Context Visual Language Models for Long Videoscited by 303
  32. 2023ConceptFusion: Open-set Multimodal 3D Mappingcited by 302
  33. 2023Robots That Ask For Help: Uncertainty Alignment for Large Language Model Plannerscited by 298
  34. 2024TraceVLA: Visual Trace Prompting Enhances Spatial-Temporal Awareness for Generalist Robotic Policiescited by 298
  35. 2023ChatGPT for Robotics: Design Principles and Model Abilitiescited by 296
  36. 2024GR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge for Robot Manipulationcited by 290
  37. 2026Causal World Modeling for Robot Controlcited by 289
  38. 2025X-VLA: Soft-Prompted Transformer as Scalable Cross-Embodiment Vision-Language-Action Modelcited by 285

50 most cited of 2,007 All 2,007 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025GigaBrain-0: A World Model-Powered Vision-Language-Action Model
  2. 2024BAKU: An Efficient Transformer for Multi-Task Policy Learning
  3. 2025Robot Control Stack: A Lean Ecosystem for Robot Learning at Scale
  4. 2024π0: A Vision-Language-Action Flow Model for General Robot Control
  5. 2024The Ingredients for Robotic Diffusion Transformers
  6. 2022Pre-Training for Robots: Offline RL Enables Learning New Tasks from a Handful of Trials
  7. 2024Generalized Robot Learning Framework
  8. 2026A Pragmatic VLA Foundation Model