mlsanityData to 14 Oct 2026

RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control

arXiv 2307.15818 ↗First posted 28 Jul 2023cs.RO cs.CL cs.CVcited by 3,080

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

also in 1 more
Abstract (arXiv)

We study how vision-language models trained on Internet-scale data can be incorporated directly into end-to-end robotic control to boost generalization and enable emergent semantic reasoning. Our goal is to enable a single end-to-end trained model to both learn to map robot observations to actions and enjoy the benefits of large-scale pretraining on language and vision-language data from the web. To this end, we propose to co-fine-tune state-of-the-art vision-language models on both robotic trajectory data and Internet-scale vision-language tasks, such as visual question answering. In contrast to other approaches, we propose a simple, general recipe to achieve this goal: in order to fit both natural language responses and robotic actions into the same format, we express the actions as text tokens and incorporate them directly into the training set of the model in the same way as natural language tokens. We refer to such category of models as vision-language-action models (VLA) and instantiate an example of such a model, which we call RT-2. Our extensive evaluation (6k evaluation trials) shows that our approach leads to performant robotic policies and enables RT-2 to obtain a range of emergent capabilities from Internet-scale training. This includes significantly improved generalization to novel objects, the ability to interpret commands not present in the robot training data (such as placing an object onto a particular number or icon), and the ability to perform rudimentary reasoning in response to user commands (such as picking up the smallest or largest object, or the one closest to another object). We further show that incorporating chain of thought reasoning allows RT-2 to perform multi-stage semantic reasoning, for example figuring out which object to pick up for use as an improvised hammer (a rock), or which type of drink is best suited for someone who is tired (an energy drink).

Key facts. Cited by 3,080 papers in the map (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 76 of its 94 references that are in the map. First posted on arXiv 28 Jul 2023.

Builds on

76 of 94 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thought24 May
  2. 2023Grounding Classical Task Planners via Vision-Language Models17 Apr
  3. 2023PaLM-E: An Embodied Multimodal Language Model6 Mar
  4. 2023Open-World Object Manipulation using Pre-trained Vision-Language Models2 Mar
  5. 2023Distilling Internet-Scale Vision-Language Models into Embodied Agents29 Jan
  6. 2022Robotic Skill Acquisition via Instruction Augmentation with Vision-Language Models21 Nov

Other references · 70

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  6. 2023Segment Anythingcited by 8,234
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  9. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  10. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  11. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
  12. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
38 more
  1. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
  2. 2023PaLM 2 Technical Reportcited by 1,162
  3. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,035
  4. 2021Scaling Vision Transformerscited by 999
  5. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 906
  6. 2022A Generalist Agentcited by 808
  7. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 766
  8. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 760
  9. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 694
  10. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 694
  11. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 622
  12. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 611
  13. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 585
  14. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 551
  15. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  16. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 537
  17. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 523
  18. 2018Universal Sentence Encodercited by 489
  19. 2020Reinforcement Learning with Augmented Datacited by 468
  20. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 439
  21. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 407
  22. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 395
  23. 2016Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collectioncited by 316
  24. 2022Simple Open-Vocabulary Object Detection with Vision Transformerscited by 305
  25. 2023ChatGPT for Robotics: Design Principles and Model Abilitiescited by 299
  26. 2022UL2: Unifying Language Learning Paradigmscited by 295
  27. 2016Deep Visual Foresight for Planning Robot Motioncited by 285
  28. 2019RoboNet: Large-Scale Multi-Robot Learningcited by 270
  29. 2017Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metricscited by 264
  30. 2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 255
  31. 2022Masked Visual Pre-training for Motor Controlcited by 248
  32. 2017One-Shot Visual Imitation Learning via Meta-Learningcited by 228
  33. 2023TidyBot: Personalized Robot Assistance with Large Language Modelscited by 221
  34. 2022Interactive Language: Talking to Robots in Real Timecited by 218
  35. 2023Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?cited by 218
  36. 2015Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hourscited by 209
  37. 2020Language Conditioned Imitation Learning over Unstructured Datacited by 209
  38. 2023LIV: Language-Image Representations and Rewards for Robotic Controlcited by 188

50 most cited of 70 All 70 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 123; 2024: 506; 2025: 900; 2026: 1551.

3,080 papers.

Sort:

By area:

  1. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,887
  2. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,407
  3. 2024Octo: An Open-Source Generalist Robot Policycited by 1,484
  4. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,439
  5. 2025GR00T N1: An Open Foundation Model for Generalist Humanoid Robotscited by 1,225
  6. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,195
  7. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,154
  8. 2025Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Successcited by 883
  9. 2023ExpeL: LLM Agents Are Experiential Learnerscited by 876
  10. 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 809
  11. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilitiescited by 706
  12. 2024RDT-1B: a Diffusion Foundation Model for Bimanual Manipulationcited by 691
38 more
  1. 2025FAST: Efficient Action Tokenization for Vision-Language-Action Modelscited by 665
  2. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 640
  3. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spacescited by 611
  4. 2023Eureka: Human-Level Reward Design via Coding Large Language Modelscited by 571
  5. 2023Identifying the Risks of LM Agents with an LM-Emulated Sandboxcited by 548
  6. 2023DriveLM: Driving with Graph Visual Question Answeringcited by 536
  7. 2025SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Roboticscited by 530
  8. 2025SpatialVLA: Exploring Spatial Representations for Visual-Language-Action Modelcited by 519
  9. 2025RoboTwin 2.0: A Scalable Data Generator and Benchmark with Strong Domain Randomization for Robust Bimanual Robotic Manipulationcited by 518
  10. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 512
  11. 2025Group-in-Group Policy Optimization for LLM Agent Trainingcited by 497
  12. 2024Evaluating Real-World Robot Manipulation Policies in Simulationcited by 495
  13. 2023VideoPoet: A Large Language Model for Zero-Shot Video Generationcited by 492
  14. 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Modelscited by 490
  15. 2026World Action Models are Zero-shot Policiescited by 465
  16. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Modelcited by 461
  17. 2023Learning Interactive Real-World Simulatorscited by 458
  18. 2023Cognitive Architectures for Language Agentscited by 429
  19. 2025Agent Laboratory: Using LLM Agents as Research Assistantscited by 425
  20. 2024CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulationcited by 422
  21. 2023AppAgent: Multimodal Agents as Smartphone Userscited by 415
  22. 2025UniVLA: Learning to Act Anywhere with Task-centric Latent Actionscited by 410
  23. 2025AgiBot World Colosseo: A Large-scale Manipulation Platform for Scalable and Intelligent Embodied Systemscited by 405
  24. 2024Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperationcited by 391
  25. 2024DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planningcited by 386
  26. 2024The Platonic Representation Hypothesiscited by 382
  27. 2025WorldVLA: Towards Autoregressive Action World Modelcited by 367
  28. 2023An Embodied Generalist Agent in 3D Worldcited by 365
  29. 2023Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulationcited by 364
  30. 2023Vision-Language Foundation Models as Effective Robot Imitatorscited by 349
  31. 2026Fast-WAM: Do World Action Models Need Test-time Future Imagination?cited by 349
  32. 2023GPT-Driver: Learning to Drive with GPTcited by 347
  33. 2025Absolute Zero: Reinforced Self-play Reasoning with Zero Datacited by 344
  34. 2024Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Modelscited by 339
  35. 2024TinyVLA: Towards Fast, Data-Efficient Vision-Language-Action Models for Robotic Manipulationcited by 339
  36. 2024VideoAgent: Long-form Video Understanding with Large Language Model as Agentcited by 332
  37. 20243D-VLA: A 3D Vision-Language-Action Generative World Modelcited by 328
  38. 2024Latent Action Pretraining from Videoscited by 321

50 most cited of 3,080 All 3,080 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025π0.5: a Vision-Language-Action Model with Open-World Generalization
  2. 2026Two Bridges, One Pathway: From VLMs to Generalizable VLAs with Embodied Trajectory-Coupled Data
  3. 2026Vision-Language-Action in Robotics: A Survey of Datasets, Benchmarks, and Data Engines
  4. 2026ElegantVLA: Learning When to Think for Efficient Vision-Language-Action Models
  5. 2026RoboSemanticBench: Diagnosing Semantic Grounding in Action Prediction for VLA Models
  6. 2025Do What You Say: Steering Vision-Language-Action Models via Runtime Reasoning-Action Alignment Verification
  7. 2025GigaBrain-0: A World Model-Powered Vision-Language-Action Model
  8. 2026Task Robustness via Re-Labelling Vision-Action Robot Data