mlsanityData to 7 Oct 2026

Vision-Language Foundation Models as Effective Robot Imitators

arXiv 2311.01378 ↗First posted 2 Nov 2023cs.RO cs.AI cs.LGcited by 344

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Recent progress in vision language foundation models has shown their ability to understand multimodal data and resolve complicated vision language tasks, including robotics manipulation. We seek a straightforward way of making use of existing vision-language models (VLMs) with simple fine-tuning on robotics data. To this end, we derive a simple and novel vision-language manipulation framework, dubbed RoboFlamingo, built upon the open-source VLMs, OpenFlamingo. Unlike prior works, RoboFlamingo utilizes pre-trained VLMs for single-step vision-language comprehension, models sequential history information with an explicit policy head, and is slightly fine-tuned by imitation learning only on language-conditioned manipulation datasets. Such a decomposition provides RoboFlamingo the flexibility for open-loop control and deployment on low-performance platforms. By exceeding the state-of-the-art performance with a large margin on the tested benchmark, we show RoboFlamingo can be an effective and competitive alternative to adapt VLMs to robot control. Our extensive experimental results also reveal several interesting conclusions regarding the behavior of different pre-trained VLMs on manipulation tasks. We believe RoboFlamingo has the potential to be a cost-effective and easy-to-use solution for robotics manipulation, empowering everyone with the ability to fine-tune their own robotics policy.

Key facts. Cited by 344 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 33 of its 41 references that are in the map. First posted on arXiv 2 Nov 2023.

Builds on

33 of 41 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control28 Jul
  2. 2023PaLM-E: An Embodied Multimodal Language Model6 Mar

Other references · 31

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  6. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  9. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  10. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  11. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  12. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 849
19 more
  1. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  2. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  3. 2023Instruction Tuning with GPT-4cited by 676
  4. 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 564
  5. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  6. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 517
  7. 2018Universal Sentence Encodercited by 485
  8. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  9. 2017Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperationcited by 286
  10. 2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 254
  11. 2020Language Conditioned Imitation Learning over Unstructured Datacited by 208
  12. 2023Language-Driven Representation Learning for Roboticscited by 163
  13. 2022Goal-Conditioned Reinforcement Learning: Problems and Solutionscited by 145
  14. 2022What Matters in Language Conditioned Robotic Imitation Learning over Unstructured Datacited by 137
  15. 2023M3IT: A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuningcited by 126
  16. 2022Grounding Language with Visual Affordances over Unstructured Datacited by 101
  17. 2022EfficientVLM: Fast and Accurate Vision-Language Models via Knowledge Distillation and Modal-adaptive Pruningcited by 56
  18. 2020Energy-Based Imitation Learningcited by 39
  19. 2022X2-VLM: All-In-One Pre-trained Model For Vision-Language Taskscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 66; 2025: 166; 2026: 109.

344 papers.

Sort:

By area:

  1. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  2. 2025GR00T N1: An Open Foundation Model for Generalist Humanoid Robotscited by 1,167
  3. 2025Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Successcited by 844
  4. 2025SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Roboticscited by 507
  5. 2025SpatialVLA: Exploring Spatial Representations for Visual-Language-Action Modelcited by 502
  6. 2024CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulationcited by 411
  7. 2025UniVLA: Learning to Act Anywhere with Task-centric Latent Actionscited by 386
  8. 2025WorldVLA: Towards Autoregressive Action World Modelcited by 351
  9. 2024GR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge for Robot Manipulationcited by 290
  10. 2024Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representationscited by 279
  11. 20243D Diffuser Actor: Policy Diffusion with 3D Scene Representationscited by 278
  12. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
38 more
  1. 2024NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigationcited by 222
  2. 2025DreamVLA: A Vision-Language-Action Model Dreamed with Comprehensive World Knowledgecited by 203
  3. 2025MemoryVLA: Perceptual-Cognitive Memory in Vision-Language-Action Models for Robotic Manipulationcited by 183
  4. 2025HybridVLA: Collaborative Diffusion and Autoregression in a Unified Vision-Language-Action Modelcited by 176
  5. 2024Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AIcited by 175
  6. 2024Agent AI: Surveying the Horizons of Multimodal Interactioncited by 167
  7. 2024RoboMIND: Benchmark on Multi-embodiment Intelligence Normative Data for Robot Manipulationcited by 166
  8. 2024Consistency Policy: Accelerated Visuomotor Policies via Consistency Distillationcited by 159
  9. 2024Predictive Inverse Dynamics Models are Scalable Learners for Robotic Manipulationcited by 159
  10. 2025VLA-Adapter: An Effective Paradigm for Tiny-Scale Vision-Language-Action Modelcited by 140
  11. 2023Large Language Models for Robotics: A Surveycited by 132
  12. 2024Multimodal Diffusion Transformer: Learning Versatile Behavior from Multimodal Goalscited by 131
  13. 2025Unified Vision-Language-Action Modelcited by 128
  14. 2026π0.7: a Steerable Generalist Robotic Foundation Model with Emergent Capabilitiescited by 127
  15. 2024DeeR-VLA: Dynamic Inference of Multimodal Large Language Models for Efficient Robot Executioncited by 116
  16. 2025ChatVLA: Unified Multimodal Understanding and Robot Control with Vision-Language-Action Modelcited by 112
  17. 2025GR-3 Technical Reportcited by 110
  18. 2025Improving Vision-Language-Action Model with Online Reinforcement Learningcited by 107
  19. 2025OneTwoVLA: A Unified Vision-Language-Action Model with Adaptive Reasoningcited by 105
  20. 2025FLARE: Robot Learning with Implicit World Modelingcited by 105
  21. 2024RoboMamba: Efficient Vision-Language-Action Model for Robotic Reasoning and Manipulationcited by 102
  22. 2025UP-VLA: A Unified Understanding and Prediction Model for Embodied Agentcited by 99
  23. 2025EfficientVLA: Training-Free Acceleration and Compression for Vision-Language-Action Modelscited by 89
  24. 2024What Matters in Building Vision-Language-Action Models for Generalist Robotscited by 88
  25. 2025PD-VLA: Accelerating Vision-Language-Action Model Integrated with Action Chunking via Parallel Decodingcited by 88
  26. 2025VLA-Cache: Efficient Vision-Language-Action Manipulation via Adaptive Token Cachingcited by 83
  27. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 82
  28. 2025ReconVLA: Reconstructive Vision-Language-Action Model as Effective Robot Perceivercited by 80
  29. 2025OpenHelix: A Short Survey, Empirical Analysis, and Open-Source Dual-System VLA Model for Robotic Manipulationcited by 79
  30. 2025BridgeVLA: Input-Output Alignment for Efficient 3D Manipulation Learning with Vision-Language Modelscited by 77
  31. 2023GPT-4V(ision) for Robotics: Multimodal Task Planning from Human Demonstrationcited by 76
  32. 2025SAFE: Multitask Failure Detection for Vision-Language-Action Modelscited by 74
  33. 2025Dita: Scaling Diffusion Transformer for Generalist Vision-Language-Action Policycited by 73
  34. 2024HiRT: Enhancing Robotic Control with Hierarchical Robot Transformerscited by 72
  35. 2024Towards Synergistic, Generalized, and Efficient Dual-System for Robotic Manipulationcited by 71
  36. 2025MoLe-VLA: Dynamic Layer-skipping Vision Language Action Model via Mixture-of-Layers for Efficient Robot Manipulationcited by 69
  37. 2024Exploring the Adversarial Vulnerabilities of Vision-Language-Action Models in Roboticscited by 68
  38. 2025Large Video Planner Enables Generalizable Robot Controlcited by 67

50 most cited of 344 All 344 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Vision Language Action Models in Robotic Manipulation: A Systematic Review
  2. 2026EmbodiedMidtrain: Bridging the Gap between Vision-Language Models and Vision-Language-Action Models via Mid-training
  3. 2026M2-VLA: Boosting Vision-Language Models for Generalizable Manipulation via Layer Mixture and Meta-Skills
  4. 2025IA-VLA: Input Augmentation for Vision-Language-Action models in settings with semantically complex tasks
  5. 2025Seeing Across Views: Benchmarking Spatial Reasoning of Vision-Language Models in Robotic Scenes
  6. 2025From Intention to Execution: Probing the Generalization Boundaries of Vision-Language-Action Models
  7. 2025Improving Generalization of Language-Conditioned Robot Manipulation
  8. 2026Transferring the Intelligence of VLMs to Robotic Control