mlsanityData to 7 Oct 2026

VIMA: General Robot Manipulation with Multimodal Prompts

arXiv 2210.03094 ↗First posted 6 Oct 2022cs.RO cs.AI cs.LGcited by 403

Not on the map. Why?

Cited by 73 in Vision-language-action models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Prompt-based learning has emerged as a successful paradigm in natural language processing, where a single general-purpose language model can be instructed to perform any task specified by input prompts. Yet task specification in robotics comes in various forms, such as imitating one-shot demonstrations, following language instructions, and reaching visual goals. They are often considered different tasks and tackled by specialized models. We show that a wide spectrum of robot manipulation tasks can be expressed with multimodal prompts, interleaving textual and visual tokens. Accordingly, we develop a new simulation benchmark that consists of thousands of procedurally-generated tabletop tasks with multimodal prompts, 600K+ expert trajectories for imitation learning, and a four-level evaluation protocol for systematic generalization. We design a transformer-based robot agent, VIMA, that processes these prompts and outputs motor actions autoregressively. VIMA features a recipe that achieves strong model scalability and data efficiency. It outperforms alternative designs in the hardest zero-shot generalization setting by up to 2.9× task success rate given the same training data. With 10× less training data, VIMA still performs 2.7× better than the best competing variant. Code and video demos are available at https://vimalabs.github.io/

Key facts. Cited by 403 papers in the map (data through 7 Oct 2026). Builds on 119 of its 174 references that are in the map. First posted on arXiv 6 Oct 2022.

Builds on

119 of 174 references.

References · 119

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  8. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  9. 2017Mask R-CNNcited by 7,259
  10. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  11. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  12. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
38 more
  1. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  2. 2022Emergent Abilities of Large Language Modelscited by 2,483
  3. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  4. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  5. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  6. 2020GLU Variants Improve Transformercited by 1,724
  7. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  8. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  9. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  10. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  11. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  12. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  13. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  14. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  15. 2021Transformers in Vision: A Surveycited by 1,009
  16. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  17. 2022A Generalist Agentcited by 802
  18. 2020Efficient Transformers: A Surveycited by 784
  19. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  20. 2019Big Transfer (BiT): General Visual Representation Learningcited by 766
  21. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  22. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
  23. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  24. 2019RLBench: The Robot Learning Benchmark & Learning Environmentcited by 669
  25. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  26. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  27. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  28. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 616
  29. 2022Google Scanned Objects: A High-Quality Dataset of 3D Scanned Household Itemscited by 601
  30. 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 564
  31. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
  32. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  33. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
  34. 2021Habitat 2.0: Training Home Assistants to Rearrange their Habitatcited by 501
  35. 2020robosuite: A Modular Simulation Framework and Benchmark for Robot Learningcited by 484
  36. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 469
  37. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 435
  38. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 420

50 most cited of 119 All 119 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 88; 2024: 119; 2025: 104; 2026: 88.

403 papers.

Sort:

By area:

  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  2. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  3. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  4. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  5. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  6. 2024RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robotscited by 391
  7. 2023An Embodied Generalist Agent in 3D Worldcited by 355
  8. 2023Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulationcited by 353
  9. 2023Vision-Language Foundation Models as Effective Robot Imitatorscited by 344
  10. 2023MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrationscited by 308
  11. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
  12. 2023ChatGPT for Robotics: Design Principles and Model Abilitiescited by 296
38 more
  1. 2024GR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge for Robot Manipulationcited by 290
  2. 2023EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thoughtcited by 276
  3. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  4. 2023Text2Motion: From Natural Language Instructions to Feasible Planscited by 223
  5. 2023MimicPlay: Long-Horizon Imitation Learning by Watching Human Playcited by 214
  6. 2023ViNT: A Foundation Model for Visual Navigationcited by 210
  7. 20233D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignmentcited by 201
  8. 2023Grounding Large Language Models in Interactive Environments with Online Reinforcement Learningcited by 193
  9. 2024PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMscited by 187
  10. 2023Scaling Robot Learning with Semantically Imagined Experiencecited by 175
  11. 2024Scene-LLM: Extending Language Model for 3D Visual Understanding and Reasoningcited by 170
  12. 2023RoboAgent: Generalization and Efficiency in Robot Manipulation via Semantic Augmentations and Action Chunkingcited by 167
  13. 2023Foundation Models for Decision Making: Problems, Methods, and Opportunitiescited by 160
  14. 2023Multimodal Web Navigation with Instruction-Finetuned Foundation Modelscited by 159
  15. 2023Open-World Object Manipulation using Pre-trained Vision-Language Modelscited by 155
  16. 2024SpatialBot: Precise Spatial Understanding with Vision Language Modelscited by 146
  17. 2024Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Makingcited by 142
  18. 2024ShapeLLM: Universal 3D Object Understanding for Embodied Interactioncited by 133
  19. 2023RT-Trajectory: Robotic Task Generalization via Hindsight Trajectory Sketchescited by 131
  20. 2024Multimodal Diffusion Transformer: Learning Versatile Behavior from Multimodal Goalscited by 131
  21. 2023Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functionscited by 129
  22. 2026π0.7: a Steerable Generalist Robotic Foundation Model with Emergent Capabilitiescited by 127
  23. 2025HAMSTER: Hierarchical Action Models For Open-World Robot Manipulationcited by 125
  24. 2023GenSim: Generating Robotic Simulation Tasks via Large Language Modelscited by 120
  25. 2025RoboTwin: Dual-Arm Robot Benchmark with Generative Digital Twinscited by 117
  26. 2024PhyScene: Physically Interactable 3D Scene Synthesis for Embodied AIcited by 116
  27. 2023Instruct2Act: Mapping Multi-modality Instructions to Robotic Actions with Large Language Modelcited by 115
  28. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 109
  29. 2023RoboCat: A Self-Improving Generalist Agent for Robotic Manipulationcited by 108
  30. 2024Best Practices and Lessons Learned on Synthetic Datacited by 104
  31. 2024DexMimicGen: Automated Data Generation for Bimanual Dexterous Manipulation via Imitation Learningcited by 104
  32. 2023ADriver-I: A General World Model for Autonomous Drivingcited by 102
  33. 2023Recent Advancements in End-to-End Autonomous Driving using Deep Learning: A Surveycited by 101
  34. 2024Steering Your Generalists: Improving Robotic Foundation Models via Value Guidancecited by 100
  35. 2023Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysiscited by 99
  36. 2023STEVE-1: A Generative Model for Text-to-Behavior in Minecraftcited by 95
  37. 2024GOAT-Bench: A Benchmark for Multi-Modal Lifelong Navigationcited by 93
  38. 2025TesserAct: Learning 4D Embodied World Modelscited by 90

50 most cited of 403 All 403 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025RoboBERT: An End-to-end Multimodal Robotic Manipulation Model
  2. 2025REALM: A Real-to-Sim Validated Benchmark for Generalization in Robotic Manipulation
  3. 2025See Once, Then Act: Vision-Language-Action Model with Task Learning from One-Shot Video Demonstrations
  4. 2024VLABench: A Large-Scale Benchmark for Language-Conditioned Robotics Manipulation with Long-Horizon Reasoning Tasks
  5. 2025Interleave-VLA: Enhancing Robot Manipulation with Interleaved Image-Text Instructions
  6. 2026RoboTalk: Learning Multi-Robot Communication and Coordination from Multimodal Demonstrations
  7. 2025From Intention to Execution: Probing the Generalization Boundaries of Vision-Language-Action Models
  8. 2024Learning Efficient and Robust Language-conditioned Manipulation using Textual-Visual Relevancy and Equivariant Language Mapping