mlsanityData to 7 Oct 2026

Instruct2Act: Mapping Multi-modality Instructions to Robotic Actions with Large Language Model

arXiv 2305.11176 ↗First posted 18 May 2023cs.RO cs.AIcited by 115

Not on the map. Why?

Cited by 31 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Foundation models have made significant strides in various applications, including text-to-image generation, panoptic segmentation, and natural language processing. This paper presents Instruct2Act, a framework that utilizes Large Language Models to map multi-modal instructions to sequential actions for robotic manipulation tasks. Specifically, Instruct2Act employs the LLM model to generate Python programs that constitute a comprehensive perception, planning, and action loop for robotic tasks. In the perception section, pre-defined APIs are used to access multiple foundation models where the Segment Anything Model (SAM) accurately locates candidate objects, and CLIP classifies them. In this way, the framework leverages the expertise of foundation models and robotic abilities to convert complex high-level instructions into precise policy codes. Our approach is adjustable and flexible in accommodating various instruction modalities and input types and catering to specific task demands. We validated the practicality and efficiency of our approach by assessing it on robotic tasks in different scenarios within tabletop manipulation domains. Furthermore, our zero-shot method outperformed many state-of-the-art learning-based policies in several tasks. The code for our proposed approach is available at https://github.com/OpenGVLab/Instruct2Act, serving as a robust benchmark for high-level robotic instruction tasks with assorted modality inputs.

Key facts. Cited by 115 papers in the map (data through 7 Oct 2026). Builds on 40 of its 55 references that are in the map. First posted on arXiv 18 May 2023.

Builds on

40 of 55 references.

References · 40

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2022Training language models to follow instructions with human feedbackcited by 15,712
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2023Segment Anythingcited by 8,158
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  9. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  10. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  11. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  12. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
28 more
  1. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  2. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  3. 2021Grounded Language-Image Pre-trainingcited by 1,147
  4. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  5. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  6. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  7. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  8. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  9. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  10. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  11. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  12. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 591
  13. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
  14. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  15. 2022Visual Programming: Compositional visual reasoning without trainingcited by 507
  16. 2022InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutionscited by 479
  17. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  18. 2020Transporter Networks: Rearranging the Visual World for Robotic Manipulationcited by 342
  19. 2022Real-World Robot Learning with Masked Visual Pre-trainingcited by 241
  20. 2023Text2Motion: From Natural Language Instructions to Feasible Planscited by 223
  21. 2022Pre-Trained Language Models for Interactive Decision-Makingcited by 220
  22. 2023TidyBot: Personalized Robot Assistance with Large Language Modelscited by 219
  23. 2021VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modelingcited by 187
  24. 2022DALL-E-Bot: Introducing Web-Scale Diffusion Models to Roboticscited by 105
  25. 2023InternGPT: Solving Vision-Centric Tasks by Interacting with ChatGPT Beyond Languagecited by 96
  26. 2022CACTI: A Framework for Scalable Multi-Task Multi-Scene Visual Imitation Learningcited by 70
  27. 2023"No, to the Right" -- Online Language Corrections for Robotic Manipulation via Shared Autonomycited by 60
  28. 2018Interactive Visual Grounding of Referring Expressions for Human-Robot Interactioncited by 55

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 23; 2024: 36; 2025: 34; 2026: 22.

115 papers.

Sort:

By area:

  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  2. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
  3. 2023Eureka: Human-Level Reward Design via Coding Large Language Modelscited by 542
  4. 2023Cognitive Architectures for Language Agentscited by 422
  5. 2023LangSplat: 3D Language Gaussian Splattingcited by 410
  6. 2023Visual Adversarial Examples Jailbreak Aligned Large Language Modelscited by 307
  7. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  8. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  9. 2025EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agentscited by 213
  10. 2023DiLu: A Knowledge-Driven Approach to Autonomous Driving with Large Language Modelscited by 204
  11. 2023RoCo: Dialectic Multi-Robot Collaboration with Large Language Modelscited by 171
  12. 2023Drive Like a Human: Rethinking Autonomous Driving with Large Language Modelscited by 158
38 more
  1. 2023SMART-LLM: Smart Multi-Agent Robot Task Planning using Large Language Modelscited by 151
  2. 2025Genie Envisioner: A Unified World Foundation Platform for Robotic Manipulationcited by 129
  3. 2024ChartX & ChartVLM: A Versatile Benchmark and Foundation Model for Complicated Chart Reasoningcited by 127
  4. 2024SG-Nav: Online 3D Scene Graph Prompting for LLM-based Zero-shot Object Navigationcited by 106
  5. 2023Drive as You Speak: Enabling Human-Like Interaction with Large Language Models in Autonomous Vehiclescited by 100
  6. 2023Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysiscited by 99
  7. 2024RAG-Driver: Generalisable Driving Explanations with Retrieval-Augmented In-Context Learning in Multi-Modal Large Language Modelcited by 91
  8. 2024Large Language Models for Robotics: Opportunities, Challenges, and Perspectivescited by 88
  9. 2024Vision Language Models are In-Context Value Learnerscited by 84
  10. 2024DrEureka: Language Model Guided Sim-To-Real Transfercited by 78
  11. 2023Octopus: Embodied Vision-Language Programmer from Environmental Feedbackcited by 73
  12. 2023Receive, Reason, and React: Drive as You Say with Large Language Models in Autonomous Vehiclescited by 72
  13. 2026CaP-X: A Framework for Benchmarking and Improving Coding Agents for Robot Manipulationcited by 72
  14. 2025Large VLM-based Vision-Language-Action Models for Robotic Manipulation: A Surveycited by 66
  15. 2025A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challengescited by 46
  16. 2025Pure Vision Language Action (VLA) Models: A Comprehensive Surveycited by 46
  17. 2023Creative Robot Tool Use with Large Language Modelscited by 42
  18. 2024ImagineNav: Prompting Vision-Language Models as Embodied Navigator through Scene Imaginationcited by 38
  19. 2026Harness VLA: Steering Frozen VLAs into Reliable Manipulation Primitives via Memory-Guided Agentscited by 37
  20. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
  21. 2024A3VLM: Actionable Articulation-Aware Vision Language Modelcited by 33
  22. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  23. 2024A Large Language Model-Driven Reward Design Framework via Dynamic Feedback for Reinforcement Learningcited by 29
  24. 2024ManipVQA: Injecting Robotic Affordance and Physically Grounded Information into Multi-Modal Large Language Modelscited by 25
  25. 2025Video2Policy: Scaling up Manipulation Tasks in Simulation through Internet Videoscited by 21
  26. 2024Bi-VLA: Vision-Language-Action Model-Based System for Bimanual Robotic Dexterous Manipulationscited by 20
  27. 2023Making Harmful Behaviors Unlearnable for Large Language Modelscited by 19
  28. 2023Holistic Evaluation of GPT-4V for Biomedical Imagingcited by 19
  29. 2025Large language models for artificial general intelligence (AGI): A survey of foundational principles and approachescited by 18
  30. 2026RHO: Your Coding Agent is Secretly a Roboticistcited by 18
  31. 2025Seeing to Act, Prompting to Specify: A Bayesian Factorization of Vision Language Action Policycited by 17
  32. 2026Show-Harness: Just a VLM Agent Can Play Robotscited by 17
  33. 2024On the Vulnerability of LLM/VLM-Controlled Roboticscited by 15
  34. 2024WorldAfford: Affordance Grounding based on Natural Language Instructionscited by 14
  35. 2024Eurekaverse: Environment Curriculum Generation via Large Language Modelscited by 14
  36. 2025ENACT: Evaluating Embodied Cognition with World Modeling of Egocentric Interactioncited by 14
  37. 2024Towards Open-World Grasping with Large Vision-Language Modelscited by 13
  38. 2025Self-Consistency of the Internal Reward Models Improves Self-Rewarding Language Modelscited by 13

50 most cited of 115 All 115 (JSON)