mlsanityData to 14 Oct 2026

EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thought

arXiv 2305.15021 ↗First posted 24 May 2023cs.RO cs.AI cs.CVcited by 279

On the map: LLM reasoning → Spatial and embodied reasoning

also in 1 more
Abstract (arXiv)

Embodied AI is a crucial frontier in robotics, capable of planning and executing action sequences for robots to accomplish long-horizon tasks in physical environments. In this work, we introduce EmbodiedGPT, an end-to-end multi-modal foundation model for embodied AI, empowering embodied agents with multi-modal understanding and execution capabilities. To achieve this, we have made the following efforts: (i) We craft a large-scale embodied planning dataset, termed EgoCOT. The dataset consists of carefully selected videos from the Ego4D dataset, along with corresponding high-quality language instructions. Specifically, we generate a sequence of sub-goals with the "Chain of Thoughts" mode for effective embodied planning. (ii) We introduce an efficient training approach to EmbodiedGPT for high-quality plan generation, by adapting a 7B large language model (LLM) to the EgoCOT dataset via prefix tuning. (iii) We introduce a paradigm for extracting task-related features from LLM-generated planning queries to form a closed loop between high-level planning and low-level control. Extensive experiments show the effectiveness of EmbodiedGPT on embodied tasks, including embodied planning, embodied control, visual captioning, and visual question answering. Notably, EmbodiedGPT significantly enhances the success rate of the embodied control task by extracting more effective features. It has achieved a remarkable 1.6 times increase in success rate on the Franka Kitchen benchmark and a 1.3 times increase on the Meta-World benchmark, compared to the BLIP-2 baseline fine-tuned with the Ego4D dataset.

Key facts. Cited by 279 papers in the map (data through 14 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 44 of its 73 references that are in the map. First posted on arXiv 24 May 2023.

Builds on

44 of 73 references.

References · 44

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  8. 2023Visual Instruction Tuningcited by 7,869
  9. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  10. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  11. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  12. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
32 more
  1. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  2. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  3. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  4. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
  5. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,204
  6. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190
  7. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  8. 2023VideoChat: Chat-Centric Video Understandingcited by 857
  9. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  10. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 694
  11. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 691
  12. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 668
  13. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 661
  14. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 622
  15. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  16. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 577
  17. 2020Rescaling Egocentric Visioncited by 540
  18. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 517
  19. 2018VirtualHome: Simulating Household Activities via Programscited by 415
  20. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 407
  21. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 395
  22. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learningcited by 384
  23. 2023LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Modelscited by 324
  24. 2022Egocentric Video-Language Pretrainingcited by 232
  25. 2021VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioningcited by 191
  26. 2018Charades-Ego: A Large-Scale Dataset of Paired Third and First Person Videoscited by 128
  27. 2023X-LLM: Bootstrapping Advanced Large Language Models by Treating Multi-Modalities as Foreign Languagescited by 128
  28. 2021VinVL: Revisiting Visual Representations in Vision-Language Modelscited by 110
  29. 2018When will you do what? - Anticipating Temporal Occurrences of Activitiescited by 107
  30. 2022VLMbench: A Compositional Benchmark for Vision-and-Language Manipulationcited by 68
  31. 2022AssistQ: Affordance-centric Question-driven Task Completion for Egocentric Assistantcited by 32
  32. 2022OpenD: A Benchmark for Language-Driven Door and Drawer Openingcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 45; 2024: 85; 2025: 104; 2026: 45.

279 papers.

Sort:

By area:

  1. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  2. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  3. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,439
  4. 2023A Survey on Multimodal Large Language Modelscited by 909
  5. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 906
  6. 2023ExpeL: LLM Agents Are Experiential Learnerscited by 876
  7. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spacescited by 611
  8. 2025CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Modelscited by 451
  9. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 424
  10. 2023An Embodied Generalist Agent in 3D Worldcited by 365
  11. 2025X-VLA: Soft-Prompted Transformer as Scalable Cross-Embodiment Vision-Language-Action Modelcited by 303
  12. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 302
38 more
  1. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 302
  2. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 279
  3. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 278
  4. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  5. 2023DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Drivingcited by 233
  6. 2024VideoLLM-online: Online Video Large Language Model for Streaming Videocited by 225
  7. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Modelscited by 217
  8. 2025DreamVLA: A Vision-Language-Action Model Dreamed with Comprehensive World Knowledgecited by 213
  9. 2025InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modelingcited by 211
  10. 2024GUIOdyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devicescited by 194
  11. 2024OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLMcited by 184
  12. 2025Multimodal Chain-of-Thought Reasoning: A Comprehensive Surveycited by 182
  13. 2024Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AIcited by 179
  14. 2023SVIT: Scaling up Visual Instruction Tuningcited by 162
  15. 2024Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learningcited by 158
  16. 2023Towards Learning a Generalist Model for Embodied Navigationcited by 138
  17. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 137
  18. 2024ShapeLLM: Universal 3D Object Understanding for Embodied Interactioncited by 137
  19. 2023Dolphins: Multimodal Language Model for Drivingcited by 124
  20. 2023MobileVLM : A Fast, Strong and Open Vision Language Assistant for Mobile Devicescited by 122
  21. 2025RoboTwin: Dual-Arm Robot Benchmark with Generative Digital Twinscited by 121
  22. 2024DeeR-VLA: Dynamic Inference of Multimodal Large Language Models for Efficient Robot Executioncited by 117
  23. 2023The All-Seeing Project: Towards Panoptic Visual Recognition and Understanding of the Open Worldcited by 116
  24. 2025OmniSpatial: Towards Comprehensive Spatial Reasoning Benchmark for Vision Language Modelscited by 114
  25. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 110
  26. 2024GRAPE: Generalizing Robot Policy via Preference Alignmentcited by 109
  27. 2025OneTwoVLA: A Unified Vision-Language-Action Model with Adaptive Reasoningcited by 107
  28. 2023Avalon's Game of Thoughts: Battle Against Deception through Recursive Contemplationcited by 104
  29. 2024VisualAgentBench: Towards Large Multimodal Models as Visual Foundation Agentscited by 103
  30. 2023Evaluation and Analysis of Hallucination in Large Vision-Language Modelscited by 99
  31. 2025EgoLife: Towards Egocentric Life Assistantcited by 91
  32. 2024MapGPT: Map-Guided Prompting with Adaptive Path Planning for Vision-and-Language Navigationcited by 86
  33. 2025Zebra-CoT: A Dataset for Interleaved Vision Language Reasoningcited by 84
  34. 2024AgentGen: Enhancing Planning Abilities for Large Language Model based Agent via Environment and Task Generationcited by 83
  35. 2025Universal Actions for Enhanced Embodied Foundation Modelscited by 82
  36. 2024NavCoT: Boosting LLM-Based Vision-and-Language Navigation via Learning Disentangled Reasoningcited by 79
  37. 2024Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Modelscited by 79
  38. 2024HiRT: Enhancing Robotic Control with Hierarchical Robot Transformerscited by 75

50 most cited of 279 All 279 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Revisiting Embodied Chain-of-Thought for Generalizable Robot Manipulation
  2. 2025EmbodiedBrain: Expanding Performance Boundaries of Task Planning for Embodied Intelligence
  3. 2026GEM: Generative Supervision Helps Embodied Intelligence
  4. 2026Thinker: A vision-language foundation model for embodied intelligence
  5. 2024Large Language Models for Robotics: Opportunities, Challenges, and Perspectives
  6. 2025Using Language Models as Closed-Loop High-Level Planners for Robotics Applications: A Brief Overview and Benchmarks
  7. 2024Multi-Agent Planning Using Visual Language Models
  8. 2024Can-Do! A Dataset and Neuro-Symbolic Grounded Framework for Embodied Planning with Large Multimodal Models