Data to 7 Oct 2026

Towards End-to-End Embodied Decision Making via Multi-modal Large Language Model: Explorations with GPT4-Vision and Beyond

arXiv 2310.02071 ↗First posted 3 Oct 2023cs.AI cs.CL cs.CVcited by 50

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

In this study, we explore the potential of Multimodal Large Language Models (MLLMs) in improving embodied decision-making processes for agents. While Large Language Models (LLMs) have been widely used due to their advanced reasoning skills and vast world knowledge, MLLMs like GPT4-Vision offer enhanced visual understanding and reasoning capabilities. We investigate whether state-of-the-art MLLMs can handle embodied decision-making in an end-to-end manner and whether collaborations between LLMs and MLLMs can enhance decision-making. To address these questions, we introduce a new benchmark called PCA-EVAL, which evaluates embodied decision-making from the perspectives of Perception, Cognition, and Action. Additionally, we propose HOLMES, a multi-agent cooperation framework that allows LLMs to leverage MLLMs and APIs to gather multimodal information for informed decision-making. We compare end-to-end embodied decision-making and HOLMES on our benchmark and find that the GPT4-Vision model demonstrates strong end-to-end embodied decision-making abilities, outperforming GPT4-HOLMES in terms of average decision accuracy (+3%). However, this performance is exclusive to the latest GPT4-Vision model, surpassing the open-source state-of-the-art MLLM by 26%. Our results indicate that powerful MLLMs like GPT4-Vision hold promise for decision-making in embodied agents, offering new avenues for MLLM research. Code and data are open at https://github.com/pkunlp-icler/PCA-EVAL/.

Key facts. Cited by 50 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 47 of its 60 references that are in the map. First posted on arXiv 3 Oct 2023.

Builds on

47 of 60 references.

Closest ancestors · 1

Same direction, newest first

2023PaLM-E: An Embodied Multimodal Language Model6 Mar

Other references · 46

2020Language Models are Few-Shot Learnerscited by 32,199
2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
2023Visual Instruction Tuningcited by 7,792
2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
2022Large Language Models are Zero-Shot Reasonerscited by 4,323
2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500

34 more

2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
2018YOLOv3: An Incremental Improvementcited by 2,652
2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
2022Emergent Abilities of Large Language Modelscited by 2,483
2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
2022Planning-oriented Autonomous Drivingcited by 1,068
2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
2023Large Language Models are not Fair Evaluatorscited by 1,050
2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 850
2023Large Language Modelscited by 774
2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 553
2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 511
2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 469
2023Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agentscited by 396
2023Chameleon: Plug-and-Play Compositional Reasoning with Large Language Modelscited by 352
2023Tool Learning with Foundation Modelscited by 302
2023Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memorycited by 288
2022Pre-Trained Language Models for Interactive Decision-Makingcited by 220
2023MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learningcited by 171
2023RestGPT: Connecting Large Language Models with Real-World RESTful APIscited by 129
2023M3IT: A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuningcited by 126
2023Retroformer: Retrospective Large Language Agents with Policy Gradient Optimizationcited by 110
2023Making Large Language Models Better Reasoners with Alignmentcited by 63
2023Prompt Pre-Training with Twenty-Thousand Classes for Open-Vocabulary Visual Recognitioncited by 35
2023Asking Before Acting: Gather Information in Embodied Decision Making with Language Modelscited by 12
2022Delving into the Openness of CLIPcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 6; 2024: 23; 2025: 18; 2026: 3.

50 papers.

Sort:

By area:

2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 895
2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Modelscited by 629
2024Exploring the Reasoning Abilities of Multimodal Large Language Models (MLLMs): A Comprehensive Survey on Emerging Trends in Multimodal Reasoningcited by 136
2024LLaVA-Critic: Learning to Evaluate Multimodal Modelscited by 132
2023Silkie: Preference Distillation for Large Visual Language Modelscited by 128
2024DeeR-VLA: Dynamic Inference of Multimodal Large Language Models for Efficient Robot Executioncited by 116
2024SafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM Agentscited by 102
2024From LLMs to LLM-based Agents for Software Engineering: A Survey of Current, Challenges and Futurecited by 85
2023EgoPlan-Bench: Benchmarking Multimodal Large Language Models for Human-Level Planningcited by 66
2024Scaffolding Coordinates to Promote Vision-Language Coordination in Large Multi-Modal Modelscited by 56
2024VLFeedback: A Large-Scale AI Feedback Dataset for Large Vision-Language Models Alignmentcited by 56
2024A Survey on Evaluation of Multimodal Large Language Modelscited by 46

38 more

2023BenchLMM: Benchmarking Cross-style Visual Capability of Large Multimodal Modelscited by 41
2024Asynchronous Large Language Model Enhanced Planner for Autonomous Drivingcited by 38
2024Debiasing Multimodal Large Language Models via Penalization of Language Priorscited by 36
2025Multi-agent Embodied AI: Advances and Future Directionscited by 36
2023Charting New Territories: Exploring the Geographic and Geospatial Capabilities of Multimodal LLMscited by 34
2024A Survey on Multimodal Benchmarks: In the Era of Large AI Modelscited by 27
2024VIALM: A Survey and Benchmark of Visually Impaired Assistance with Large Modelscited by 23
2024Step-by-Step Reasoning for Math Problems via Twisted Sequential Monte Carlocited by 21
2025Agent-RewardBench: Towards a Unified Benchmark for Reward Modeling across Perception, Planning, and Safety in Real-World Multimodal Agentscited by 19
2024MMEvalPro: Calibrating Multimodal Benchmarks Towards Trustworthy and Efficient Evaluationcited by 17
2024Cross-Modal Consistency in Multimodal Large Language Modelscited by 17
2025HSSBench: Benchmarking Humanities and Social Sciences Ability for Multimodal Large Language Modelscited by 17
2024Towards Rationality in Language and Multimodal Agents: A Surveycited by 15
2025CompileAgent: Automated Real-World Repo-Level Compilation with Tool-Integrated LLM-based Agent Systemcited by 15
2025VeriOS: Query-Driven Proactive Human-Agent-GUI Interaction for Trustworthy OS Agentscited by 15
2023NERIF: GPT-4V for Automatic Scoring of Drawn Modelscited by 13
2025Generative Multi-Agent Collaboration in Embodied AI: A Systematic Reviewcited by 13
2025Advancing Embodied Agent Security: From Safety Benchmarks to Input Moderationcited by 12
2024Problem Solving Through Human-AI Preference-Based Cooperationcited by 11
2025VisualSphinx: Large-Scale Synthetic Vision Logic Puzzles for RLcited by 11
2025Neural Brain: A Neuroscience-inspired Framework for Embodied Agentscited by 10
2025DIP-R1: Deep Inspection and Perception with RL Looking Through and Understanding Complex Scenescited by 10
2024Professional Agents -- Evolving Large Language Models into Autonomous Experts with Human-Level Competenciescited by 9
2024Modeling Collaborator: Enabling Subjective Vision Classification With Minimal Human Effort via LLM Tool-Usecited by 9
2024A Review of Multimodal Explainable Artificial Intelligence: Past, Present and Futurecited by 9
2025VLMPlanner: Integrating Visual Language Models with Motion Planningcited by 6
2026MMEmb-R1: Reasoning-Enhanced Multimodal Embedding with Pair-Aware Selection and Adaptive Controlcited by 6
2025VCRBench: Exploring Long-form Causal Reasoning Capabilities of Large Video Language Modelscited by 4
2025Towards Effective MLLM Jailbreaking Through Balanced On-Topicness and OOD-Intensitycited by 3
2025Foundation Models for Trajectory Planning in Autonomous Driving: A Review of Progress and Open Challengescited by 3
2024Automatic benchmarking of large multimodal models via iterative experiment programmingcited by 2
2026When Should a Robot Think? Resource-Aware Reasoning via Reinforcement Learning for Embodied Robotic Decision-Makingcited by 2
2025MOVE: A Mixture-of-Vision-Encoders Approach for Domain-Focused Vision-Language Processingcited by 1
2025CoSpace: Benchmarking Continuous Space Perception Ability for Vision-Language Modelscited by 1
2026ASSCG: Just-Right Gating over Chattering for Fast-Slow LLM Planning in Autonomous Drivingcited by 1
2024MLAD: A Unified Model for Multi-system Log Anomaly Detectioncited by 0
2025VIVA+: Human-Centered Situational Decision-Makingcited by 0
2025ColorEcosystem: Powering Personalized, Standardized, and Trustworthy Agentic Service in massive-agent Ecosystemcited by 0

Similar, not cited

Close in wording, no citation either way