mlsanityData to 14 Oct 2026

EgoPlan-Bench: Benchmarking Multimodal Large Language Models for Human-Level Planning

arXiv 2312.06722 ↗First posted 11 Dec 2023cs.CV cs.CL cs.ROcited by 69

On the map: LLM reasoning → Spatial and embodied reasoning

also in 1 more
Abstract (arXiv)

The pursuit of artificial general intelligence (AGI) has been accelerated by Multimodal Large Language Models (MLLMs), which exhibit superior reasoning, generalization capabilities, and proficiency in processing multimodal inputs. A crucial milestone in the evolution of AGI is the attainment of human-level planning, a fundamental ability for making informed decisions in complex environments, and solving a wide range of real-world problems. Despite the impressive advancements in MLLMs, a question remains: How far are current MLLMs from achieving human-level planning? To shed light on this question, we introduce EgoPlan-Bench, a comprehensive benchmark to evaluate the planning abilities of MLLMs in real-world scenarios from an egocentric perspective, mirroring human perception. EgoPlan-Bench emphasizes the evaluation of planning capabilities of MLLMs, featuring realistic tasks, diverse action plans, and intricate visual observations. Our rigorous evaluation of a wide range of MLLMs reveals that EgoPlan-Bench poses significant challenges, highlighting a substantial scope for improvement in MLLMs to achieve human-level task planning. To facilitate this advancement, we further present EgoPlan-IT, a specialized instruction-tuning dataset that effectively enhances model performance on EgoPlan-Bench. We have made all codes, data, and a maintained benchmark leaderboard available to advance future research.

Key facts. Cited by 69 papers in the map (data through 14 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 62 of its 94 references that are in the map. First posted on arXiv 11 Dec 2023.

Builds on

62 of 94 references.

References · 62

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2023Visual Instruction Tuningcited by 7,869
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  8. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  9. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  10. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  11. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  12. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
38 more
  1. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  2. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  3. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  4. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  5. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  6. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  7. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  8. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  9. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  10. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,109
  11. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  12. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
  13. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  14. 2023VideoChat: Chat-Centric Video Understandingcited by 857
  15. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 708
  16. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 691
  17. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  18. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 550
  19. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 547
  20. 2020Rescaling Egocentric Visioncited by 540
  21. 2019COIN: A Large-scale Dataset for Comprehensive Instructional Video Analysiscited by 318
  22. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  23. 2023MultiModal-GPT: A Vision and Language Model for Dialogue with Humanscited by 255
  24. 2023Valley: Video Assistant with Large Language model Enhanced abilitYcited by 234
  25. 2022Egocentric Video-Language Pretrainingcited by 232
  26. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Modelscited by 217
  27. 2016Deep Predictive Coding Networks for Video Prediction and Unsupervised Learningcited by 204
  28. 2019Cross-task weakly supervised learning from instructional videoscited by 199
  29. 2023Making LLaMA SEE and Draw with SEED Tokenizercited by 185
  30. 2017Decomposing Motion and Content for Natural Video Sequence Predictioncited by 169
  31. 2023LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmarkcited by 169
  32. 2023Structured World Models from Human Videoscited by 147
  33. 2023Planting a SEED of Vision in Large Language Modelcited by 140
  34. 2018Charades-Ego: A Large-Scale Dataset of Paired Third and First Person Videoscited by 128
  35. 2022EgoTaskQA: Understanding Human Tasks in Egocentric Videoscited by 115
  36. 2022Joint Hand Motion and Interaction Hotspots Prediction from Egocentric Videoscited by 111
  37. 2023SEED-Bench-2: Benchmarking Multimodal Large Language Modelscited by 97
  38. 2019Procedure Planning in Instructional Videoscited by 92

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 28; 2026: 36.

69 papers.

Sort:

By area:

  1. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 279
  2. 2025ThinkAct: Vision-Language-Action Reasoning via Reinforced Visual Latent Planningcited by 156
  3. 2024A Survey on Benchmarks of Multimodal Large Language Modelscited by 82
  4. 2025WoW: Towards a World omniscient World model Through Embodied Interactioncited by 67
  5. 2025DeepVideo-R1: Video Reinforcement Fine-Tuning via Difficulty-aware Regressive GRPOcited by 61
  6. 2025EmbodiedEval: Evaluate Multimodal LLMs as Embodied Agentscited by 57
  7. 2025GRPO-CARE: Consistency-Aware Reinforcement Learning for Multimodal Reasoningcited by 47
  8. 2025ARC-Hunyuan-Video-7B: Structured Video Comprehension of Real-World Shortscited by 41
  9. 2024VidEgoThink: Assessing Egocentric Video Understanding Capabilities for Embodied AIcited by 34
  10. 2025Eyes Wide Open: Ego Proactive Video-LLM for Streaming Videocited by 28
  11. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Modelscited by 27
  12. 2025Exploring the Effect of Reinforcement Learning on Video Understanding: Insights from SEED-Bench-R1cited by 25
38 more
  1. 2025Robobench: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models as Embodied Braincited by 25
  2. 2026Fast-ThinkAct: Efficient Vision-Language-Action Reasoning via Verbalizable Latent Planningcited by 24
  3. 2026DIAL: Decoupling Intent and Action via Latent World Modeling for End-to-End VLAcited by 19
  4. 2024Eureka: Evaluating and Understanding Large Foundation Modelscited by 18
  5. 2025HSSBench: Benchmarking Humanities and Social Sciences Ability for Multimodal Large Language Modelscited by 17
  6. 2026Wow, wo, val! A Comprehensive Embodied World Model Evaluation Turing Testcited by 13
  7. 2025EgoToM: Benchmarking Theory of Mind Reasoning from Egocentric Videoscited by 12
  8. 2025EgoVLM: Policy Optimization for Egocentric Video Understandingcited by 12
  9. 2025PhysBrain: Human Egocentric Data as a Bridge from Vision Language Models to Physical Intelligencecited by 11
  10. 2026PRTS: A Primitive Reasoning and Tasking System via Contrastive Representationscited by 9
  11. 2025Dissecting Embodied Abilities in Multimodal Language Models through Skill-level Evaluation and Diagnosiscited by 8
  12. 2025Mind to Hand: Purposeful Robotic Control via Embodied Reasoningcited by 8
  13. 2025PLANET: A Collection of Benchmarks for Evaluating LLMs' Planning Capabilitiescited by 7
  14. 2025Unified Embodied VLM Reasoning with Robotic Action via Autoregressive Discretized Pre-trainingcited by 7
  15. 2025Cook and Clean Together: Teaching Embodied Agents for Parallel Task Executioncited by 6
  16. 2026ROMA: Real-time Omni-Multimodal Assistant with Interactive Streaming Understandingcited by 6
  17. 2026Embodied-R1.5: Evolving Physical Intelligence via Embodied Foundation Modelscited by 6
  18. 2026EgoSAT: A Comprehensive Benchmark of Egocentric Streaming Interaction Understandingcited by 6
  19. 2026Data Pyramid for Embodied Manipulation: A Surveycited by 6
  20. 2025VCRBench: Exploring Long-form Causal Reasoning Capabilities of Large Video Language Modelscited by 5
  21. 2025ViPlan: A Benchmark for Visual Planning with Symbolic Predicates and Vision-Language Modelscited by 5
  22. 2025MindPower: Enabling Theory-of-Mind Reasoning in VLM-based Embodied Agentscited by 5
  23. 2026SPIRAL: Self-Evolving Action-Conditioned Video Generation via Reflective Planning Agentscited by 5
  24. 2025Surgical Action Planning with Large Language Modelscited by 4
  25. 2025EmbodiedBrain: Expanding Performance Boundaries of Task Planning for Embodied Intelligencecited by 4
  26. 2025EgoVITA: Learning to Plan and Verify for Egocentric Video Reasoningcited by 4
  27. 2025Ego-EXTRA: video-language Egocentric Dataset for EXpert-TRAinee assistancecited by 4
  28. 2026EgoProx: Evaluating MLLMs on Egocentric 3D Proximity Reasoning Across a Cognitive Hierarchycited by 4
  29. 2026LABSHIELD: A Multimodal Benchmark for Safety-Critical Reasoning and Planning in Scientific Laboratoriescited by 3
  30. 2026Towards Predictive, Aligned, and Scalable Robot Learningcited by 3
  31. 2025TIME: Temporal-Sensitive Multi-Dimensional Instruction Tuning and Robust Benchmarking for Video-LLMscited by 2
  32. 2026Children's Intelligence Tests Pose Challenges for MLLMs? KidGym: A 2D Grid-Based Reasoning Benchmark for MLLMscited by 2
  33. 2025SAP-Bench: Benchmarking Multimodal Large Language Models in Surgical Action Planningcited by 1
  34. 2026Thinker: A vision-language foundation model for embodied intelligencecited by 1
  35. 2026LLM-WikiRace Benchmark: How Far Can LLMs Plan over Real-World Knowledge Graphs?cited by 1
  36. 2026EXPLORE-Bench: Egocentric Scene Prediction with Long-Horizon Reasoningcited by 1
  37. 2026EgoIntent: A Pre-Outcome Micro-Step Benchmark for Understanding What, Why, and Nextcited by 1
  38. 2026SVI-Bench: A Dynamic Microworld for Strategic Video Intelligencecited by 1

50 most cited of 69 All 69 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024EgoPlan-Bench2: A Benchmark for Multimodal Large Language Model Planning in Real-World Scenarios
  2. 2025On Path to Multimodal Generalist: General-Level and General-Bench
  3. 2024ActPlan-1K: Benchmarking the Procedural Planning Ability of Visual Language Models in Household Activities
  4. 2025LLMs Can Plan Only If We Tell Them
  5. 2024Egocentric Vision Language Planning
  6. 2024PCA-Bench: Evaluating Multimodal Large Language Models in Perception-Cognition-Action Chain
  7. 2023On the Planning Abilities of Large Language Models (A Critical Investigation with a Proposed Benchmark)
  8. 2025MPCC: A Novel Benchmark for Multimodal Planning with Complex Constraints in Multimodal Large Language Models