mlsanityData to 7 Oct 2026

EgoThink: Evaluating First-Person Perspective Thinking Capability of Vision-Language Models

arXiv 2311.15596 ↗First posted 27 Nov 2023cs.CV cs.CLcited by 59

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

Vision-language models (VLMs) have recently shown promising results in traditional downstream tasks. Evaluation studies have emerged to assess their abilities, with the majority focusing on the third-person perspective, and only a few addressing specific tasks from the first-person perspective. However, the capability of VLMs to "think" from a first-person perspective, a crucial attribute for advancing autonomous agents and robotics, remains largely unexplored. To bridge this research gap, we introduce EgoThink, a novel visual question-answering benchmark that encompasses six core capabilities with twelve detailed dimensions. The benchmark is constructed using selected clips from egocentric videos, with manually annotated question-answer pairs containing first-person information. To comprehensively assess VLMs, we evaluate eighteen popular VLMs on EgoThink. Moreover, given the open-ended format of the answers, we use GPT-4 as the automatic judge to compute single-answer grading. Experimental results indicate that although GPT-4V leads in numerous dimensions, all evaluated VLMs still possess considerable potential for improvement in first-person perspective tasks. Meanwhile, enlarging the number of trainable parameters has the most significant impact on model performance on EgoThink. In conclusion, EgoThink serves as a valuable addition to existing evaluation benchmarks for VLMs, providing an indispensable resource for future research in the realm of embodied artificial intelligence and robotics.

Key facts. Cited by 59 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 33 of its 91 references that are in the map. First posted on arXiv 27 Nov 2023.

Builds on

33 of 91 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  2. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model28 Apr

Other references · 31

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  6. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  7. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  8. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  9. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  10. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  11. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  12. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
19 more
  1. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  2. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  3. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
  4. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  5. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  6. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 330
  7. 2023Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4Vcited by 291
  8. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 264
  9. 2023OpenChat: Advancing Open-source Language Models with Mixed-Quality Datacited by 250
  10. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Modelscited by 217
  11. 2023Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved with Textcited by 208
  12. 2023Physically Grounded Vision-Language Models for Robotic Manipulationcited by 163
  13. 2022F-VLM: Open-Vocabulary Object Detection upon Frozen Vision and Language Modelscited by 149
  14. 2022VL-CheckList: Evaluating Pre-trained Vision-Language Models with Objects, Attributes and Relationscited by 113
  15. 2023Octopus: Embodied Vision-Language Programmer from Environmental Feedbackcited by 73
  16. 2023Towards Language Models That Can See: Computer Vision Through the LENS of Natural Languagecited by 53
  17. 2023Distilling Internet-Scale Vision-Language Models into Embodied Agentscited by 30
  18. 2022VLUE: A Multi-Task Benchmark for Evaluating Vision-Language Modelscited by 11
  19. 2023TinyLVLM-eHub: Towards Comprehensive and Efficient Evaluation for Large Vision-Language Modelscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 31; 2026: 22.

59 papers.

Sort:

By area:

  1. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 257
  2. 2025VisuLogic: A Benchmark for Evaluating Visual Reasoning in Multi-modal Large Language Modelscited by 102
  3. 2025EgoLife: Towards Egocentric Life Assistantcited by 86
  4. 2025EgoTextVQA: Towards Egocentric Scene-Text Aware Video Question Answeringcited by 43
  5. 2025Omnia de EgoTempo: Benchmarking Temporal Understanding of Multi-Modal LLMs in Egocentric Videoscited by 41
  6. 2024EVA: An Embodied World Model for Future Video Anticipationcited by 28
  7. 2024MM-Ego: Towards Building Egocentric Multimodal LLMs for Video QAcited by 24
  8. 2025Robobench: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models as Embodied Braincited by 24
  9. 2025In the Eye of MLLM: Benchmarking Egocentric Video Intent Understanding with Gaze-Guided Promptingcited by 21
  10. 2025Challenges and Trends in Egocentric Vision: A Surveycited by 20
  11. 2025EgoBlind: Towards Egocentric Visual Assistance for the Blindcited by 19
  12. 2025EgoCross: Benchmarking Multimodal Large Language Models for Cross-Domain Egocentric Video Question Answeringcited by 19
38 more
  1. 2024Instruction-Guided Visual Maskingcited by 17
  2. 2025EgoNight: Towards Egocentric Vision Understanding at Night with a Challenging Benchmarkcited by 16
  3. 2024ET-Plan-Bench: Embodied Task-level Planning Benchmark Towards Spatial-Temporal Cognition with Foundation Modelscited by 11
  4. 2025Euclid's Gift: Enhancing Spatial Perception and Reasoning in Vision-Language Models via Geometric Surrogate Taskscited by 11
  5. 2025StreamEQA: Towards Streaming Video Understanding for Embodied Scenarioscited by 11
  6. 2026Wow, wo, val! A Comprehensive Embodied World Model Evaluation Turing Testcited by 11
  7. 2025PhysBrain: Human Egocentric Data as a Bridge from Vision Language Models to Physical Intelligencecited by 10
  8. 2025TeleEgo: Benchmarking Egocentric AI Assistants in the Wildcited by 9
  9. 2025Advancing Egocentric Video Question Answering with Multimodal Large Language Modelscited by 8
  10. 2025Benchmarking Egocentric Multimodal Goal Inference for Assistive Wearable Agentscited by 8
  11. 2026EgoSound: Benchmarking Sound Understanding in Egocentric Videoscited by 8
  12. 2025HIS-GPT: Towards 3D Human-In-Scene Multimodal Understandingcited by 6
  13. 2025Towards Comprehensive Scene Understanding: Integrating First and Third-Person Views for LVLMscited by 6
  14. 2026Recurrent Reasoning with Vision-Language Models for Estimating Long-Horizon Embodied Task Progresscited by 6
  15. 2026Ego-Grounding for Personalized Question-Answering in Egocentric Videoscited by 6
  16. 2026EgoAction: Egocentric Action Composition with Reliability-Aware Temporal Fusion for the EPIC-KITCHENS Action Detection Challenge at CVPR 2026cited by 6
  17. 2025ANNEXE: Unified Analyzing, Answering, and Pixel Grounding for Egocentric Interactioncited by 5
  18. 2025EASG-Bench: Video Q&A Benchmark with Egocentric Action Scene Graphscited by 5
  19. 2026MA-EgoQA: Question Answering over Egocentric Videos from Multiple Embodied Agentscited by 5
  20. 2025EmbodiedBrain: Expanding Performance Boundaries of Task Planning for Embodied Intelligencecited by 4
  21. 2025EgoExo-Con: Exploring View-Invariant Video Temporal Understandingcited by 4
  22. 2025EgoVITA: Learning to Plan and Verify for Egocentric Video Reasoningcited by 4
  23. 2025MindPower: Enabling Theory-of-Mind Reasoning in VLM-based Embodied Agentscited by 4
  24. 2026Ego2Web: A Web Agent Benchmark Grounded in Egocentric Videoscited by 4
  25. 2026The Yes-Man Syndrome: Benchmarking Abstention in Embodied Robotic Agentscited by 4
  26. 2026OmniEgo-R2: A Routed Reasoning Framework for the 1st Cross-Domain EgoCross Challenge at CVPR 2026cited by 4
  27. 2025UVLM: Benchmarking Video Language Model for Underwater World Understandingcited by 3
  28. 2025Ego-EXTRA: video-language Egocentric Dataset for EXpert-TRAinee assistancecited by 3
  29. 2024First Multi-Dimensional Evaluation of Flowchart Comprehension for Multimodal Large Language Modelscited by 2
  30. 2025Zero-Shot Temporal Interaction Localization for Egocentric Videoscited by 2
  31. 2025BLM1: A Boundless Large Model for Cross-Space, Cross-Task, and Cross-Embodiment Learningcited by 2
  32. 2025HanDyVQA: A Video QA Benchmark for Fine-Grained Hand-Object Interaction Dynamicscited by 2
  33. 2026EgoAVU: Egocentric Audio-Visual Understandingcited by 2
  34. 2026EgoSelf: From Memory to Personalized Egocentric Assistantcited by 2
  35. 2026VisualClaw: A Real-Time, Personalized Agent for the Physical Worldcited by 2
  36. 2026EgoSAT: A Comprehensive Benchmark of Egocentric Streaming Interaction Understandingcited by 2
  37. 2026Vinci2: Providing Proactive Assistance in Continuous Egocentric Videoscited by 2
  38. 2026EXPLORE-Bench: Egocentric Scene Prediction with Long-Horizon Reasoningcited by 1

50 most cited of 59 All 59 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025EgoToM: Benchmarking Theory of Mind Reasoning from Egocentric Videos
  2. 2025EgoExoBench: A Benchmark for First- and Third-person View Video Understanding in MLLMs
  3. 2024VidEgoThink: Assessing Egocentric Video Understanding Capabilities for Embodied AI
  4. 2025ECBench: Can Multi-modal Foundation Models Understand the Egocentric World? A Holistic Embodied Cognition Benchmark
  5. 2025How Well Can Vison-Language Models Understand Humans' Intention? An Open-ended Theory of Mind Question Evaluation Benchmark
  6. 2026Egocentric Bias in Vision-Language Models
  7. 2025Analyzing the Sensitivity of Vision Language Models in Visual Question Answering
  8. 2026EgoExo-Next:Benchmarking Vision-Language Models on Visual-Option Next-State and Cross-View Reasoning