# Towards Comprehensive Scene Understanding: Integrating First and Third-Person Views for LVLMs

arXiv: https://arxiv.org/abs/2505.21955 · HTML: https://mlsanity.com/paper/2505.21955/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 41 of its 57 references that are in the map.
- First posted on arXiv 28 May 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)

## Builds on (most cited first, 30 of 41)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models](https://mlsanity.com/paper/2504.10479/) (2025)
- [DeepSeek-VL: Towards Real-World Vision-Language Understanding](https://mlsanity.com/paper/2403.05525/) (2024)
- [EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding](https://mlsanity.com/paper/2308.09126/) (2023)
- [LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models](https://mlsanity.com/paper/2407.07895/) (2024)
- [Faith and Fate: Limits of Transformers on Compositionality](https://mlsanity.com/paper/2305.18654/) (2023)
- [Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives](https://mlsanity.com/paper/2311.18259/) (2023)
- [Embodied Question Answering](https://mlsanity.com/paper/1711.11543/) (2017)
- [Can language models learn from explanations in context?](https://mlsanity.com/paper/2204.02329/) (2022)
- [Egocentric Video-Language Pretraining](https://mlsanity.com/paper/2206.01670/) (2022)
- [MANTIS: Interleaved Multi-Image Instruction Tuning](https://mlsanity.com/paper/2405.01483/) (2024)
- [Compositional Chain-of-Thought Prompting for Large Multimodal Models](https://mlsanity.com/paper/2311.17076/) (2023)
- [EgoVLPv2: Egocentric Video-Language Pre-training with Fusion in the Backbone](https://mlsanity.com/paper/2307.05463/) (2023)
- [Charades-Ego: A Large-Scale Dataset of Paired Third and First Person Videos](https://mlsanity.com/paper/1804.09626/) (2018)
- [DDCoT: Duty-Distinct Chain-of-Thought Prompting for Multimodal Reasoning in Language Models](https://mlsanity.com/paper/2310.16436/) (2023)
- [Retrieval-Augmented Multimodal Language Modeling](https://mlsanity.com/paper/2211.12561/) (2022)
- [Panoptic Scene Graph Generation](https://mlsanity.com/paper/2207.11247/) (2022)
- [Ego-Exo: Transferring Visual Representations from Third-person to First-person Videos](https://mlsanity.com/paper/2104.07905/) (2021)
- [Pano-AVQA: Grounded Audio-Visual Question Answering on 360^circ Videos](https://mlsanity.com/paper/2110.05122/) (2021)
- [EgoThink: Evaluating First-Person Perspective Thinking Capability of Vision-Language Models](https://mlsanity.com/paper/2311.15596/) (2023)
- [Needle In A Multimodal Haystack](https://mlsanity.com/paper/2406.07230/) (2024)
- [CoCoT: Contrastive Chain-of-Thought Prompting for Large Multimodal Models with Multiple Image Inputs](https://mlsanity.com/paper/2401.02582/) (2024)
- [LEMMA: A Multi-view Dataset for Learning Multi-agent Multi-task Activities](https://mlsanity.com/paper/2007.15781/) (2020)
- [Learning Fine-grained View-Invariant Representations from Unpaired Ego-Exo Videos via Temporal Alignment](https://mlsanity.com/paper/2306.05526/) (2023)
- [Retrieval-Augmented Egocentric Video Captioning](https://mlsanity.com/paper/2401.00789/) (2024)

## Built on by (most cited first, 6 of 6)

- [Exo2EgoSyn: Unlocking Foundation Video Generation Models for Exocentric-to-Egocentric Video Synthesis](https://mlsanity.com/paper/2511.20186/) (2025)
- [Building Egocentric Procedural AI Assistant: Methods, Benchmarks, and Challenges](https://mlsanity.com/paper/2511.13261/) (2025)
- [From Synchrony to Sequence: Exo-to-Ego Generation via Interpolation](https://mlsanity.com/paper/2604.13793/) (2026)
- [Revealing Multi-View Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2603.23934/) (2026)
- [MVVBench: Benchmarking 4D Reasoning in Vision-Language Models](https://mlsanity.com/paper/2609.30952/) (2026)
- [Ego-Forge: Text and Geometric-Attention Free Exo-to-Egocentric Video Generation](https://mlsanity.com/paper/2609.35368/) (2026)

## Abstract (arXiv)

Large vision-language models (LVLMs) are increasingly deployed in interactive applications such as virtual and augmented reality, where a first-person (egocentric) view captured by head-mounted cameras serves as key input. While this view offers fine-grained cues about user attention and hand-object interactions, its narrow field of view and lack of global context often lead to failures on spatially or contextually demanding queries. To address this, we introduce a framework that augments egocentric inputs with third-person (exocentric) views, providing complementary information such as global scene layout and object visibility to LVLMs. We present E3VQA, the first benchmark for multi-view question answering with 4K high-quality question-answer pairs grounded in synchronized ego-exo image pairs. Additionally, we propose M3CoT, a training-free prompting technique that constructs a unified scene representation by integrating scene graphs from three complementary perspectives. M3CoT enables LVLMs to reason more effectively across views, yielding consistent performance gains (4.84% for GPT-4o and 5.94% for Gemini 2.0 Flash) over a recent CoT baseline. Our extensive evaluation reveals key strengths and limitations of LVLMs in multi-view reasoning and highlights the value of leveraging both egocentric and exocentric inputs. The dataset and source code are available at https://github.com/Leeinsu1/Towards-Comprehensive-Scene-Understanding.