# Exo2Ego: Exocentric Knowledge Guided MLLM for Egocentric Video Understanding

arXiv: https://arxiv.org/abs/2503.09143 · HTML: https://mlsanity.com/paper/2503.09143/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 83 of its 108 references that are in the map.
- First posted on arXiv 12 Mar 2025.

## Builds on (most cited first, 30 of 83)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](https://mlsanity.com/paper/2304.10592/) (2023)
- [PaLM-E: An Embodied Multimodal Language Model](https://mlsanity.com/paper/2303.03378/) (2023)
- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding](https://mlsanity.com/paper/2306.02858/) (2023)
- [Video-LLaVA: Learning United Visual Representation by Alignment Before Projection](https://mlsanity.com/paper/2311.10122/) (2023)
- [Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts](https://mlsanity.com/paper/2102.08981/) (2021)
- [Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval](https://mlsanity.com/paper/2104.00650/) (2021)
- [MVBench: A Comprehensive Multi-modal Video Understanding Benchmark](https://mlsanity.com/paper/2311.17005/) (2023)
- [The "something something" video database for learning and evaluating visual common sense](https://mlsanity.com/paper/1706.04261/) (2017)
- [Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models](https://mlsanity.com/paper/2306.05424/) (2023)
- [mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality](https://mlsanity.com/paper/2304.14178/) (2023)
- [VideoChat: Chat-Centric Video Understanding](https://mlsanity.com/paper/2305.06355/) (2023)
- [VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs](https://mlsanity.com/paper/2406.07476/) (2024)
- [EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding](https://mlsanity.com/paper/2308.09126/) (2023)

## Built on by (most cited first, 25 of 25)

- [Spatial Understanding from Videos: Structured Prompts Meet Simulation Data](https://mlsanity.com/paper/2506.03642/) (2025)
- [EgoNight: Towards Egocentric Vision Understanding at Night with a Challenging Benchmark](https://mlsanity.com/paper/2510.06218/) (2025)
- [EgoThinker: Unveiling Egocentric Reasoning with Spatio-Temporal CoT](https://mlsanity.com/paper/2510.23569/) (2025)
- [EgoForge: Goal-Directed Egocentric World Simulator](https://mlsanity.com/paper/2603.20169/) (2026)
- [RynnEC: Bringing MLLMs into Embodied World](https://mlsanity.com/paper/2508.14160/) (2025)
- [VideoChat3: Fully Open Video MLLM for Efficient and Generalist Video Understanding](https://mlsanity.com/paper/2607.14935/) (2026)
- [Technical Report for Ego4D Long-Term Action Anticipation Challenge 2025](https://mlsanity.com/paper/2506.02550/) (2025)
- [HCQA-1.5 @ Ego4D EgoSchema Challenge 2025](https://mlsanity.com/paper/2505.20644/) (2025)
- [Towards Comprehensive Scene Understanding: Integrating First and Third-Person Views for LVLMs](https://mlsanity.com/paper/2505.21955/) (2025)
- [OSGNet @ Ego4D Episodic Memory Challenge 2025](https://mlsanity.com/paper/2506.03710/) (2025)
- [EgoExo-Con: Exploring View-Invariant Video Temporal Understanding](https://mlsanity.com/paper/2510.26113/) (2025)
- [Vinci2: Providing Proactive Assistance in Continuous Egocentric Videos](https://mlsanity.com/paper/2607.11523/) (2026)
- [EgoVITA: Learning to Plan and Verify for Egocentric Video Reasoning](https://mlsanity.com/paper/2511.18242/) (2025)
- [OutSafe-Bench: A Benchmark for Multimodal Offensive Content Detection in Large Language Models](https://mlsanity.com/paper/2511.10287/) (2025)
- [Egocentric Co-Pilot: Web-Native Smart-Glasses Agents for Assistive Egocentric AI](https://mlsanity.com/paper/2603.01104/) (2026)
- [Test-time Ego-Exo-centric Adaptation for Action Anticipation via Multi-Label Prototype Growing and Dual-Clue Consistency](https://mlsanity.com/paper/2603.09798/) (2026)
- [VISTA: Technical Report for the Ego4D Short-Term Object Interaction Anticipation at EgoVis 2026](https://mlsanity.com/paper/2605.20901/) (2026)
- [JFAA: Technical Report for the EPIC-KITCHENS-100 Action Anticipation Challenge at EgoVis 2026](https://mlsanity.com/paper/2605.20904/) (2026)
- [SneakPeek: Future-Guided Instructional Streaming Video Generation](https://mlsanity.com/paper/2512.13019/) (2025)
- [EgoReasoner: Learning Egocentric 4D Reasoning via Task-Adaptive Structured Thinking](https://mlsanity.com/paper/2603.06561/) (2026)
- [MARS: Technical Report for the CASTLE Challenge at EgoVis 2026](https://mlsanity.com/paper/2605.18176/) (2026)
- [SpaceEra++: A Unified Framework Towards 3D Spatial Reasoning in Video](https://mlsanity.com/paper/2607.01784/) (2026)
- [CrossView: Can Vision-Language Models Reason Across Cameras?](https://mlsanity.com/paper/2608.15539/) (2026)
- [OSGNet with MLLM Reranking @ Ego4D Episodic Memory Challenge 2026](https://mlsanity.com/paper/2605.20818/) (2026)
- [Exo2EgoPose: Leveraging Exocentric Demonstrations for Vision-Language guided Egocentric 3D Hand Pose Forecasting](https://mlsanity.com/paper/2607.15890/) (2026)

## Abstract (arXiv)

AI personal assistants, deployed through robots or wearables, require embodied understanding to collaborate effectively with humans. However, current Multimodal Large Language Models (MLLMs) primarily focus on third-person (exocentric) vision, overlooking the unique challenges of first-person (egocentric) videos. Additionally, high acquisition costs limit data size, impairing MLLM performance. To address these challenges, we propose learning the mapping between exocentric and egocentric domains, leveraging the extensive exocentric knowledge within existing MLLMs to enhance egocentric video understanding. To this end, we introduce Ego-ExoClip, a pre-training dataset comprising 1.1M synchronized ego-exo clip-text pairs derived from Ego-Exo4D, together with the instruction-tuning dataset EgoIT, which is collected from multiple sources to enhance the model's instruction-following capabilities. Building upon the datasets, we propose a migration strategy and further design a progressive mapping learning pipeline with three stages: Demonstrator Self-Preparation, Demonstrator-Learner Guidance, and Learner Self-Practice. Extensive experiments across diverse egocentric tasks reveal that existing MLLMs perform inadequately in egocentric video understanding, while our model significantly outperforms these leading models.