EgoThinker: Unveiling Egocentric Reasoning with Spatio-Temporal CoT
On the map: LLM reasoning → Spatial and embodied reasoning
Abstract (arXiv)
Egocentric video reasoning centers on an unobservable agent behind the camera who dynamically shapes the environment, requiring inference of hidden intentions and recognition of fine-grained interactions. This core challenge limits current multimodal large language models MLLMs, which excel at visible event reasoning but lack embodied, first-person understanding. To bridge this gap, we introduce EgoThinker, a novel framework that endows MLLMs with robust egocentric reasoning capabilities through spatio-temporal chain-of-thought supervision and a two-stage learning curriculum. First, we introduce EgoRe-5M, a large-scale egocentric QA dataset constructed from 13M diverse egocentric video clips. This dataset features multi-minute segments annotated with detailed CoT rationales and dense hand-object grounding. Second, we employ SFT on EgoRe-5M to instill reasoning skills, followed by reinforcement fine-tuning RFT to further enhance spatio-temporal localization. Experimental results show that EgoThinker outperforms existing methods across multiple egocentric benchmarks, while achieving substantial improvements in fine-grained spatio-temporal localization tasks. Full code and data are released at https://github.com/InternRobotics/EgoThinker.
Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 63 of its 112 references that are in the map. First posted on arXiv 27 Oct 2025.
Builds on
63 of 112 references.
References · 63
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
- 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
- 2025Qwen2.5-VL Technical Reportcited by 5,587
- 2024GPT-4o System Cardcited by 4,588
- 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
- 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
- 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,887
- 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
- 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
- 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
- 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
- 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,413
38 more
- 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,402
- 2024OpenAI o1 System Cardcited by 1,244
- 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,079
- 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
- 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 1,053
- 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
- 2023VideoChat: Chat-Centric Video Understandingcited by 857
- 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 723
- 2022Automatic Chain of Thought Prompting in Large Language Modelscited by 688
- 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 583
- 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 518
- 2024Long Context Transfer from Language to Visioncited by 515
- 2025Visual-RFT: Visual Reinforcement Fine-Tuningcited by 513
- 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 497
- 2019CLEVRER: CoLlision Events for Video REpresentation and Reasoningcited by 484
- 2018TVQA: Localized, Compositional Video Question Answeringcited by 467
- 2025R1-Onevision: Advancing Generalized Multimodal Reasoning through Cross-Modal Formalizationcited by 370
- 2018A Short Note about Kinetics-600cited by 333
- 2024InternVideo2: Scaling Foundation Models for Multimodal Video Understandingcited by 317
- 2025R1-VL: Learning to Reason with Multimodal Large Language Models via Step-wise Group Relative Policy Optimizationcited by 301
- 2025VideoChat-R1: Enhancing Spatio-Temporal Perception via Reinforcement Fine-Tuningcited by 255
- 2025LMM-R1: Empowering 3B LMMs with Strong Reasoning Abilities Through Two-Stage Rule-Based RLcited by 248
- 2025Seg-Zero: Reasoning-Chain Guided Segmentation via Cognitive Reinforcementcited by 239
- 2025InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modelingcited by 211
- 2025R1-Zero's "Aha Moment" in Visual Reasoning on a 2B Non-SFT Modelcited by 165
- 2024M3CoT: A Novel Benchmark for Multi-Domain Multi-step Multi-modal Chain-of-Thoughtcited by 121
- 2025Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Modelscited by 100
- 2025EgoLife: Towards Egocentric Life Assistantcited by 91
- 2024VideoHallucer: Evaluating Intrinsic and Extrinsic Hallucinations in Large Video-Language Modelscited by 90
- 2024CG-Bench: Clue-grounded Question Answering Benchmark for Long Video Understandingcited by 85
- 2025Gemini Robotics: Bringing AI into the Physical Worldcited by 83
- 2025Boosting the Generalization and Reasoning of Vision Language Models with Curriculum Reinforcement Learningcited by 71
- 2025Eagle 2: Building Post-Training Data Strategies from Scratch for Frontier Vision-Language Modelscited by 70
- 2025Vision-R1: Evolving Human-Free Alignment in Large Vision-Language Models via Vision-Guided Reinforcement Learningcited by 69
- 2025Eagle 2.5: Boosting Long-Context Post-Training for Frontier Vision-Language Modelscited by 64
- 2023LifelongMemory: Leveraging LLMs for Answering Queries in Long-form Egocentric Videoscited by 51
- 2025STORM: Token-Efficient Long Video Understanding for Multimodal LLMscited by 49
- 2022InternVideo-Ego4D: A Pack of Champion Solutions to Ego4D Challengescited by 48
50 most cited of 63 All 63 (JSON)
Built on by
Citing papers per year: 2025: 3; 2026: 13.
16 papers.
Sort:
By area:
- 2026Towards Multimodal Lifelong Understanding: A Dataset and Agentic Baselinecited by 8
- 2026SUREON: A Benchmark and Vision-Language-Model for Surgical Reasoningcited by 7
- 2026Embodied-R1.5: Evolving Physical Intelligence via Embodied Foundation Modelscited by 6
- 2026Vinci2: Providing Proactive Assistance in Continuous Egocentric Videoscited by 5
- 2025EgoVITA: Learning to Plan and Verify for Egocentric Video Reasoningcited by 4
- 2026EgoTL: Egocentric Think-Aloud Chains for Long-Horizon Taskscited by 3
- 2026Towards Predictive, Aligned, and Scalable Robot Learningcited by 3
- 2026EgoMonth: A Month-Level Egocentric Video Benchmark for Long-Term Spatiotemporal Memorycited by 3
- 2025Beyond Multiple Choice: Verifiable OpenQA for Robust Vision-Language RFTcited by 1
- 2025SFHand: Learning Embodied Manipulation by Streaming Egocentric 3D Hand Forecastingcited by 1
- 2026EgoReasoner: Learning Egocentric 4D Reasoning via Task-Adaptive Structured Thinkingcited by 1
- 2026EXPLORE-Bench: Egocentric Scene Prediction with Long-Horizon Reasoningcited by 1
4 more
- 2026EARL: Towards a Unified Analysis-Guided Reinforcement Learning Framework for Egocentric Interaction Reasoning and Pixel Groundingcited by 1
- 2026Learning to Evolve Scenes: Reasoning about Human Activities with Scene Graphscited by 1
- 2026Post-Training VLMs for Video Mistake Detectioncited by 1
- 2026Rethinking VLM Representation for VLA Initializationcited by 0
- 2026Post-Training VLMs for Video Mistake Detection28 Aug 2026
- 2026EgoMonth: A Month-Level Egocentric Video Benchmark for Long-Term Spatiotemporal Memory13 Aug 2026
- 2026Vinci2: Providing Proactive Assistance in Continuous Egocentric Videos13 Jul 2026
- 2026Towards Predictive, Aligned, and Scalable Robot Learning13 Jul 2026
- 2026Learning to Evolve Scenes: Reasoning about Human Activities with Scene Graphs2 Jul 2026
- 2026Embodied-R1.5: Evolving Physical Intelligence via Embodied Foundation Models9 Jun 2026
- 2026Rethinking VLM Representation for VLA Initialization25 May 2026
- 2026EARL: Towards a Unified Analysis-Guided Reinforcement Learning Framework for Egocentric Interaction Reasoning and Pixel Grounding14 May 2026
- 2026EgoTL: Egocentric Think-Aloud Chains for Long-Horizon Tasks10 Apr 2026
- 2026EXPLORE-Bench: Egocentric Scene Prediction with Long-Horizon Reasoning10 Mar 2026
- 2026SUREON: A Benchmark and Vision-Language-Model for Surgical Reasoning6 Mar 2026
- 2026EgoReasoner: Learning Egocentric 4D Reasoning via Task-Adaptive Structured Thinking6 Mar 2026
4 more
- 2026Towards Multimodal Lifelong Understanding: A Dataset and Agentic Baseline5 Mar 2026
- 2025EgoVITA: Learning to Plan and Verify for Egocentric Video Reasoning23 Nov 2025
- 2025SFHand: Learning Embodied Manipulation by Streaming Egocentric 3D Hand Forecasting22 Nov 2025
- 2025Beyond Multiple Choice: Verifiable OpenQA for Robust Vision-Language RFT21 Nov 2025
- 2026SUREON: A Benchmark and Vision-Language-Model for Surgical Reasoningcited by 7
- 2026Embodied-R1.5: Evolving Physical Intelligence via Embodied Foundation Modelscited by 6
- 2025EgoVITA: Learning to Plan and Verify for Egocentric Video Reasoningcited by 4
- 2026EXPLORE-Bench: Egocentric Scene Prediction with Long-Horizon Reasoningcited by 1
- 2026EARL: Towards a Unified Analysis-Guided Reinforcement Learning Framework for Egocentric Interaction Reasoning and Pixel Groundingcited by 1
- 2026Learning to Evolve Scenes: Reasoning about Human Activities with Scene Graphscited by 1
- 2026Rethinking VLM Representation for VLA Initializationcited by 0
- 2026Learning to Evolve Scenes: Reasoning about Human Activities with Scene Graphs2 Jul 2026
- 2026Embodied-R1.5: Evolving Physical Intelligence via Embodied Foundation Models9 Jun 2026
- 2026Rethinking VLM Representation for VLA Initialization25 May 2026
- 2026EARL: Towards a Unified Analysis-Guided Reinforcement Learning Framework for Egocentric Interaction Reasoning and Pixel Grounding14 May 2026
- 2026EXPLORE-Bench: Egocentric Scene Prediction with Long-Horizon Reasoning10 Mar 2026
- 2026SUREON: A Benchmark and Vision-Language-Model for Surgical Reasoning6 Mar 2026
- 2025EgoVITA: Learning to Plan and Verify for Egocentric Video Reasoning23 Nov 2025
- 2026Towards Predictive, Aligned, and Scalable Robot Learning13 Jul 2026
- 2026EgoTL: Egocentric Think-Aloud Chains for Long-Horizon Tasks10 Apr 2026
- 2026Vinci2: Providing Proactive Assistance in Continuous Egocentric Videoscited by 5
- 2025SFHand: Learning Embodied Manipulation by Streaming Egocentric 3D Hand Forecastingcited by 1
- 2026EgoReasoner: Learning Egocentric 4D Reasoning via Task-Adaptive Structured Thinkingcited by 1
- 2026Post-Training VLMs for Video Mistake Detectioncited by 1
- 2026Post-Training VLMs for Video Mistake Detection28 Aug 2026
- 2026Vinci2: Providing Proactive Assistance in Continuous Egocentric Videos13 Jul 2026
- 2026EgoReasoner: Learning Egocentric 4D Reasoning via Task-Adaptive Structured Thinking6 Mar 2026
- 2025SFHand: Learning Embodied Manipulation by Streaming Egocentric 3D Hand Forecasting22 Nov 2025
Similar, not cited
Close in wording, no citation either way
- 2026Minerva-Ego: Spatiotemporal Hints for Egocentric Video Understanding
- 2026EgoCoT-Bench: Benchmarking Grounded and Verifiable Operation-Centric Chain of Thought Reasoning for MLLMs
- 2025EgoVLM: Policy Optimization for Egocentric Video Understanding
- 2025EgoToM: Benchmarking Theory of Mind Reasoning from Egocentric Videos
- 2025Omnia de EgoTempo: Benchmarking Temporal Understanding of Multi-Modal LLMs in Egocentric Videos
- 2026From Perception to Planning: Evolving Ego-Centric Task-Oriented Spatiotemporal Reasoning via Curriculum Learning
- 2026Spatial-Conditioned Reasoning in Long-Egocentric Videos
- 2026EgoSAT: A Comprehensive Benchmark of Egocentric Streaming Interaction Understanding