mlsanityData to 14 Oct 2026

Ego-R1: Chain-of-Tool-Thought for Ultra-Long Egocentric Video Reasoning

arXiv 2506.13654 ↗First posted 16 Jun 2025cs.CV cs.AIcited by 68

On the map: LLM reasoning → Spatial and embodied reasoning

Abstract (arXiv)

We introduce Ego-R1, a novel framework for reasoning over ultra-long (i.e., in days and weeks) egocentric videos, which leverages a structured Chain-of-Tool-Thought (CoTT) process, orchestrated by an Ego-R1 Agent trained via reinforcement learning (RL). Inspired by human problem-solving strategies, CoTT decomposes complex reasoning into modular steps, with the RL agent invoking specific tools, one per step, to iteratively and collaboratively answer sub-questions tackling such tasks as temporal retrieval and multi-modal understanding. We design a two-stage training paradigm involving supervised finetuning (SFT) of a pretrained language model using CoTT data and RL to enable our agent to dynamically propose step-by-step tools for long-range reasoning. To facilitate training, we construct a dataset called Ego-R1 Data, which consists of Ego-CoTT-25K for SFT and Ego-QA-4.4K for RL. Furthermore, our Ego-R1 agent is evaluated on a newly curated week-long video QA benchmark, Ego-R1 Bench, which contains human-verified QA pairs from hybrid sources. Extensive results demonstrate that the dynamic, tool-augmented chain-of-thought reasoning by our Ego-R1 Agent can effectively tackle the unique challenges of understanding ultra-long egocentric videos, significantly extending the time coverage from few hours to a week.

Key facts. Cited by 68 papers in the map (data through 14 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 61 of its 73 references that are in the map. First posted on arXiv 16 Jun 2025.

Builds on

61 of 73 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025EgoToM: Benchmarking Theory of Mind Reasoning from Egocentric Videos28 Mar

Other references · 60

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  5. 2025Qwen2.5-VL Technical Reportcited by 5,587
  6. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  7. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  8. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  9. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
  10. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  11. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,444
  12. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,413
38 more
  1. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  2. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 795
  3. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 680
  4. 2025Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Modelscited by 665
  5. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 542
  6. 2024Long Context Transfer from Language to Visioncited by 515
  7. 2023Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectivescited by 490
  8. 2025Video-R1: Reinforcing Video Reasoning in MLLMscited by 484
  9. 2024LLaVA-CoT: Let Vision Language Models Reason Step-by-Stepcited by 454
  10. 2025ReTool: Reinforcement Learning for Strategic Tool Use in LLMscited by 392
  11. 2025R1-Onevision: Advancing Generalized Multimodal Reasoning through Cross-Modal Formalizationcited by 370
  12. 2025ToolRL: Reward is All Tool Learning Needscited by 362
  13. 2024VideoAgent: Long-form Video Understanding with Large Language Model as Agentcited by 332
  14. 2024LongVILA: Scaling Long-Context Visual Language Models for Long Videoscited by 309
  15. 2025R1-VL: Learning to Reason with Multimodal Large Language Models via Step-wise Group Relative Policy Optimizationcited by 301
  16. 2025DeepResearcher: Scaling Deep Research via Reinforcement Learning in Real-world Environmentscited by 286
  17. 2023ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solvingcited by 282
  18. 2024VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videoscited by 251
  19. 2024VideoAgent: A Memory-augmented Multimodal Agent for Video Understandingcited by 229
  20. 2024Video-XL: Extra-Long Vision Language Model for Hour-Scale Video Understandingcited by 213
  21. 2025InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modelingcited by 211
  22. 2022TALM: Tool Augmented Language Modelscited by 197
  23. 2025ToRL: Scaling Tool-Integrated RLcited by 169
  24. 2023Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with Code-based Self-Verificationcited by 166
  25. 2024Oryx MLLM: On-Demand Spatial-Temporal Understanding at Arbitrary Resolutioncited by 159
  26. 2023MathCoder: Seamless Code Integration in LLMs for Enhanced Mathematical Reasoningcited by 156
  27. 2025LlamaV-o1: Rethinking Step-by-step Visual Reasoning in LLMscited by 147
  28. 2024Video-RAG: Visually-aligned Retrieval-Augmented Long Video Comprehensioncited by 138
  29. 2024Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Modelscited by 116
  30. 2024HourVideo: 1-Hour Video-Language Understandingcited by 113
  31. 2025HD-EPIC: A Highly-Detailed Egocentric Video Datasetcited by 104
  32. 2025Ola: Pushing the Frontiers of Omni-Modal Language Modelcited by 83
  33. 2025T*: Re-thinking Temporal Search for Long-Form Video Understandingcited by 82
  34. 2024MARIO: MAth Reasoning with code Interpreter Output -- A Reproducible Pipelinecited by 81
  35. 2023Octopus: Embodied Vision-Language Programmer from Environmental Feedbackcited by 73
  36. 2024Chain-of-Spot: Interactive Reasoning Improves Large Vision-Language Modelscited by 66
  37. 2023Grounded Question-Answering in Long Egocentric Videoscited by 64
  38. 2025Grounded Chain-of-Thought for Multimodal Large Language Modelscited by 59

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 17; 2026: 51.

68 papers.

Sort:

By area:

  1. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  2. 2025A Survey of Context Engineering for Large Language Modelscited by 126
  3. 2025LongVT: Incentivizing "Thinking with Long Videos" via Native Tool Callingcited by 61
  4. 2026Video-MME-v2: Towards the Next Stage in Benchmarks for Comprehensive Video Understandingcited by 55
  5. 2025WorldMM: Dynamic Multimodal Memory Agent for Long Video Reasoningcited by 53
  6. 2025Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecyclecited by 35
  7. 2025Video-RTS: Rethinking Reinforcement Learning and Test-Time Scaling for Efficient and Enhanced Video Reasoningcited by 33
  8. 2025StreamAgent: Towards Anticipatory Agents for Streaming Video Understandingcited by 33
  9. 2026LVOmniBench: Pioneering Long Audio-Video Understanding Evaluation for Omnimodal LLMscited by 24
  10. 2026Agentic Very Long Video Understandingcited by 20
  11. 2025VideoMem: Enhancing Ultra-Long Video Understanding via Adaptive Memory Managementcited by 19
  12. 2026VideoSeek: Long-Horizon Video Agent with Tool-Guided Seekingcited by 17
38 more
  1. 2026LensWalk: Agentic Video Understanding by Planning How You See in Videoscited by 17
  2. 2026Thinking in Streaming Videocited by 16
  3. 2025Reinforcement Learning Foundations for Deep Research Systems: A Surveycited by 14
  4. 2025Active Perception Agent for Omnimodal Audio-Video Understandingcited by 13
  5. 2026EgoMemReason: A Memory-Driven Reasoning Benchmark for Long-Horizon Egocentric Video Understandingcited by 13
  6. 2025ERA: Transforming VLMs into Embodied Agents via Embodied Prior Learning and Online Reinforcement Learningcited by 10
  7. 2026Video-KTR: Reinforcing Video Reasoning via Key Token Attributioncited by 10
  8. 2025Reinforcement Learning for Large Model: A Surveycited by 9
  9. 2025Video-LMM Post-Training: A Deep Dive into Video Reasoning with Large Multimodal Modelscited by 8
  10. 2026Demo-ICL: In-Context Learning for Procedural Video Knowledge Acquisitioncited by 8
  11. 2026LongVideo-R1: Smart Navigation for Low-cost Long Video Understandingcited by 7
  12. 2026EVA: Efficient Reinforcement Learning for End-to-End Video Agentcited by 6
  13. 2026VideoSEAL: Mitigating Evidence Misalignment in Agentic Long Video Understanding by Decoupling Answer Authoritycited by 6
  14. 2025TuneAgent: Agentic Operating System Kernel Tuning with Reinforcement Learningcited by 5
  15. 2026MA-EgoQA: Question Answering over Egocentric Videos from Multiple Embodied Agentscited by 5
  16. 2026Ego2Web: A Web Agent Benchmark Grounded in Egocentric Videoscited by 5
  17. 2025Rethinking the Design of Reinforcement Learning-Based Deep Research Agentscited by 4
  18. 2025EgoVITA: Learning to Plan and Verify for Egocentric Video Reasoningcited by 4
  19. 2026ArrowGEV: Grounding Events in Video via Learning the Arrow of Timecited by 4
  20. 2026LongVidSearch: An Agentic Benchmark for Multi-hop Evidence Retrieval Planning in Long Videoscited by 4
  21. 2026VisionCoach: Reinforcing Grounded Video Reasoning via Visual-Perception Promptingcited by 4
  22. 2026Bridging Modalities, Spanning Time: Structured Memory for Ultra-Long Agentic Video Reasoningcited by 4
  23. 2026Watch, Remember, Reason: Human-View Video Understanding with MLLMscited by 4
  24. 2026LightMem-Ego: Your AI Memory for Everyday Lifecited by 4
  25. 2026EgoGraph: Temporal Knowledge Graph for Egocentric Video Understandingcited by 3
  26. 2026Egocentric Co-Pilot: Web-Native Smart-Glasses Agents for Assistive Egocentric AIcited by 3
  27. 2026Evaluating Memory Capability in Continuous Lifelog Scenariocited by 3
  28. 2026VideoSeeker: Incentivizing Instance-level Video Understanding via Native Agentic Tool Invocationcited by 3
  29. 2026Insight-V++: Towards Advanced Long-Chain Visual Reasoning with Multimodal Large Language Modelscited by 2
  30. 2026Video-DeepResearch: Towards the Next-Generation Multimodal Deepresearch Agentcited by 2
  31. 2026Routing Before Looking: Query-Adaptive Evidence Acquisition for Long-form Video Understandingcited by 2
  32. 2025TAMA: Tool-Augmented Multimodal Agent for Procedural Activity Understandingcited by 1
  33. 2026Shape of Thought: Progressive Object Assembly via Visual Chain-of-Thoughtcited by 1
  34. 2026EgoReasoner: Learning Egocentric 4D Reasoning via Task-Adaptive Structured Thinkingcited by 1
  35. 2026Do You See What I Am Pointing At? Gesture-Based Egocentric Video Question Answeringcited by 1
  36. 2026HippoCamp: Benchmarking Contextual Agents on Personal Computerscited by 1
  37. 2026Chain of Modality: From Static Fusion to Dynamic Orchestration in Omni-MLLMscited by 1
  38. 2026ReTool-Video: Recursive Tool-Using Video Agents with Meta-Augmented Tool Groundingcited by 1

50 most cited of 68 All 68 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025EgoThinker: Unveiling Egocentric Reasoning with Spatio-Temporal CoT
  2. 2025EgoVLM: Policy Optimization for Egocentric Video Understanding
  3. 2026EgoCoT-Bench: Benchmarking Grounded and Verifiable Operation-Centric Chain of Thought Reasoning for MLLMs
  4. 2025Omnia de EgoTempo: Benchmarking Temporal Understanding of Multi-Modal LLMs in Egocentric Videos
  5. 2025ST-Think: How Multimodal Large Language Models Reason About 4D Worlds from Ego-Centric Videos
  6. 2026EgoSAT: A Comprehensive Benchmark of Egocentric Streaming Interaction Understanding
  7. 2025Thinking With Videos: Multimodal Tool-Augmented Reinforcement Learning for Long Video Reasoning
  8. 2026OmniEgo-R2: A Routed Reasoning Framework for the 1st Cross-Domain EgoCross Challenge at CVPR 2026