mlsanityData to 14 Oct 2026

MARS: Technical Report for the CASTLE Challenge at EgoVis 2026

arXiv 2605.18176 ↗First posted 18 May 2026cs.CV cs.AIcited by 1

On the map: LLM agents and tool use → Multimodal and embodied agents

Abstract (arXiv)

This report presents MARS, short for Multimodal Agentic Reasoning with Source selection, our system for the CASTLE Challenge at EgoVis 2026. Participants must answer 185 closed-form questions over the CASTLE 2024 dataset. In contrast to prior single-video egocentric benchmarks, CASTLE requires reasoning over four days of activity, 15 synchronized perspectives, official transcripts, and multiple auxiliary modalities, including personal photos, auxiliary videos, gaze, thermal imagery, and heartrate measurements. MARS therefore treats the task as an agentic evidence-selection problem over multimodal sources rather than a purely text-only pipeline. MARS first follows the official CASTLE directory organization to build evidence memories from two primary sources, videos and transcripts, and four auxiliary sources, gaze, heartrate, photos, and thermal imagery. Long videos are converted into captions and DeepSeek-based summaries only because CASTLE videos are too long to fit directly into the model context for every question; this step compresses temporal evidence while keeping photos and other auxiliary media available as source-specific evidence. At inference time, a GPT-5.4 decision agent repeatedly chooses whether to continue reasoning, request a specific missing modality, produce an answer, or fall back to a random option when the evidence remains insufficient. The resulting system achieved second place on the final CASTLE Challenge leaderboard. Our codes are available at https://github.com/Hyu-Zhang/MARS.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use. Builds on 16 of its 22 references that are in the map. First posted on arXiv 18 May 2026.

Builds on

16 of 22 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024VideoAgent: Long-form Video Understanding with Large Language Model as Agent15 Mar

Other references · 15

  1. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 542
  2. 2023Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectivescited by 490
  3. 2023TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understandingcited by 406
  4. 2023Grounded Question-Answering in Long Egocentric Videoscited by 64
  5. 2024Retrieval-Augmented Egocentric Video Captioningcited by 42
  6. 2025Spatial Understanding from Videos: Structured Prompts Meet Simulation Datacited by 28
  7. 2025Exo2Ego: Exocentric Knowledge Guided MLLM for Egocentric Video Understandingcited by 25
  8. 2025The CASTLE 2024 Dataset: Advancing the Art of Multimodal Understandingcited by 16
  9. 2025Object-Shot Enhanced Grounding Network for Egocentric Videocited by 11
  10. 2024ObjectNLQ @ Ego4D Episodic Memory Challenge 2024cited by 10
  11. 2025Intention-Guided Cognitive Reasoning for Egocentric Long-Term Action Anticipationcited by 10
  12. 2024HCQA @ Ego4D EgoSchema Challenge 2024cited by 9
3 more
  1. 2025Technical Report for Ego4D Long-Term Action Anticipation Challenge 2025cited by 9
  2. 2025OSGNet @ Ego4D Episodic Memory Challenge 2025cited by 6
  3. 2023Uncovering Hidden Connections: Iterative Search and Reasoning for Video-grounded Dialogcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

Similar, not cited

Close in wording, no citation either way

  1. 2026CASTLE2026 Team WDL Technical Report
  2. 2026CuriosAI Submission to the CASTLE Challenge at EgoVis 2026
  3. 20263rd Place at CVPR 2026 CASTLE Challenge: Agentic Multi-View Long-Context Video Understanding via Hierarchical Knowledge Graph Retrieval
  4. 2025SciVideoBench: Benchmarking Scientific Video Reasoning in Large Multimodal Models
  5. 2025MARS2 2025 Challenge on Multimodal Reasoning: Datasets, Methods, Results, Discussion, and Outlook
  6. 2026Mr.LHDR: A Benchmark for Multimodal Real-World Long-Horizon Deep Research Agents
  7. 2026MemEye: A Visual-Centric Evaluation Framework for Multimodal Agent Memory
  8. 2026SMMBench: A Benchmark for Source-Distributed Multimodal Agent Memory