mlsanityData to 7 Oct 2026

TraveLER: A Modular Multi-LMM Agent Framework for Video Question-Answering

arXiv 2404.01476 ↗First posted 1 Apr 2024cs.CV cs.AI cs.CLcited by 24

On the map: Multimodal LLMs / vision-language models → Video understanding with MLLMs

Abstract (arXiv)

Recently, image-based Large Multimodal Models (LMMs) have made significant progress in video question-answering (VideoQA) using a frame-wise approach by leveraging large-scale pretraining in a zero-shot manner. Nevertheless, these models need to be capable of finding relevant information, extracting it, and answering the question simultaneously. Currently, existing methods perform all of these steps in a single pass without being able to adapt if insufficient or incorrect information is collected. To overcome this, we introduce a modular multi-LMM agent framework based on several agents with different roles, instructed by a Planner agent that updates its instructions using shared feedback from the other agents. Specifically, we propose TraveLER, a method that can create a plan to "Traverse" through the video, ask questions about individual frames to "Locate" and store key information, and then "Evaluate" if there is enough information to answer the question. Finally, if there is not enough information, our method is able to "Replan" based on its collected knowledge. Through extensive experiments, we find that the proposed TraveLER approach improves performance on several VideoQA benchmarks without the need to fine-tune on specific datasets. Our code is available at https://github.com/traveler-framework/TraveLER.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 46 of its 59 references that are in the map. First posted on arXiv 1 Apr 2024.

Builds on

46 of 59 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projection16 Nov
  2. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding5 Jun
  3. 2023Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning27 Feb

Other references · 43

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  3. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  4. 2018Relational inductive biases, deep learning, and graph networkscited by 1,341
  5. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  6. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 666
  7. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 591
  8. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  9. 2022Visual Programming: Compositional visual reasoning without trainingcited by 507
  10. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 488
  11. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  12. 2015Neural Module Networkscited by 375
31 more
  1. 2023Perception Test: A Diagnostic Benchmark for Multimodal Video Modelscited by 310
  2. 2020Just Ask: Learning to Answer Questions from Millions of Narrated Videoscited by 248
  3. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
  4. 2023VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Datasetcited by 168
  5. 2022Revisiting the "Video" in Video-Language Understandingcited by 167
  6. 2023Compositional Chain-of-Thought Prompting for Large Multimodal Modelscited by 160
  7. 2022Language Models with Image Descriptors are Strong Few-Shot Video-Language Learnerscited by 117
  8. 2023MM-VID: Advancing Video Understanding with GPT-4V(ision)cited by 82
  9. 2022An Empirical Study of End-to-End Video-Language Transformers with Masked Visual Modelingcited by 74
  10. 2022CLIP-ViP: Adapting Pre-trained Image-Text Model to Video-Language Representation Alignmentcited by 74
  11. 2022LAVENDER: Unifying Video-Language Understanding as Masked Language Modelingcited by 73
  12. 2021QVHighlights: Detecting Moments and Highlights in Videos via Natural Language Queriescited by 72
  13. 2022HiTeA: Hierarchical Temporal-Aware Video-Language Pre-trainingcited by 72
  14. 2018Object Level Visual Reasoning in Videoscited by 67
  15. 2019Multi-Agent Reinforcement Learning Based Frame Sampling for Effective Untrimmed Video Recognitioncited by 66
  16. 2019Learning Canonical Representations for Scene Graph to Image Generationcited by 60
  17. 2022Long-Form Video-Language Pre-Training with Multimodal Temporal Contrastive Learningcited by 59
  18. 2016Leveraging Video Descriptions to Learn Video Question Answeringcited by 55
  19. 2023Modular Visual Question Answering via Code Generationcited by 47
  20. 2023MM-Narrator: Narrating Long-form Videos with Multimodal In-Context Learningcited by 46
  21. 2024Memory Consolidation Enables Long-Context Video Understandingcited by 45
  22. 2018Mapping Images to Scene Graphs with Permutation-Invariant Structured Predictioncited by 44
  23. 2016A dataset and exploration of models for understanding video data through fill-in-the-blank question-answeringcited by 41
  24. 2023Incorporating Structured Representations into Pretrained Vision & Language Models Using Scene Graphscited by 39
  25. 2023Zero-Shot Video Question Answering with Procedural Programscited by 37
  26. 2020Compositional Video Synthesis with Action Graphscited by 34
  27. 2023A Simple Recipe for Contrastively Pre-training Video-First Encoders Beyond 16 Framescited by 34
  28. 2018Referring Relationshipscited by 29
  29. 2020Learning Object Detection from Captions via Textual Scene Attributescited by 14
  30. 2022PromptonomyViT: Multi-Task Prompt Learning Improves Video Transformers using Synthetic Scene Datacited by 14
  31. 2022LGDN: Language-Guided Denoising Network for Video-Language Modelingcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 10; 2026: 10.

24 papers.

Sort:

By area:

  1. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  2. 2024VLM-Grounder: A VLM Agent for Zero-Shot 3D Visual Groundingcited by 56
  3. 2024VideoQA in the Era of LLMs: An Empirical Studycited by 29
  4. 2024VideoINSTA: Zero-shot Long Video Understanding via Informative Spatial-Temporal Reasoning with LLMscited by 29
  5. 2025RefactorBench: Evaluating Stateful Reasoning in Language Agents Through Codecited by 28
  6. 2024Question-Answering Dense Video Eventscited by 11
  7. 2025VTimeCoT: Thinking by Drawing for Video Temporal Grounding and Reasoningcited by 11
  8. 2026Think, Then Verify: A Hypothesis-Verification Multi-Agent Framework for Long Video Understandingcited by 8
  9. 2025ENTER: Event Based Interpretable Reasoning for VideoQAcited by 5
  10. 2025ViQAgent: Zero-Shot Video Question Answering via Agent with Open-Vocabulary Grounding Validationcited by 5
  11. 2026VideoSEAL: Mitigating Evidence Misalignment in Agentic Long Video Understanding by Decoupling Answer Authoritycited by 5
  12. 2025A Neurosymbolic Agent System for Compositional Visual Reasoningcited by 4
12 more
  1. 2025Select Less, Reason More: Prioritizing Evidence Purity for Video Reasoningcited by 4
  2. 2025Empowering Multimodal LLMs with External Tools: A Comprehensive Surveycited by 2
  3. 2026A Multi-Agent Perception-Action Alliance for Efficient Long Video Reasoningcited by 2
  4. 2026Progressive Video Condensation with MLLM Agent for Long-form Video Understandingcited by 2
  5. 2025Does It Run and Is That Enough? Revisiting Text-to-Chart Generation with a Multi-Agent Approachcited by 1
  6. 2025DIVE: Deep-search Iterative Video Exploration A Technical Report for the CVRR Challenge at CVPR 2025cited by 1
  7. 2026Video Active Perception: Effective Inference-Time Long-Form Video Understanding with Vision-Language Modelscited by 1
  8. 2026UpstreamQA: A Modular Framework for Explicit Reasoning on Video Question Answering Taskscited by 0
  9. 2026Tracking the Truth: Object-Centric Spatio-Temporal Monitoring for Video Large Language Modelscited by 0
  10. 2026ADEPT: An Entropy-Driven Dual-Strategy Agent for Interactive Video Retrievalcited by 0
  11. 2026VideoScout: Learning Agentic Active Exploration with Adaptive Reasoning Pacing for Long Video Understandingcited by 0
  12. 2026STRAND: Benchmarking and Improving Object-Centric Spatio-Temporal Monitoring in Video Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025VideoMultiAgents: A Multi-Agent Framework for Video Question Answering
  2. 2026AgentVidBench: A Multi-Hop Video Question Answering Benchmark for Evaluating MLLM Agents
  3. 2025Team of One: Cracking Complex Video QA with Model Synergy
  4. 2025Rethinking Information Synthesis in Multimodal Question Answering A Multi-Agent Perspective
  5. 2025MAGNET: A Multi-agent Framework for Finding Audio-Visual Needles by Reasoning over Multi-Video Haystacks
  6. 2024VidCtx: Context-aware Video Question Answering with Image Models
  7. 2026VTAgent: Agentic Keyframe Anchoring for Evidence-Aware Video TextVQA
  8. 2024Multi-Agents Based on Large Language Models for Knowledge-based Visual Question Answering