mlsanityData to 7 Oct 2026

A Simple LLM Framework for Long-Range Video Question-Answering

arXiv 2312.17235 ↗First posted 28 Dec 2023cs.CVcited by 176

Not on the map. Why?

Cited by 54 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present LLoVi, a language-based framework for long-range video question-answering (LVQA). Unlike prior long-range video understanding methods, which are often costly and require specialized long-range video modeling design (e.g., memory queues, state-space layers, etc.), our approach uses a frame/clip-level visual captioner (e.g., BLIP2, LaViLa, LLaVA) coupled with a Large Language Model (GPT-3.5, GPT-4) leading to a simple yet surprisingly effective LVQA framework. Specifically, we decompose short and long-range modeling aspects of LVQA into two stages. First, we use a short-term visual captioner to generate textual descriptions of short video clips (0.5-8s in length) densely sampled from a long input video. Afterward, an LLM aggregates the densely extracted short-term captions to perform long-range temporal reasoning needed to understand the whole video and answer a question. To analyze what makes our simple framework so effective, we thoroughly evaluate various components of our system. Our empirical analysis reveals that the choice of the visual captioner and LLM is critical for good LVQA performance. Furthermore, we show that a specialized prompt that asks the LLM first to summarize the noisy short-term visual captions and then answer a given input question leads to a significant LVQA performance boost. On EgoSchema, which is best known as a very long-form video question-answering benchmark, our method achieves 50.3% accuracy, outperforming the previous best-performing approach by 18.1% (absolute gain). In addition, our approach outperforms the previous state-of-the-art by 4.1% and 3.1% on NeXT-QA and IntentQA. We also extend LLoVi to grounded LVQA and show that it outperforms all prior methods on the NeXT-GQA dataset. We will release our code at https://github.com/CeeZh/LLoVi.

Key facts. Cited by 176 papers in the map (data through 7 Oct 2026). Builds on 79 of its 90 references that are in the map. First posted on arXiv 28 Dec 2023.

Builds on

79 of 90 references.

References · 79

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  8. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  9. 2023Visual Instruction Tuningcited by 7,792
  10. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  11. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  12. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
38 more
  1. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  2. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  3. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  4. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  5. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  6. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  7. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
  8. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  9. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,020
  10. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  11. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  12. 2023Reasoning with Language Model is Planning with World Modelcited by 846
  13. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  14. 2023CogAgent: A Visual Language Model for GUI Agentscited by 692
  15. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 666
  16. 2023Automatic Prompt Optimization with "Gradient Descent" and Beam Searchcited by 628
  17. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 591
  18. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 559
  19. 2023Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Modelscited by 550
  20. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 536
  21. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  22. 2023LLM+P: Empowering Large Language Models with Optimal Planning Proficiencycited by 496
  23. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  24. 2018TVQA: Localized, Compositional Video Question Answeringcited by 463
  25. 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Modelscited by 459
  26. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 441
  27. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 350
  28. 2015MovieQA: Understanding Stories in Movies through Question-Answeringcited by 304
  29. 2022UL2: Unifying Language Learning Paradigmscited by 292
  30. 2020Just Ask: Learning to Answer Questions from Millions of Narrated Videoscited by 248
  31. 2018Long-Term Feature Banks for Detailed Video Understandingcited by 239
  32. 2022Egocentric Video-Language Pretrainingcited by 231
  33. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
  34. 2022Learning Video Representations from Large Language Modelscited by 204
  35. 2023Self-Chained Image-Language Model for Video Localization and Question Answeringcited by 200
  36. 2022MeMViT: Memory-Augmented Multiscale Vision Transformer for Efficient Long-Term Video Recognitioncited by 189
  37. 2021VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modelingcited by 187
  38. 2023GPT-NER: Named Entity Recognition via Large Language Modelscited by 158

50 most cited of 79 All 79 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 51; 2025: 83; 2026: 41.

176 papers.

Sort:

By area:

  1. 2024VideoAgent: Long-form Video Understanding with Large Language Model as Agentcited by 323
  2. 2024PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioningcited by 320
  3. 2024VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videoscited by 242
  4. 2024VideoAgent: A Memory-augmented Multimodal Agent for Video Understandingcited by 224
  5. 2023Video Understanding with Large Language Models: A Surveycited by 203
  6. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Outputcited by 195
  7. 2024Streaming Long Video Understanding with Large Language Modelscited by 164
  8. 2024Tarsier: Recipes for Training and Evaluating Large Video Description Modelscited by 140
  9. 2024Video-RAG: Visually-aligned Retrieval-Augmented Long Video Comprehensioncited by 134
  10. 2024SlowFast-LLaVA: A Strong Training-Free Baseline for Video Large Language Modelscited by 123
  11. 2025Dispider: Enabling Video LLMs with Active Real-Time Interaction via Disentangled Perception, Decision, and Reactioncited by 122
  12. 2024CinePile: A Long Video Question Answering Dataset and Benchmarkcited by 112
38 more
  1. 2025Streaming Video Question-Answering with In-context Video KV-Cache Retrievalcited by 112
  2. 2024HourVideo: 1-Hour Video-Language Understandingcited by 110
  3. 2024An Image Grid Can Be Worth a Video: Zero-shot Video Question Answering Using a VLMcited by 97
  4. 2024Grounded-VideoLLM: Sharpening Fine-grained Temporal Grounding in Video Large Language Modelscited by 87
  5. 2025T*: Re-thinking Temporal Search for Long-Form Video Understandingcited by 82
  6. 2025BOLT: Boost Large Vision-Language Model Without Training for Long-form Video Understandingcited by 72
  7. 2024Too Many Frames, Not All Useful: Efficient Strategies for Long-Form Video QAcited by 68
  8. 2024DrVideo: Document Retrieval Based Long Video Understandingcited by 63
  9. 2025VideoChat-R1.5: Visual Test-Time Scaling to Reinforce Multimodal Reasoning by Iterative Perceptioncited by 60
  10. 2024Language Repository for Long Video Understandingcited by 55
  11. 2024VCA: Video Curious Agent for Long Video Understandingcited by 54
  12. 2024InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for Long-term Streaming Video and Audio Interactionscited by 48
  13. 2024Memory Consolidation Enables Long-Context Video Understandingcited by 45
  14. 2025VideoMind: A Chain-of-LoRA Agent for Temporal-Grounded Video Reasoningcited by 41
  15. 2025VideoAgent2: Enhancing the LLM-Based Agent System for Long-Form Video Understanding by Uncertainty-Aware CoTcited by 36
  16. 2024AMEGO: Active Memory from long EGOcentric videoscited by 35
  17. 2024xGen-MM-Vid (BLIP-3-Video): You Only Need 32 Tokens to Represent a Video Even in VLMscited by 34
  18. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  19. 2025A Shortcut-aware Video-QA Benchmark for Physical Understanding via Minimal Video Pairscited by 30
  20. 2025Vgent: Graph-based Retrieval-Reasoning-Augmented Generation For Long Video Understandingcited by 30
  21. 2024VideoQA in the Era of LLMs: An Empirical Studycited by 29
  22. 2024VideoINSTA: Zero-shot Long Video Understanding via Informative Spatial-Temporal Reasoning with LLMscited by 29
  23. 2024Task Preference Optimization: Improving Multimodal Large Language Models with Vision Task Alignmentcited by 29
  24. 2025Active Video Perception: Iterative Evidence Seeking for Agentic Long Video Understandingcited by 26
  25. 2024Understanding Long Videos with Multimodal Language Modelscited by 25
  26. 2025LLaVA-Scissor: Token Compression with Semantic Connected Components for Video LLMscited by 25
  27. 2025VideoARM: Agentic Reasoning over Hierarchical Memory for Long-Form Video Understandingcited by 25
  28. 2025BIMBA: Selective-Scan Compression for Long-Range Video Question Answeringcited by 23
  29. 2025MR. Video: "MapReduce" is the Principle for Long Video Understandingcited by 23
  30. 2024Coarse Correspondences Boost Spatial-Temporal Reasoning in Multimodal Language Modelcited by 21
  31. 2025VideoLucy: Deep Memory Backtracking for Long Video Understandingcited by 21
  32. 2024VideoLLaMB: Long Streaming Video Understanding with Recurrent Memory Bridgescited by 20
  33. 2025Building a Mind Palace: Structuring Environment-Grounded Semantic Graphs for Effective Long Video Analysis with LLMscited by 19
  34. 2024Motion-Grounded Video Reasoning: Understanding and Perceiving Motion at Pixel Levelcited by 18
  35. 2024ReVisionLLM: Recursive Vision-Language Model for Temporal Grounding in Hour-Long Videoscited by 18
  36. 2024Chrono: A Simple Blueprint for Representing Time in MLLMscited by 17
  37. 2024Adaptive Video Understanding Agent: Enhancing efficiency with dynamic frame sampling and feedback-driven reasoningcited by 16
  38. 2025VideoMultiAgents: A Multi-Agent Framework for Video Question Answeringcited by 16

50 most cited of 176 All 176 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding
  2. 2024Question-Instructed Visual Descriptions for Zero-Shot Video Question Answering
  3. 2025TextVidBench: A Benchmark for Long Video Scene Text Understanding
  4. 2024Perceive, Query & Reason: Enhancing Video QA with Question-Guided Temporal Queries
  5. 2025Video Panels for Long Video Understanding
  6. 2025An Empirical Study for Representations of Videos in Video Question Answering via MLLMs
  7. 2023VaQuitA: Enhancing Alignment in LLM-Assisted Video Understanding
  8. 2025HLV-1K: A Large-scale Hour-Long Video Benchmark for Time-Specific Long Video Understanding