mlsanityData to 7 Oct 2026

Too Many Frames, Not All Useful: Efficient Strategies for Long-Form Video QA

arXiv 2406.09396 ↗First posted 13 Jun 2024cs.CVcited by 68

Not on the map. Why?

Cited by 25 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Long-form videos that span across wide temporal intervals are highly information redundant and contain multiple distinct events or entities that are often loosely related. Therefore, when performing long-form video question answering (LVQA), all information necessary to generate a correct response can often be contained within a small subset of frames. Recent literature leverage large language models (LLMs) in LVQA benchmarks, achieving exceptional performance, while relying on vision language models (VLMs) to convert all visual content within videos into natural language. Such VLMs often independently caption a large number of frames uniformly sampled from long videos, which is not efficient and can mostly be redundant. Motivated by this inefficiency, we propose LVNet, a modular and training-free framework featuring a novel Hierarchical Keyframe Selector (HKS) that efficiently selects a minimal set of informative frames tailored to each question. LVNet's modularity allows easy integration with existing approaches for more efficient LVQA. We achieve state-of-the-art performance among similarly configured models across four benchmark LVQA datasets: EgoSchema, NExT-QA, IntentQA, VideoMME. The code can be found at https://github.com/jongwoopark7978/LVNet

Key facts. Cited by 68 papers in the map (data through 7 Oct 2026). Builds on 49 of its 71 references that are in the map. First posted on arXiv 13 Jun 2024.

Builds on

49 of 71 references.

References · 49

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  5. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  6. 2015VQA: Visual Question Answeringcited by 2,489
  7. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  8. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  9. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  10. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  11. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  12. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
37 more
  1. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  2. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 666
  3. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 591
  4. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 559
  5. 2018TVQA: Localized, Compositional Video Question Answeringcited by 463
  6. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 441
  7. 2024VideoAgent: Long-form Video Understanding with Large Language Model as Agentcited by 323
  8. 2015MovieQA: Understanding Stories in Movies through Question-Answeringcited by 304
  9. 2024VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videoscited by 242
  10. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
  11. 2024VideoAgent: A Memory-augmented Multimodal Agent for Video Understandingcited by 224
  12. 2023Self-Chained Image-Language Model for Video Localization and Question Answeringcited by 200
  13. 2023A Simple LLM Framework for Long-Range Video Question-Answeringcited by 176
  14. 2019TVQA+: Spatio-Temporal Grounding for Video Question Answeringcited by 153
  15. 2024CinePile: A Long Video Question Answering Dataset and Benchmarkcited by 112
  16. 2024An Image Grid Can Be Worth a Video: Zero-shot Video Question Answering Using a VLMcited by 97
  17. 2022Perceptual Grouping in Contrastive Vision-Language Modelscited by 82
  18. 2023Verbs in Action: Improving verb understanding in video-language modelscited by 76
  19. 2022An Empirical Study of End-to-End Video-Language Transformers with Masked Visual Modelingcited by 74
  20. 2024MoReVQA: Exploring Modular Reasoning Models for Video Question Answeringcited by 74
  21. 2022HiTeA: Hierarchical Temporal-Aware Video-Language Pre-trainingcited by 72
  22. 2021Video as Conditional Graph Hierarchy for Multi-Granular Question Answeringcited by 68
  23. 2022Video Graph Transformer for Video Question Answeringcited by 63
  24. 2016Leveraging Video Descriptions to Learn Video Question Answeringcited by 55
  25. 2024Language Repository for Long Video Understandingcited by 55
  26. 2022From Representation to Reasoning: Towards both Evidence and Commonsense Reasoning for Video Question-Answeringcited by 51
  27. 2023Large Language Models are Temporal and Causal Reasoners for Video Question Answeringcited by 51
  28. 2023LifelongMemory: Leveraging LLMs for Answering Queries in Long-form Egocentric Videoscited by 51
  29. 2024Memory Consolidation Enables Long-Context Video Understandingcited by 45
  30. 2023Vamos: Versatile Action Models for Video Understandingcited by 40
  31. 2023Zero-Shot Video Question Answering with Procedural Programscited by 37
  32. 2023A Simple Recipe for Contrastively Pre-training Video-First Encoders Beyond 16 Framescited by 34
  33. 2023Contrastive Video Question Answering via Video Graph Transformercited by 32
  34. 2024Understanding Long Videos with Multimodal Language Modelscited by 25
  35. 2016Single Image Action Recognition using Semantic Body Part Actionscited by 14
  36. 2023Semi-Parametric Video-Grounded Text Generationcited by 13
  37. 2021Knowledge-Augmented Language Models for Cause-Effect Relation Classificationcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 44; 2026: 20.

68 papers.

Sort:

By area:

  1. 2025M-LLM Based Video Frame Selection for Efficient Video Understandingcited by 87
  2. 2025T*: Re-thinking Temporal Search for Long-Form Video Understandingcited by 82
  3. 2024VCA: Video Curious Agent for Long Video Understandingcited by 54
  4. 2025WorldMM: Dynamic Multimodal Memory Agent for Long Video Reasoningcited by 50
  5. 2025VideoAgent2: Enhancing the LLM-Based Agent System for Long-Form Video Understanding by Uncertainty-Aware CoTcited by 36
  6. 2024AMEGO: Active Memory from long EGOcentric videoscited by 35
  7. 2024xGen-MM-Vid (BLIP-3-Video): You Only Need 32 Tokens to Represent a Video Even in VLMscited by 34
  8. 2025Logic-in-Frames: Dynamic Keyframe Search via Visual Semantic-Logical Verification for Long Video Understandingcited by 30
  9. 2025Temporal Chain of Thought: Long-Video Understanding by Thinking in Framescited by 27
  10. 2025FRAG: Frame Selection Augmented Generation for Long Video and Long Document Understandingcited by 26
  11. 2025Active Video Perception: Iterative Evidence Seeking for Agentic Long Video Understandingcited by 26
  12. 2025VideoLucy: Deep Memory Backtracking for Long Video Understandingcited by 21
38 more
  1. 2024PPLLaVA: Varied Video Sequence Understanding With Prompt Guidancecited by 20
  2. 2025Keyframe-oriented Vision Token Pruning: Enhancing Efficiency of Large Vision Language Models on Long-Form Video Processingcited by 18
  3. 2025TimeSearch-R: Adaptive Temporal Search for Long-Form Video Understanding via Self-Verification Reinforcement Learningcited by 18
  4. 2025Time Blindness: Why Video-Language Models Can't See What Humans Can?cited by 17
  5. 2025VideoMultiAgents: A Multi-Agent Framework for Video Question Answeringcited by 16
  6. 2026LensWalk: Agentic Video Understanding by Planning How You See in Videoscited by 14
  7. 2025Agentic Keyframe Search for Video Question Answeringcited by 13
  8. 2025CLiViS: Unleashing Cognitive Map through Linguistic-Visual Synergy for Embodied Visual Reasoningcited by 13
  9. 2025Active Perception Agent for Omnimodal Audio-Video Understandingcited by 13
  10. 2025Threading Keyframe with Narratives: MLLMs as Strong Long Video Comprehenderscited by 10
  11. 2025Tool-Augmented Spatiotemporal Reasoning for Streamlining Video Question Answering Taskcited by 10
  12. 2025AdaRD-key: Adaptive Relevance-Diversity Keyframe Sampling for Long-form Video understandingcited by 9
  13. 2025Flow4Agent: Long-form Video Understanding via Motion Prior from Optical Flowcited by 9
  14. 2026Think, Then Verify: A Hypothesis-Verification Multi-Agent Framework for Long Video Understandingcited by 8
  15. 2025APVR: Hour-Level Long Video Understanding with Adaptive Pivot Visual Information Retrievalcited by 7
  16. 2025NeuS-QA: Grounding Long-Form Video Understanding in Temporal Logic and Neuro-Symbolic Reasoningcited by 7
  17. 2025FrameOracle: Learning What to See and How Much to See in Videoscited by 7
  18. 2026Event-Anchored Frame Selection for Effective Long-Video Understandingcited by 7
  19. 2025Video-LevelGauge: Investigating Contextual Positional Bias in Large Video Language Modelscited by 6
  20. 2025ReasVQA: Advancing VideoQA with Imperfect Reasoning Processcited by 5
  21. 2025ENTER: Event Based Interpretable Reasoning for VideoQAcited by 5
  22. 2025Moment Sampling in Video LLMs for Long-Form Video QAcited by 5
  23. 2025FineQuest: Adaptive Knowledge-Assisted Sports Video Understanding via Agent-of-Thoughts Reasoningcited by 5
  24. 2025AirVista-II: An Agentic System for Embodied UAVs Toward Dynamic Scene Semantic Understandingcited by 4
  25. 2025MASR: Self-Reflective Reasoning through Multimodal Hierarchical Attention Focusing for Agent-based Video Understandingcited by 4
  26. 2025LeAdQA: LLM-Driven Context-Aware Temporal Grounding for Video Question Answeringcited by 4
  27. 2025Select Less, Reason More: Prioritizing Evidence Purity for Video Reasoningcited by 4
  28. 2025PhyVLLM: Physics-Guided Video Language Model with Motion-Appearance Disentanglementcited by 4
  29. 2025Video-QTR: Query-Driven Temporal Reasoning Framework for Lightweight Video Understandingcited by 4
  30. 2026One Token per Highly Selective Frame: Towards Extreme Compression for Long Video Understandingcited by 4
  31. 2025Everything Can Be Described in Words: A Simple Unified Multi-Modal Framework with Semantic and Temporal Alignmentcited by 3
  32. 2026FocusGraph: Graph-Structured Frame Selection for Embodied Long Video Question Answeringcited by 3
  33. 2025TriPSS: A Tri-Modal Keyframe Extraction Framework Using Perceptual, Structural, and Semantic Representationscited by 2
  34. 2025Empowering Multimodal LLMs with External Tools: A Comprehensive Surveycited by 2
  35. 2025EEA: Exploration-Exploitation Agent for Long Video Understandingcited by 2
  36. 2025FC-MIR: A Mobile Screen Awareness Framework for Intent-Aware Recommendation based on Frame-Compressed Multimodal Trajectory Reasoningcited by 2
  37. 2026TiFRe: Text-guided Video Frame Reduction for Efficient Video Multi-modal Large Language Modelscited by 2
  38. 2026RT-NeuS: Towards Real-Time Neuro-Symbolic Video Understanding via Adaptive Temporal Verificationcited by 2

50 most cited of 68 All 68 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025A.I.R.: Enabling Adaptive, Iterative, and Reasoning-based Frame Selection For Video Question Answering
  2. 2025Enhancing Long Video Question Answering with Scene-Localized Frame Grouping
  3. 2026Improving Video Question Answering through query-based frame selection
  4. 2026Adaptive Greedy Frame Selection for Long Video Understanding
  5. 2025Less is More: Token-Efficient Video-QA via Adaptive Frame-Pruning and Semantic Graph Integration
  6. 2024VidCtx: Context-aware Video Question Answering with Image Models
  7. 2025HLV-1K: A Large-scale Hour-Long Video Benchmark for Time-Specific Long Video Understanding
  8. 2025KFS-Bench: Comprehensive Evaluation of Key Frame Sampling in Long Video Understanding