mlsanityData to 14 Oct 2026

Frame-Voyager: Learning to Query Frames for Video Large Language Models

arXiv 2410.03226 ↗First posted 4 Oct 2024cs.CV cs.CLcited by 83

Not on the map. Why?

Cited by 43 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Video Large Language Models (Video-LLMs) have made remarkable progress in video understanding tasks. However, they are constrained by the maximum length of input tokens, making it impractical to input entire videos. Existing frame selection approaches, such as uniform frame sampling and text-frame retrieval, fail to account for the information density variations in the videos or the complex instructions in the tasks, leading to sub-optimal performance. In this paper, we propose Frame-Voyager that learns to query informative frame combinations, based on the given textual queries in the task. To train Frame-Voyager, we introduce a new data collection and labeling pipeline, by ranking frame combinations using a pre-trained Video-LLM. Given a video of M frames, we traverse its T-frame combinations, feed them into a Video-LLM, and rank them based on Video-LLM's prediction losses. Using this ranking as supervision, we train Frame-Voyager to query the frame combinations with lower losses. In experiments, we evaluate Frame-Voyager on four Video Question Answering benchmarks by plugging it into two different Video-LLMs. The experimental results demonstrate that Frame-Voyager achieves impressive results in all settings, highlighting its potential as a plug-and-play solution for Video-LLMs.

Key facts. Cited by 83 papers in the map (data through 14 Oct 2026). Builds on 57 of its 74 references that are in the map. First posted on arXiv 4 Oct 2024.

Builds on

57 of 74 references.

References · 57

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2022Training language models to follow instructions with human feedbackcited by 15,971
  6. 2024The Llama 3 Herd of Modelscited by 14,715
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  8. 2023Visual Instruction Tuningcited by 7,869
  9. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  10. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  11. 2017On Calibration of Modern Neural Networkscited by 4,960
  12. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
38 more
  1. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  2. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  3. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  4. 2020Learning to summarize from human feedbackcited by 2,588
  5. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  6. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  7. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  8. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,413
  9. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,402
  10. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,109
  11. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  12. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
  13. 2023VideoChat: Chat-Centric Video Understandingcited by 857
  14. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  15. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 771
  16. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 744
  17. 2023VILA: On Pre-training for Visual Language Modelscited by 692
  18. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 680
  19. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 584
  20. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 561
  21. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 542
  22. 2024Long Context Transfer from Language to Visioncited by 515
  23. 2019VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Researchcited by 414
  24. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 407
  25. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 371
  26. 2024VideoAgent: Long-form Video Understanding with Large Language Model as Agentcited by 332
  27. 2024LongVILA: Scaling Long-Context Visual Language Models for Long Videoscited by 309
  28. 2023Effective Long-Context Scaling of Foundation Modelscited by 303
  29. 2024VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videoscited by 251
  30. 2020Span-based Localizing Network for Natural Language Video Localizationcited by 235
  31. 2024STAR: A Benchmark for Situated Reasoning in Real-World Videoscited by 213
  32. 2023Self-Chained Image-Language Model for Video Localization and Question Answeringcited by 203
  33. 2023Efficient Large Language Models: A Surveycited by 186
  34. 2024VITA: Towards Open-Source Interactive Omni Multimodal LLMcited by 173
  35. 2023Can I Trust Your Answer? Visually Grounded Video Question Answeringcited by 135
  36. 2024ST-LLM: Large Language Models Are Effective Temporal Learnerscited by 126
  37. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 102
  38. 2018Video Summarization Using Fully Convolutional Sequence Networkscited by 71

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 38; 2026: 45.

83 papers.

Sort:

By area:

  1. 2025T*: Re-thinking Temporal Search for Long-Form Video Understandingcited by 82
  2. 2025Q-Frame: Query-aware Frame Selection and Multi-Resolution Adaptation for Video-LLMscited by 80
  3. 2025BOLT: Boost Large Vision-Language Model Without Training for Long-form Video Understandingcited by 75
  4. 2025Eagle 2.5: Boosting Long-Context Post-Training for Frontier Vision-Language Modelscited by 64
  5. 2025MDP3: A Training-free Approach for List-wise Frame Selection in Video-LLMscited by 43
  6. 2025FOCUS: Efficient Keyframe Selection for Long Video Understandingcited by 37
  7. 2025Logic-in-Frames: Dynamic Keyframe Search via Visual Semantic-Logical Verification for Long Video Understandingcited by 34
  8. 2025VideoITG: Multimodal Video Understanding with Instructed Temporal Groundingcited by 29
  9. 2025FRAG: Frame Selection Augmented Generation for Long Video and Long Document Understandingcited by 27
  10. 2025VideoZoomer: Reinforcement-Learned Temporal Focusing for Long Video Reasoningcited by 27
  11. 2025Improving LLM Video Understanding with 16 Frames Per Secondcited by 26
  12. 2025TimeSearch-R: Adaptive Temporal Search for Long-Form Video Understanding via Self-Verification Reinforcement Learningcited by 20
38 more
  1. 2026OASIS: On-Demand Hierarchical Event Memory for Streaming Video Reasoningcited by 19
  2. 2025MaxInfo: A Training-Free Key-Frame Selection Method Using Maximum Volume for Enhanced Video Understandingcited by 18
  3. 2026LensWalk: Agentic Video Understanding by Planning How You See in Videoscited by 17
  4. 2025ReFoCUS: Reinforcement-guided Frame Optimization for Contextual Understandingcited by 16
  5. 2025Less Is More, but Where? Dynamic Token Compression via LLM-Guided Keyframe Priorcited by 15
  6. 2026EgoMemReason: A Memory-Driven Reasoning Benchmark for Long-Horizon Egocentric Video Understandingcited by 13
  7. 2025ViaRL: Adaptive Temporal Grounding via Visual Iterated Amplification Reinforcement Learningcited by 11
  8. 2025K-frames: Scene-Driven Any-k Keyframe Selection for long video understandingcited by 11
  9. 2025Divide, then Ground: Adapting Frame Selection to Query Types for Long-Form Video Understandingcited by 11
  10. 2025Threading Keyframe with Narratives: MLLMs as Strong Long Video Comprehenderscited by 10
  11. 2025Video-EM: Event-Centric Episodic Memory for Long-Form Video Understandingcited by 10
  12. 2025From Frames to Clips: Training-free Adaptive Key Clip Selection for Long-Form Video Understandingcited by 9
  13. 2025AdaRD-key: Adaptive Relevance-Diversity Keyframe Sampling for Long-form Video understandingcited by 9
  14. 2026Attend Before Attention: Efficient and Scalable Video Understanding via Autoregressive Gazingcited by 9
  15. 2026Non-Markovian Long-Horizon Robot Manipulation via Keyframe Chainingcited by 8
  16. 2026GIFT: Global Irreplaceability Frame Targeting for Efficient Video Understandingcited by 8
  17. 2025FrameOracle: Learning What to See and How Much to See in Videoscited by 7
  18. 2026Event-Anchored Frame Selection for Effective Long-Video Understandingcited by 7
  19. 2026AdaCodec: A Predictive Visual Code for Video MLLMscited by 7
  20. 2025Frame Sampling Strategies Matter: A Benchmark for small vision language modelscited by 6
  21. 2025ViSS-R1: Self-Supervised Reinforcement Video Reasoningcited by 6
  22. 2025KFS-Bench: Comprehensive Evaluation of Key Frame Sampling in Long Video Understandingcited by 6
  23. 2025TimeScope: Towards Task-Oriented Temporal Grounding In Long Videoscited by 5
  24. 2025SeViCES: Unifying Semantic-Visual Evidence Consensus for Long Video Understandingcited by 5
  25. 2025Poisoning Prompt-Guided Sampling in Video Large Language Modelscited by 4
  26. 2025Training-free Uncertainty Guidance for Complex Visual Tasks with MLLMscited by 4
  27. 2026HiMu: Hierarchical Multimodal Frame Selection for Long Video Question Answeringcited by 4
  28. 2026Scaling Video Understanding via Compact Latent Multi-Agent Collaborationcited by 4
  29. 2026GridProbe: Posterior-Probing for Adaptive Test-Time Compute in Long-Video VLMscited by 4
  30. 2025A.I.R.: Enabling Adaptive, Iterative, and Reasoning-based Frame Selection For Video Question Answeringcited by 3
  31. 2025Test-Time Temporal Sampling for Efficient MLLM Video Understandingcited by 3
  32. 2026Question-guided Visual Compression with Memory Feedback for Long-Term Video Understandingcited by 3
  33. 2026ViKey: Enhancing Temporal Understanding in Videos via Visual Promptingcited by 3
  34. 2026VisualClaw: A Real-Time, Personalized Agent for the Physical Worldcited by 3
  35. 2025AutoV: Loss-Oriented Ranking for Visual Prompt Retrieval in LVLMscited by 2
  36. 2025FC-MIR: A Mobile Screen Awareness Framework for Intent-Aware Recommendation based on Frame-Compressed Multimodal Trajectory Reasoningcited by 2
  37. 2026AdaptToken: Entropy-based Adaptive Token Selection for MLLM Long Video Understandingcited by 2
  38. 2026LDDR: Linear-DPP-Based Dynamic-Resolution Frame Sampling for Video MLLMscited by 2

50 most cited of 83 All 83 (JSON)