mlsanityData to 7 Oct 2026

M-LLM Based Video Frame Selection for Efficient Video Understanding

arXiv 2502.19680 ↗First posted 27 Feb 2025cs.CV cs.AIcited by 87

On the map: Multimodal LLMs / vision-language models → Video understanding with MLLMs

Abstract (arXiv)

Recent advances in Multi-Modal Large Language Models (M-LLMs) show promising results in video reasoning. Popular Multi-Modal Large Language Model (M-LLM) frameworks usually apply naive uniform sampling to reduce the number of video frames that are fed into an M-LLM, particularly for long context videos. However, it could lose crucial context in certain periods of a video, so that the downstream M-LLM may not have sufficient visual information to answer a question. To attack this pain point, we propose a light-weight M-LLM -based frame selection method that adaptively select frames that are more relevant to users' queries. In order to train the proposed frame selector, we introduce two supervision signals (i) Spatial signal, where single frame importance score by prompting a M-LLM; (ii) Temporal signal, in which multiple frames selection by prompting Large Language Model (LLM) using the captions of all frame candidates. The selected frames are then digested by a frozen downstream video M-LLM for visual reasoning and question answering. Empirical results show that the proposed M-LLM video frame selector improves the performances various downstream video Large Language Model (video-LLM) across medium (ActivityNet, NExT-QA) and long (EgoSchema, LongVideoBench) context video question answering benchmarks.

Key facts. Cited by 87 papers in the map (data through 7 Oct 2026). Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 51 of its 60 references that are in the map. First posted on arXiv 27 Feb 2025.

Builds on

51 of 60 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024VideoEspresso: A Large-Scale Chain-of-Thought Dataset for Fine-Grained Video Reasoning via Core Frame Selection22 Nov
  2. 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding22 Jul
  3. 2024MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding20 Jun
  4. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis31 May
  5. 2024PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioning25 Apr
  6. 2024MiniGPT4-Video: Advancing Multimodal LLMs for Video Understanding with Interleaved Visual-Textual Tokens4 Apr
  7. 2023TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding4 Dec
  8. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models28 Nov
  9. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmark28 Nov
  10. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projection16 Nov

Other references · 41

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  6. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  7. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  8. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  9. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  10. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  11. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  12. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
29 more
  1. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  2. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 666
  3. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 559
  4. 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 550
  5. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 536
  6. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 368
  7. 2024VideoAgent: Long-form Video Understanding with Large Language Model as Agentcited by 323
  8. 2024What matters when building vision-language models?cited by 321
  9. 2024VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videoscited by 242
  10. 2023Valley: Video Assistant with Large Language model Enhanced abilitYcited by 231
  11. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-trainingcited by 230
  12. 2023Self-Chained Image-Language Model for Video Localization and Question Answeringcited by 200
  13. 2022Revisiting the "Video" in Video-Language Understandingcited by 167
  14. 2024LongVLM: Efficient Long Video Understanding via Large Language Modelscited by 146
  15. 2024Tarsier: Recipes for Training and Evaluating Large Video Description Modelscited by 140
  16. 2024SlowFast-LLaVA: A Strong Training-Free Baseline for Video Large Language Modelscited by 123
  17. 2022TS2-Net: Token Shift and Selection Transformer for Text-Video Retrievalcited by 104
  18. 2024Too Many Frames, Not All Useful: Efficient Strategies for Long-Form Video QAcited by 68
  19. 2024Koala: Key frame-conditioned long video-LLMcited by 57
  20. 2022MIST: Multi-modal Iterative Spatial-Temporal Transformer for Long-form Video Question Answeringcited by 55
  21. 2023ViLA: Efficient Video-Language Alignment for Video Question Answeringcited by 23
  22. 2022Video Moment Retrieval from Text Queries via Single Frame Annotationcited by 19
  23. 2022Locate before Answering: Answer Guided Question Localization for Video Question Answeringcited by 19
  24. 2024End-to-End Video Question Answering with Frame Scoring Mechanisms and Adaptive Samplingcited by 19
  25. 2024POINTS: Improving Your Vision-language Model with Affordable Strategiescited by 17
  26. 2023Semi-Parametric Video-Grounded Text Generationcited by 13
  27. 2023Video-Teller: Enhancing Cross-Modal Generation with Fusion and Decouplingcited by 12
  28. 2022LGDN: Language-Guided Denoising Network for Video-Language Modelingcited by 11
  29. 2022Contrastive Video-Language Learning with Fine-grained Frame Samplingcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 31; 2026: 56.

87 papers.

Sort:

By area:

  1. 2025MemER: Scaling Up Memory for Robot Control via Experience Retrievalcited by 64
  2. 2025StreamMem: Query-Agnostic KV Cache Memory for Streaming Video Understandingcited by 53
  3. 2025Chain-of-Frames: Advancing Video Understanding in Multimodal LLMs via Frame-Aware Reasoningcited by 36
  4. 2025FOCUS: Efficient Keyframe Selection for Long Video Understandingcited by 30
  5. 2025StreamAgent: Towards Anticipatory Agents for Streaming Video Understandingcited by 29
  6. 2025Universal Video Temporal Grounding with Generative Multi-modal Large Language Modelscited by 26
  7. 2025TSPO: Temporal Sampling Policy Optimization for Long-form Video Language Understandingcited by 23
  8. 2025VideoLucy: Deep Memory Backtracking for Long Video Understandingcited by 21
  9. 2025Towards Embodied Agentic AI: Review and Classification of LLM- and VLM-Driven Robot Autonomy and Interactioncited by 18
  10. 2025ReFoCUS: Reinforcement-guided Frame Optimization for Contextual Understandingcited by 15
  11. 2025Less Is More, but Where? Dynamic Token Compression via LLM-Guided Keyframe Priorcited by 15
  12. 2026LensWalk: Agentic Video Understanding by Planning How You See in Videoscited by 14
38 more
  1. 2025See&Trek: Training-Free Spatial Prompting for Multimodal Large Language Modelcited by 13
  2. 2025ViaRL: Adaptive Temporal Grounding via Visual Iterated Amplification Reinforcement Learningcited by 11
  3. 2025K-frames: Scene-Driven Any-k Keyframe Selection for long video understandingcited by 11
  4. 2025Threading Keyframe with Narratives: MLLMs as Strong Long Video Comprehenderscited by 10
  5. 2025Tool-Augmented Spatiotemporal Reasoning for Streamlining Video Question Answering Taskcited by 10
  6. 2025AdaRD-key: Adaptive Relevance-Diversity Keyframe Sampling for Long-form Video understandingcited by 9
  7. 2025Video-LMM Post-Training: A Deep Dive into Video Reasoning with Large Multimodal Modelscited by 8
  8. 2025Zoom-Zero: Reinforced Coarse-to-Fine Video Understanding via Temporal Zoom-incited by 8
  9. 2026Hierarchical Long Video Understanding with Audiovisual Entity Cohesion and Agentic Searchcited by 8
  10. 2026Efficient Inference for Large Vision-Language Models: Bottlenecks, Techniques, and Prospectscited by 8
  11. 2025FrameOracle: Learning What to See and How Much to See in Videoscited by 7
  12. 2025From Frames to Clips: Training-free Adaptive Key Clip Selection for Long-Form Video Understandingcited by 6
  13. 2026MSJoE: Jointly Evolving MLLM and Sampler for Efficient Long-Form Video Understandingcited by 6
  14. 2026CurveStream: Boosting Streaming Video Understanding in MLLMs via Curvature-Aware Hierarchical Visual Memory Managementcited by 5
  15. 2026EVA: Efficient Reinforcement Learning for End-to-End Video Agentcited by 5
  16. 2026Vesta: A Generalist Embodied Reasoning Modelcited by 5
  17. 2025VIBE: Can a VLM Read the Room?cited by 4
  18. 2026StreamingClaw Technical Reportcited by 4
  19. 2026SVAgent: Storyline-Guided Long Video Understanding via Cross-Modal Multi-Agent Collaborationcited by 4
  20. 2025Poisoning Prompt-Guided Sampling in Video Large Language Modelscited by 3
  21. 2025A.I.R.: Enabling Adaptive, Iterative, and Reasoning-based Frame Selection For Video Question Answeringcited by 3
  22. 2025Test-Time Temporal Sampling for Efficient MLLM Video Understandingcited by 3
  23. 2026Stateful Token Reduction for Long-Video Hybrid VLMscited by 3
  24. 2026HiMu: Hierarchical Multimodal Frame Selection for Long Video Question Answeringcited by 3
  25. 2026ViKey: Enhancing Temporal Understanding in Videos via Visual Promptingcited by 3
  26. 2026VideoTIR: Accurate Understanding for Long Videos with Efficient Tool-Integrated Reasoningcited by 3
  27. 2026Query-Conditioned Evidential Keyframe Sampling for MLLM-Based Long-Form Video Understandingcited by 3
  28. 2025VideoCompressa: Data-Efficient Video Understanding via Joint Temporal Compression and Spatial Reconstructioncited by 2
  29. 2026SoPE: Spherical Coordinate-Based Positional Embedding for Enhancing Spatial Perception of 3D LVLMscited by 2
  30. 2026A Multi-Agent Perception-Action Alliance for Efficient Long Video Reasoningcited by 2
  31. 2026LDDR: Linear-DPP-Based Dynamic-Resolution Frame Sampling for Video MLLMscited by 2
  32. 2026PEEK: Picking Essential frames via Efficient Knowledge distillationcited by 2
  33. 2026ReQuest: Rethinking-based Question-Aware Frame Selection for Long-Form Video QAcited by 2
  34. 2026LENS: Adaptive Spatio-Temporal Zooming for Keyframe Sampling in Long-Form Videoscited by 2
  35. 2025VSI: Visual Subtitle Integration for Keyframe Selection to enhance Long Video Understandingcited by 1
  36. 2025TempCore: Are Video QA Benchmarks Temporally Grounded?cited by 1
  37. 2025SPIKE-RL: Video-LLMs meet Bayesian Surprisecited by 1
  38. 2026LFS: Learnable Frame Selector for Event-Aware and Temporally Diverse Video Captioningcited by 1

50 most cited of 87 All 87 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Q-Frame: Query-aware Frame Selection and Multi-Resolution Adaptation for Video-LLMs
  2. 2026Efficient Frame Selection for Long Videos at Test Time with Attention-Based MLLM Selectors
  3. 2025Adaptive Keyframe Sampling for Long Video Understanding
  4. 2026Think-Clip-Sample: Slow-Fast Frame Selection for Video Understanding
  5. 2025BOLT: Boost Large Vision-Language Model Without Training for Long-form Video Understanding
  6. 2026Towards Fast and Effective Long Video Understanding of Multimodal Large Language Models via Adaptive Quasi-Gaussian Sampling
  7. 2026Improving Video Question Answering through query-based frame selection
  8. 2025How Important are Videos for Training Video LLMs?