mlsanityData to 7 Oct 2026

Adaptive Keyframe Sampling for Long Video Understanding

arXiv 2502.21271 ↗First posted 28 Feb 2025cs.CV cs.AI cs.LGcited by 153

On the map: Multimodal LLMs / vision-language models → Video understanding with MLLMs

Abstract (arXiv)

Multimodal large language models (MLLMs) have enabled open-world visual understanding by injecting visual input as extra tokens into large language models (LLMs) as contexts. However, when the visual input changes from a single image to a long video, the above paradigm encounters difficulty because the vast amount of video tokens has significantly exceeded the maximal capacity of MLLMs. Therefore, existing video-based MLLMs are mostly established upon sampling a small portion of tokens from input data, which can cause key information to be lost and thus produce incorrect answers. This paper presents a simple yet effective algorithm named Adaptive Keyframe Sampling (AKS). It inserts a plug-and-play module known as keyframe selection, which aims to maximize the useful information with a fixed number of video tokens. We formulate keyframe selection as an optimization involving (1) the relevance between the keyframes and the prompt, and (2) the coverage of the keyframes over the video, and present an adaptive algorithm to approximate the best solution. Experiments on two long video understanding benchmarks validate that Adaptive Keyframe Sampling improves video QA accuracy (beyond strong baselines) upon selecting informative keyframes. Our study reveals the importance of information pre-filtering in video-based MLLMs. Code is available at https://github.com/ncTimTang/AKS.

Key facts. Cited by 153 papers in the map (data through 7 Oct 2026). Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 50 of its 65 references that are in the map. First posted on arXiv 28 Feb 2025.

Builds on

50 of 65 references.

Closest ancestors · 11

Same direction, newest first

  1. 2024LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding22 Oct
  2. 2024Kangaroo: A Powerful Video-Language Model Supporting Long-context Video Input28 Aug
  3. 2024LongVILA: Scaling Long-Context Visual Language Models for Long Videos19 Aug
  4. 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding22 Jul
  5. 2024Long Context Transfer from Language to Vision24 Jun
  6. 2024Artemis: Towards Referential Understanding in Complex Videos1 Jun
  7. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis31 May
  8. 2024PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioning25 Apr
  9. 2024MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understanding8 Apr
  10. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projection16 Nov
  11. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding5 Jun

Other references · 39

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  7. 2023Segment Anythingcited by 8,158
  8. 2023Visual Instruction Tuningcited by 7,792
  9. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  10. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  11. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  12. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
27 more
  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  2. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  3. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  4. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  5. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  6. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  7. 2024Qwen2 Technical Reportcited by 2,076
  8. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  9. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
  10. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  11. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  12. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 770
  13. 2023VILA: On Pre-training for Visual Language Modelscited by 686
  14. 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 550
  15. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 536
  16. 2024LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Modelscited by 344
  17. 2024VideoAgent: Long-form Video Understanding with Large Language Model as Agentcited by 323
  18. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 300
  19. 2024VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videoscited by 242
  20. 2023Valley: Video Assistant with Large Language model Enhanced abilitYcited by 231
  21. 2023Self-Chained Image-Language Model for Video Localization and Question Answeringcited by 200
  22. 2024Streaming Long Video Understanding with Large Language Modelscited by 164
  23. 2024LongVLM: Efficient Long Video Understanding via Large Language Modelscited by 146
  24. 2023CrossGET: Cross-Guided Ensemble of Tokens for Accelerating Vision-Language Transformerscited by 44
  25. 2024Goldfish: Vision-Language Understanding of Arbitrarily Long Videoscited by 38
  26. 2024ChatterBox: Multi-round Multimodal Referring and Groundingcited by 24
  27. 2021Semantic-Aware Generation for Self-Supervised Visual Representation Learningcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 53; 2026: 100.

153 papers.

Sort:

By area:

  1. 2025StreamMem: Query-Agnostic KV Cache Memory for Streaming Video Understandingcited by 53
  2. 2025LiveVLM: Efficient Online Video Understanding via Streaming-Oriented KV Cache and Retrievalcited by 50
  3. 2025WorldMM: Dynamic Multimodal Memory Agent for Long Video Reasoningcited by 50
  4. 2025Open-o3-Video: Grounded Video Reasoning with Explicit Spatio-Temporal Evidencecited by 43
  5. 2025Generative Frame Sampler for Long Video Understandingcited by 35
  6. 2025MMSI-Video-Bench: A Holistic Benchmark for Video-Based Spatial Intelligencecited by 34
  7. 2026LLaVA-OneVision-2: Towards Next-Generation Perceptual Intelligencecited by 31
  8. 2025FOCUS: Efficient Keyframe Selection for Long Video Understandingcited by 30
  9. 2025VideoITG: Multimodal Video Understanding with Instructed Temporal Groundingcited by 28
  10. 2025Universal Video Temporal Grounding with Generative Multi-modal Large Language Modelscited by 26
  11. 2025Active Video Perception: Iterative Evidence Seeking for Agentic Long Video Understandingcited by 26
  12. 2026FluxMem: Adaptive Hierarchical Memory for Streaming Video Understandingcited by 26
38 more
  1. 2025TSPO: Temporal Sampling Policy Optimization for Long-form Video Language Understandingcited by 23
  2. 2025VideoLucy: Deep Memory Backtracking for Long Video Understandingcited by 21
  3. 2025SpecVLM: Enhancing Speculative Decoding of Video LLMs via Verifier-Guided Token Pruningcited by 19
  4. 2025Less Is More, but Where? Dynamic Token Compression via LLM-Guided Keyframe Priorcited by 15
  5. 2026LensWalk: Agentic Video Understanding by Planning How You See in Videoscited by 14
  6. 2025See&Trek: Training-Free Spatial Prompting for Multimodal Large Language Modelcited by 13
  7. 2025ViaRL: Adaptive Temporal Grounding via Visual Iterated Amplification Reinforcement Learningcited by 11
  8. 2025K-frames: Scene-Driven Any-k Keyframe Selection for long video understandingcited by 11
  9. 2025Divide, then Ground: Adapting Frame Selection to Query Types for Long-Form Video Understandingcited by 11
  10. 2025Threading Keyframe with Narratives: MLLMs as Strong Long Video Comprehenderscited by 10
  11. 2025Video-EM: Event-Centric Episodic Memory for Long-Form Video Understandingcited by 10
  12. 2025Tool-Augmented Spatiotemporal Reasoning for Streamlining Video Question Answering Taskcited by 10
  13. 2025AdaRD-key: Adaptive Relevance-Diversity Keyframe Sampling for Long-form Video understandingcited by 9
  14. 2025Zoom-Zero: Reinforced Coarse-to-Fine Video Understanding via Temporal Zoom-incited by 8
  15. 2026Attend Before Attention: Efficient and Scalable Video Understanding via Autoregressive Gazingcited by 8
  16. 2026ResAdapt: Adaptive Resolution for Efficient Multimodal Reasoningcited by 8
  17. 2026Efficient Inference for Large Vision-Language Models: Bottlenecks, Techniques, and Prospectscited by 8
  18. 2025APVR: Hour-Level Long Video Understanding with Adaptive Pivot Visual Information Retrievalcited by 7
  19. 2025FrameOracle: Learning What to See and How Much to See in Videoscited by 7
  20. 2026VideoBrain: Learning Adaptive Frame Sampling for Long Video Understandingcited by 7
  21. 2026Event-Anchored Frame Selection for Effective Long-Video Understandingcited by 7
  22. 2026Non-Markovian Long-Horizon Robot Manipulation via Keyframe Chainingcited by 7
  23. 2026Scaling the Long Video Understanding of Multimodal Large Language Models via Visual Memory Mechanismcited by 7
  24. 2025From Frames to Clips: Training-free Adaptive Key Clip Selection for Long-Form Video Understandingcited by 6
  25. 2025Agentic Video Intelligence: A Flexible Framework for Advanced Video Exploration and Understandingcited by 6
  26. 2025KFS-Bench: Comprehensive Evaluation of Key Frame Sampling in Long Video Understandingcited by 6
  27. 2026Event-VStream: Event-Driven Real-Time Understanding for Long Video Streamscited by 6
  28. 2026LongVideo-R1: Smart Navigation for Low-cost Long Video Understandingcited by 6
  29. 2026MSJoE: Jointly Evolving MLLM and Sampler for Efficient Long-Form Video Understandingcited by 6
  30. 2026AdaCodec: A Predictive Visual Code for Video MLLMscited by 6
  31. 2025SeViCES: Unifying Semantic-Visual Evidence Consensus for Long Video Understandingcited by 5
  32. 2026CurveStream: Boosting Streaming Video Understanding in MLLMs via Curvature-Aware Hierarchical Visual Memory Managementcited by 5
  33. 2026EVA: Efficient Reinforcement Learning for End-to-End Video Agentcited by 5
  34. 2026VideoStir: Understanding Long Videos via Spatio-Temporally Structured and Intent-Aware RAGcited by 5
  35. 2025Failures to Surface Harmful Contents in Video Large Language Modelscited by 4
  36. 2025Training-free Uncertainty Guidance for Complex Visual Tasks with MLLMscited by 4
  37. 2025video-SALMONN S: Memory-Enhanced Streaming Audio-Visual LLMcited by 4
  38. 2025Venus: An Efficient Edge Memory-and-Retrieval System for VLM-based Online Video Understandingcited by 4

50 most cited of 153 All 153 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Coverage-Driven Adaptive Keyframe Selection for Video Understanding
  2. 2025Q-Frame: Query-aware Frame Selection and Multi-Resolution Adaptation for Video-LLMs
  3. 2025BOLT: Boost Large Vision-Language Model Without Training for Long-form Video Understanding
  4. 2026Think-Clip-Sample: Slow-Fast Frame Selection for Video Understanding
  5. 2026MarKey: Marginal Utility Guided Greedy Keyframe Selection for Long Video Understanding
  6. 2026Efficient Frame Selection for Long Videos at Test Time with Attention-Based MLLM Selectors
  7. 2024TS-LLaVA: Constructing Visual Tokens through Thumbnail-and-Sampling for Training-Free Video Large Language Models
  8. 2025Logic-in-Frames: Dynamic Keyframe Search via Visual Semantic-Logical Verification for Long Video Understanding