mlsanityData to 14 Oct 2026

Dynamic-VLM: Simple Dynamic Visual Token Compression for VideoLLM

arXiv 2412.09530 ↗First posted 12 Dec 2024cs.CVcited by 32

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

also in 1 more
Abstract (arXiv)

The application of Large Vision-Language Models (LVLMs) for analyzing images and videos is an exciting and rapidly evolving field. In recent years, we've seen significant growth in high-quality image-text datasets for fine-tuning image understanding, but there is still a lack of comparable datasets for videos. Additionally, many VideoLLMs are extensions of single-image VLMs, which may not efficiently handle the complexities of longer videos. In this study, we introduce a large-scale synthetic dataset created from proprietary models, using carefully designed prompts to tackle a wide range of questions. We also explore a dynamic visual token compression architecture that strikes a balance between computational efficiency and performance. Our proposed model achieves state-of-the-art results across various video tasks and shows impressive generalization, setting new baselines in multi-image understanding. Notably, model delivers an absolute improvement of 2.7% over LLaVA-OneVision on VideoMME and 10.7% on MuirBench. Codes are available at https://github.com/Hon-Wong/ByteVideoLLM

Key facts. Cited by 32 papers in the map (data through 14 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 64 of its 78 references that are in the map. First posted on arXiv 12 Dec 2024.

Builds on

64 of 78 references.

References · 64

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  4. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  5. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  6. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  7. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  8. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  9. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  10. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  11. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  12. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,413
38 more
  1. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,402
  2. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  3. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,177
  4. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  5. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,109
  6. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
  7. 2022Token Merging: Your ViT But Fastercited by 1,021
  8. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,005
  9. 2023VideoChat: Chat-Centric Video Understandingcited by 857
  10. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  11. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 771
  12. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 744
  13. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 710
  14. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 680
  15. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 625
  16. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 584
  17. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 561
  18. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 546
  19. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 542
  20. 2024Long Context Transfer from Language to Visioncited by 515
  21. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 497
  22. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 388
  23. 2017TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answeringcited by 378
  24. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 371
  25. 2024TempCompass: Do Video LLMs Really Understand Videos?cited by 363
  26. 2018DVQA: Understanding Data Visualizations via Question Answeringcited by 340
  27. 2015Visual7W: Grounded Question Answering in Imagescited by 318
  28. 2023Perception Test: A Diagnostic Benchmark for Multimodal Video Modelscited by 314
  29. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 302
  30. 2021IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoningcited by 267
  31. 2023Valley: Video Assistant with Large Language model Enhanced abilitYcited by 234
  32. 2021Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptionscited by 226
  33. 2024ALLaVA: Harnessing GPT4V-Synthesized Data for Lite Vision-Language Modelscited by 205
  34. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Outputcited by 195
  35. 2024MuirBench: A Comprehensive Benchmark for Robust Multi-image Understandingcited by 188
  36. 2024VITA: Towards Open-Source Interactive Omni Multimodal LLMcited by 173
  37. 2023Q-Instruct: Improving Low-level Visual Abilities for Multi-modality Foundation Modelscited by 152
  38. 2024Multimodal ArXiv: A Dataset for Improving Scientific Comprehension of Large Vision-Language Modelscited by 139

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 13; 2026: 19.

32 papers.

Sort:

By area:

  1. 2026HERMES: KV Cache as Hierarchical Memory for Efficient Streaming Video Understandingcited by 41
  2. 2025Vision as LoRAcited by 37
  3. 2025A Survey of Token Compression for Efficient Multimodal Large Language Modelscited by 25
  4. 2025VideoMem: Enhancing Ultra-Long Video Understanding via Adaptive Memory Managementcited by 19
  5. 2025Less Is More, but Where? Dynamic Token Compression via LLM-Guided Keyframe Priorcited by 15
  6. 2026Scaling the Long Video Understanding of Multimodal Large Language Models via Visual Memory Mechanismcited by 9
  7. 2026Efficient Inference for Large Vision-Language Models: Bottlenecks, Techniques, and Prospectscited by 9
  8. 2025DynTok: Dynamic Compression of Visual Tokens for Efficient and Effective Video Understandingcited by 7
  9. 2025APVR: Hour-Level Long Video Understanding with Adaptive Pivot Visual Information Retrievalcited by 7
  10. 2025MARC: Memory-Augmented RL Token Compression for Efficient Video Understandingcited by 6
  11. 2025ChineseVideoBench: Benchmarking Multi-modal Large Models for Chinese Video Question Answeringcited by 4
  12. 2026OmniEarth: A Benchmark for Evaluating Vision-Language Models in Geospatial Taskscited by 3
20 more
  1. 2026SAFE-Pruner: Semantic Attention-Guided Future-Aware Token Pruning for Efficient Vision-Language-Action Manipulationcited by 3
  2. 2025Task-Related Token Compression in Multimodal Large Language Models from an Explainability Perspectivecited by 2
  3. 2025SUMMA: A Multimodal Large Language Model for Advertisement Summarizationcited by 2
  4. 2025Towards Effective Long Video Understanding of Multimodal Large Language Models via One-shot Clip Retrievalcited by 2
  5. 2026SoPE: Spherical Coordinate-Based Positional Embedding for Enhancing Spatial Perception of 3D LVLMscited by 2
  6. 2026AdaptToken: Entropy-based Adaptive Token Selection for MLLM Long Video Understandingcited by 2
  7. 2026HieraVid: Hierarchical Token Pruning for Fast Video Large Language Modelscited by 2
  8. 2025Iterative Zoom-In: Temporal Interval Exploration for Long Video Understandingcited by 1
  9. 2025ChronoForge-RL: Chronological Forging through Reinforcement Learning for Enhanced Video Understandingcited by 1
  10. 2026UniCompress: Token Compression for Unified Vision-Language Understanding and Generationcited by 1
  11. 2026Rethinking RAG in Long Videos: What to Retrieve and How to Use It?cited by 1
  12. 2026Adaptive Volumetric Mechanical Property Fields Invariant to Resolutioncited by 1
  13. 2026Baseline-Relative Counterfactual Refinement for Bit-Aware Visual Token Communicationcited by 1
  14. 2026APB-V: Accelerating Long-Video Understanding via Sequence-Parallelism-aware Approximate Attentioncited by 0
  15. 2026Towards Efficient Large Vision-Language Models: A Comprehensive Survey on Inference Strategiescited by 0
  16. 2026GSTEP: Global Spatio-Temporal Density-Driven Visual Token Pruning for Efficient Video Large Language Modelscited by 0
  17. 2026Adaptive Two-Stage Visual Token Pruning for Efficient Inference in Video-Language Modelscited by 0
  18. 2026Multi-Image Visual Token Pruning in Large Visual Language Modelscited by 0
  19. 2026Still There, No Longer Seen: Exposing Compression-Induced Risk in Large Vision-Language Modelscited by 0
  20. 2026Video Codec-Guided Training-Free Token Compression for Video Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Visual Token Coding for Video Multimodal Large Language Models
  2. 2024PVC: Progressive Visual Token Compression for Unified Image and Video Processing in Large Vision-Language Models
  3. 2025DyMU: Dynamic Merging and Virtual Unmerging for Efficient VLMs
  4. 2026Rethinking Visual Token Compression for Video Large Language Models: A Simple Yet Strong Baseline
  5. 2024DyCoke: Dynamic Compression of Tokens for Fast Video Large Language Models
  6. 2026ApET: Approximation-Error Guided Token Compression for Efficient VLMs
  7. 2024FocusLLaVA: A Coarse-to-Fine Approach for Efficient and Effective Visual Token Compression
  8. 2025FCoT-VL:Advancing Text-oriented Large Vision-Language Models with Efficient Visual Token Compression