mlsanityData to 7 Oct 2026

DyCoke: Dynamic Compression of Tokens for Fast Video Large Language Models

arXiv 2411.15024 ↗First posted 22 Nov 2024cs.CV cs.LGcited by 93

On the map: Efficient LLM inference and serving → Visual token pruning in multimodal models

Abstract (arXiv)

Video large language models (VLLMs) have significantly advanced recently in processing complex video content, yet their inference efficiency remains constrained because of the high computational cost stemming from the thousands of visual tokens generated from the video inputs. We empirically observe that, unlike single image inputs, VLLMs typically attend visual tokens from different frames at different decoding iterations, making a one-shot pruning strategy prone to removing important tokens by mistake. Motivated by this, we present DyCoke, a training-free token compression method to optimize token representation and accelerate VLLMs. DyCoke incorporates a plug-and-play temporal compression module to minimize temporal redundancy by merging redundant tokens across frames, and applies dynamic KV cache reduction to prune spatially redundant tokens selectively. It ensures high-quality inference by dynamically retaining the critical tokens at each decoding step. Extensive experimental results demonstrate that DyCoke can outperform the prior SoTA counterparts, achieving 1.5X inference speedup, 1.4X memory reduction against the baseline VLLM, while still improving the performance, with no training.

Key facts. Cited by 93 papers in the map (data through 7 Oct 2026). Placed in Visual token pruning in multimodal models, one of 12 directions in Efficient LLM inference and serving. Builds on 49 of its 53 references that are in the map. First posted on arXiv 22 Nov 2024.

Builds on

49 of 53 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024TempMe: Video Temporal Token Merging for Efficient Text-Video Retrieval2 Sep
  2. 2024LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models22 Mar
  3. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models11 Mar

Other references · 46

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  6. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  7. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  8. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  9. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  10. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  11. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,305
  12. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
34 more
  1. 2022Token Merging: Your ViT But Fastercited by 995
  2. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  3. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  4. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  5. 2023VILA: On Pre-training for Visual Language Modelscited by 686
  6. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 571
  7. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 559
  8. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 539
  9. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 368
  10. 2024LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Modelscited by 344
  11. 2024PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioningcited by 320
  12. 2023Perception Test: A Diagnostic Benchmark for Multimodal Video Modelscited by 310
  13. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 298
  14. 2023Valley: Video Assistant with Large Language model Enhanced abilitYcited by 231
  15. 2024MobileVLM V2: Faster and Stronger Baseline for Vision Language Modelcited by 172
  16. 2024TinyLLaVA: A Framework of Small-scale Large Multimodal Modelscited by 164
  17. 2024Tarsier: Recipes for Training and Evaluating Large Video Description Modelscited by 140
  18. 2024LLaVA-Phi: Efficient Multi-Modal Assistant with Small Language Modelcited by 128
  19. 2024LITA: Language Instructed Temporal-Localization Assistantcited by 124
  20. 2024Mini-InternVL: A Flexible-Transfer Pocket Multimodal Model with 5% Parameters and 90% Performancecited by 94
  21. 2024LOOK-M: Look-Once Optimization in KV Cache for Efficient Multimodal Long-Context Inferencecited by 93
  22. 2023PB-LLM: Partially Binarized Large Language Modelscited by 80
  23. 2023TinyGPT-V: Efficient Multimodal Large Language Model via Small Backbonescited by 77
  24. 2024LazyLLM: Dynamic Token Pruning for Efficient Long Context LLM Inferencecited by 70
  25. 2024An empirical study of LLaMA3 quantization: from LLMs to MLLMscited by 63
  26. 2023TESTA: Temporal-Spatial Token Aggregation for Long-form Video-Language Understandingcited by 49
  27. 2024MambaLLIE: Implicit Retinex-Aware Low Light Enhancement with Global-then-Local State Spacecited by 48
  28. 2024Small Language Model Meets with Reinforced Vision Vocabularycited by 43
  29. 2024xGen-MM-Vid (BLIP-3-Video): You Only Need 32 Tokens to Represent a Video Even in VLMscited by 34
  30. 2024Effectiveness Assessment of Recent Large Vision-Language Modelscited by 29
  31. 2023RobustMQ: Benchmarking Robustness of Quantized Modelscited by 19
  32. 2023TP2O: Creative Text Pair-to-Object Generation using Balance Swap-Samplingcited by 16
  33. 2023Video-Teller: Enhancing Cross-Modal Generation with Fusion and Decouplingcited by 12
  34. 2024Novel Object Synthesis via Adaptive Text-Image Harmonycited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 36; 2026: 56.

93 papers.

Sort:

By area:

  1. 2025HoliTom: Holistic Token Merging for Fast Video Large Language Modelscited by 58
  2. 2025TopV: Compatible Token Pruning with Inference Time Optimization for Fast and Low-Memory Multimodal Vision Language Modelcited by 57
  3. 2025StreamMem: Query-Agnostic KV Cache Memory for Streaming Video Understandingcited by 53
  4. 2025Token Reduction Should Go Beyond Efficiency in Generative Models -- From Vision, Language to Multimodalitycited by 49
  5. 2025FastVID: Dynamic Density Pruning for Fast Video Large Language Modelscited by 47
  6. 2024AIM: Adaptive Inference of Multi-Modal LLMs via Token Merging and Pruningcited by 43
  7. 2025StreamingTOM: Streaming Token Compression for Efficient Video Understandingcited by 41
  8. 2025Video-MMLU: A Massive Multi-Discipline Lecture Understanding Benchmarkcited by 37
  9. 2026HERMES: KV Cache as Hierarchical Memory for Efficient Streaming Video Understandingcited by 36
  10. 2025Accelerating Streaming Video Large Language Models via Hierarchical Token Compressioncited by 34
  11. 2025Multi-Granular Spatio-Temporal Token Merging for Training-Free Acceleration of Video LLMscited by 30
  12. 2026FluxMem: Adaptive Hierarchical Memory for Streaming Video Understandingcited by 26
38 more
  1. 2025LLaVA-Scissor: Token Compression with Semantic Connected Components for Video LLMscited by 25
  2. 2025Variation-aware Vision Token Dropping for Faster Large Vision-Language Modelscited by 25
  3. 2025VideoARM: Agentic Reasoning over Hierarchical Memory for Long-Form Video Understandingcited by 25
  4. 2025A Survey of Token Compression for Efficient Multimodal Large Language Modelscited by 22
  5. 2026LVOmniBench: Pioneering Long Audio-Video Understanding Evaluation for Omnimodal LLMscited by 22
  6. 2025OmniZip: Audio-Guided Dynamic Token Compression for Fast Omnimodal Large Language Modelscited by 21
  7. 2025A Glimpse to Compress: Dynamic Visual Token Pruning for Large Vision-Language Modelscited by 18
  8. 2025FlexSelect: Flexible Token Selection for Efficient Long Video Understandingcited by 17
  9. 2025Less Is More, but Where? Dynamic Token Compression via LLM-Guided Keyframe Priorcited by 15
  10. 2026FlashVID: Efficient Video Large Language Models via Training-free Tree-based Spatiotemporal Token Mergingcited by 14
  11. 2025Mixing Importance with Diversity: Joint Optimization for KV Cache Compression in Large Vision-Language Modelscited by 13
  12. 2025Plug-and-Play 1.x-Bit KV Cache Quantization for Video Large Language Modelscited by 12
  13. 2025VQToken: Neural Discrete Token Representation Learning for Extreme Token Reduction in Video Large Language Modelscited by 12
  14. 2025MVP: Multiple View Prediction Improves GUI Groundingcited by 10
  15. 2025Poison as Cure: Visual Noise for Mitigating Object Hallucinations in LVMscited by 9
  16. 2025AuroraLong: Bringing RNNs Back to Efficient Open-Ended Video Understandingcited by 9
  17. 2025Breaking the Encoder Barrier for Seamless Video-Language Understandingcited by 8
  18. 2026CoPE-VideoLM: Leveraging Codec Primitives For Efficient Video Language Modelingcited by 8
  19. 2025Beyond Intermediate States: Explaining Visual Redundancy through Languagecited by 7
  20. 2025MergeMix: A Unified Augmentation Paradigm for Visual and Multi-Modal Understandingcited by 7
  21. 2025UniComp: Rethinking Video Compression Through Informational Uniquenesscited by 7
  22. 2025FiLA-Video: Spatio-Temporal Compression for Fine-Grained Long Video Understandingcited by 6
  23. 2025FLoC: Facility Location-Based Efficient Visual Token Compression for Long Video Understandingcited by 6
  24. 2026DASH: Dynamic Audio-Driven Semantic Chunking for Efficient Omnimodal Token Compressioncited by 6
  25. 2026Unified Spatiotemporal Token Compression for Video-LLMs at Ultra-Low Retentioncited by 6
  26. 2026Why and When Visual Token Pruning Fails? A Study on Relevant Visual Information Shift in MLLMs Decodingcited by 6
  27. 2026AdaCodec: A Predictive Visual Code for Video MLLMscited by 6
  28. 2026Token Reduction via Local and Global Contexts Optimization for Efficient Video Large Language Modelscited by 5
  29. 2025SpecVLM: Fast Speculative Decoding in Vision-Language Modelscited by 4
  30. 2026StreamingClaw Technical Reportcited by 4
  31. 2026ForestPrune: High-ratio Visual Token Compression for Video Multimodal Large Language Models via Spatial-Temporal Forest Modelingcited by 4
  32. 2026Towards Fast and Effective Long Video Understanding of Multimodal Large Language Models via Adaptive Quasi-Gaussian Samplingcited by 4
  33. 2025PruneHal: Reducing Hallucinations in Multi-modal Large Language Models through Adaptive KV Cache Pruningcited by 3
  34. 2026Stateful Token Reduction for Long-Video Hybrid VLMscited by 3
  35. 2026ViKey: Enhancing Temporal Understanding in Videos via Visual Promptingcited by 3
  36. 2026Geometry-Guided 3D Visual Token Pruning for Video-Language Modelscited by 3
  37. 2026Keep What Audio Cannot Say: Context-Preserving Token Pruning for Omni-LLMscited by 3
  38. 2026OmniSelect: Dynamic Modality-Aware Token Compression for Efficient Omni-modal Large Language Modelscited by 3

50 most cited of 93 All 93 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Video Compression Commander: Plug-and-Play Inference Acceleration for Video Large Language Models
  2. 2026TTF: Temporal Token Fusion for Efficient Video-Language Model
  3. 2026Rethinking Video Token Compression with a Global Codebook: Learning Once, Compressing Everywhere
  4. 2024Dynamic-VLM: Simple Dynamic Visual Token Compression for VideoLLM
  5. 2025METok: Multi-Stage Event-based Token Compression for Efficient Long Video Understanding
  6. 2026KiToke: Kernel-based Interval-aware Token Compression for Video Large Language Models
  7. 2024VL-Cache: Sparsity and Modality-Aware KV Cache Compression for Vision-Language Model Inference Acceleration
  8. 2026VisCo: Leveraging Large Language Models as Intrinsic Encoders for Visual Token Compression