mlsanityData to 13 Oct 2026

SpecVLM: Enhancing Speculative Decoding of Video LLMs via Verifier-Guided Token Pruning

arXiv 2508.16201 ↗First posted 22 Aug 2025cs.CV cs.AI cs.CLcited by 19

On the map: Efficient LLM inference and serving → Fast autoregressive visual generation

Abstract (arXiv)

Video large language models (Vid-LLMs) have shown strong capabilities in understanding video content. However, their reliance on dense video token representations introduces substantial memory and computational overhead in both prefilling and decoding. To mitigate the information loss of recent video token reduction methods and accelerate the decoding stage of Vid-LLMs losslessly, we introduce SpecVLM, a training-free speculative decoding (SD) framework tailored for Vid-LLMs that incorporates staged video token pruning. Building on our novel finding that the draft model's speculation exhibits low sensitivity to video token pruning, SpecVLM prunes up to 90% of video tokens to enable efficient speculation without sacrificing accuracy. To achieve this, we performs a two-stage pruning process: Stage I selects highly informative tokens guided by attention signals from the verifier (target model), while Stage II prunes remaining redundant ones in a spatially uniform manner. Extensive experiments on four video understanding benchmarks demonstrate the effectiveness and robustness of SpecVLM, which achieves up to 2.68× decoding speedup for LLaVA-OneVision-72B and 2.11× speedup for Qwen2.5-VL-32B. Code is available at https://github.com/zju-jiyicheng/SpecVLM.

Key facts. Cited by 19 papers in the map (data through 13 Oct 2026). Placed in Fast autoregressive visual generation, one of 12 directions in Efficient LLM inference and serving. Builds on 26 of its 44 references that are in the map. First posted on arXiv 22 Aug 2025.

Builds on

26 of 44 references.

References · 26

  1. 2017Attention Is All You Needcited by 60,106
  2. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,692
  3. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,137
  4. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,625
  5. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,392
  6. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 829
  7. 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understandingcited by 653
  8. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 569
  9. 2024SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inferencecited by 381
  10. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Treescited by 362
  11. 2024LongVILA: Scaling Long-Context Visual Language Models for Long Videoscited by 306
  12. 2024PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reductioncited by 284
14 more
  1. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inferencecited by 232
  2. 2024MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understandingcited by 180
  3. 2022Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generationcited by 169
  4. 2023Speculative Decoding with Big Little Decodercited by 162
  5. 2025Adaptive Keyframe Sampling for Long Video Understandingcited by 159
  6. 2023LongCoder: A Long-Range Pre-trained Language Model for Code Completioncited by 111
  7. 2024MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decodingcited by 87
  8. 2024GliDe with a CaPE: A Low-Hassle Method to Accelerate Speculative Decodingcited by 74
  9. 2024SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Accelerationcited by 69
  10. 2024VL-Cache: Sparsity and Modality-Aware KV Cache Compression for Vision-Language Model Inference Accelerationcited by 41
  11. 2024A Stitch in Time Saves Nine: Small VLM is a Precise Guidance for Accelerating Large VLMscited by 35
  12. 2025CriticalKV: Optimizing KV Cache Eviction from an Output Perturbation Perspectivecited by 26
  13. 2025Growing a Multi-head Twig via Distillation and Reinforcement Learning to Accelerate Large Vision-Language Modelscited by 15
  14. 2025FloE: On-the-Fly MoE Inference on Memory-constrained GPUcited by 9

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2025: 1; 2026: 18.

19 papers.

Sort:

By area:

  1. 2026Efficient Inference for Large Vision-Language Models: Bottlenecks, Techniques, and Prospectscited by 8
  2. 2026Draft Less, Retrieve More: Hybrid Tree Construction for Speculative Decodingcited by 6
  3. 2026Vegas: Self-Speculative Decoding with Verification-Guided Sparse Attentioncited by 5
  4. 2026See the Forest for the Trees: Loosely Speculative Decoding via Visual-Semantic Guidance for Efficient Inference of Video LLMscited by 5
  5. 2025SpecVLM: Fast Speculative Decoding in Vision-Language Modelscited by 4
  6. 2026SAGE: Accelerating Vision-Language Models via Entropy-Guided Adaptive Speculative Decodingcited by 3
  7. 2026Sparrow: Text-Anchored Window Attention with Visual-Semantic Glimpsing for Speculative Decoding in Video LLMscited by 3
  8. 2026HIPPO: Accelerating Video Large Language Models Inference via Holistic-aware Parallel Speculative Decodingcited by 2
  9. 2026V-CAST: Video Curvature-Aware Spatio-Temporal Pruning for Efficient Video Large Language Modelscited by 2
  10. 2026CoVSpec: Efficient Device-Edge Co-Inference for Vision-Language Models via Speculative Decodingcited by 2
  11. 2026TailSieve: Partial-Rollout-Guided Tail Routing for LLM Rolloutscited by 2
  12. 2026EchoPrune: Interpreting Redundancy as Temporal Echoes for Efficient VideoLLMscited by 1
7 more
  1. 2026VideoMM: Adaptive Macro-Micro Inference for Efficient Video MLLMscited by 1
  2. 2026ASPIRE: Asynchronous Batched Self-Speculative Decoding for Long-Context LLM Inferencecited by 1
  3. 2026Parallel In-context Learning for Large Vision Language Modelscited by 0
  4. 2026Towards Efficient Large Vision-Language Models: A Comprehensive Survey on Inference Strategiescited by 0
  5. 2026Counterfactual Anatomy-guided Spatial-Temporal Decoding for Annotation-Free Hallucination Mitigation in Medical VLMscited by 0
  6. 2026Is Multimodal Speculative Decoding Ready for Diffusion-Based Parallel Drafting? A Survey and Empirical Diagnosiscited by 0
  7. 2026Vision Is Not Overhead: One-Pass Block Drafting for Lossless Speculative Decoding in Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026ParallelVLM: Lossless Video-LLM Acceleration with Visual Alignment Aware Parallel Speculative Decoding
  2. 2026Sink-Token-Aware Pruning for Fine-Grained Video Understanding in Efficient Video LLMs
  3. 2024DyCoke: Dynamic Compression of Tokens for Fast Video Large Language Models
  4. 2025MMG-Vid: Maximizing Marginal Gains at Segment-level and Token-level for Efficient Video LLMs
  5. 2026TTF: Temporal Token Fusion for Efficient Video-Language Model
  6. 2025Efficient Video Sampling: Pruning Temporally Redundant Tokens for Faster VLM Inference
  7. 2025ShaRP: SHAllow-LayeR Pruning for Efficient Video Large Language Models
  8. 2025AdaTP: Attention-Debiased Token Pruning for Video Large Language Models