mlsanityData to 14 Oct 2026

SP2ec: Adaptive Self-Speculative Decoding for Vision-Language Models

arXiv 2610.18735 ↗First posted 13 Oct 2026cs.LGcited by 0

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

also in 1 more
Abstract (arXiv)

Speculative Decoding (SD) has demonstrated its efficacy in accelerating large language model inference, but for Vision-Language Models (VLMs), the diverse input modalities make it difficult for a small trained drafter to achieve consistently high throughput. Self-speculative decoding reuses parts of the target model as a drafter which retains the multi-modal capability of the VLMs, yet its efficiency depends on choosing a suitable draft configuration for each prompt. We propose Single-Peak SPeculative decoding (SP2ec), a training-free bandit algorithm that adapts the draft configuration during generation. SP2ec exploits the single-peak throughput pattern observed in existing self-speculative methods, including SWIFT, KnapSpec, SpecVLM, and our simple cosSim. Our algorithmic design explicitly minimizes a novel notion of wall-time regret for each given query, corresponding to the empirical wall-time latency. Theoretical guarantees on the wall-time regret show that SP2ec can exploit the optimal draft configuration, with negligible exploration cost. Extensive experiments on Qwen2.5-VL-7B-Instruct and LLaVA-NeXT-Mistral-7B across visual and textual tasks show consistent improvements of up to 11% over the original self-speculative methods, demonstrating the wide applicability of the proposed SP2ec algorithm.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 66 of its 87 references that are in the map. First posted on arXiv 13 Oct 2026.

Builds on

66 of 87 references.

Closest ancestors · 10

Same direction, newest first

  1. 2026Sparrow: Text-Anchored Window Attention with Visual-Semantic Glimpsing for Speculative Decoding in Video LLMs17 Feb
  2. 2026SAGE: Accelerating Vision-Language Models via Entropy-Guided Adaptive Speculative Decoding31 Jan
  3. 2025HiViS: Hiding Visual Tokens from the Drafter for Speculative Decoding in Vision-Language Models28 Sep
  4. 2025ViSpec: Accelerating Vision-Language Models with Vision-Aware Speculative Decoding17 Sep
  5. 2025Spec-LLaVA: Accelerating Vision-Language Models with Dynamic Tree-Based Speculative Decoding15 Sep
  6. 2025Spec-VLA: Speculative Decoding for Vision-Language-Action Models with Relaxed Acceptance30 Jul
  7. 2025DREAM: Drafting with Refined Target Features and Entropy-Adaptive Cross-Attention Fusion for Multimodal Speculative Decoding25 May
  8. 2025Speculative Decoding Reimagined for Multimodal Large Language Models20 May
  9. 2025MASSV: Multimodal Adaptation and Self-Data Distillation for Speculative Decoding of Vision-Language Models15 May
  10. 2024On Speculative Decoding for Multimodal Large Language Models13 Apr

Other references · 56

  1. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  2. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,413
  3. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,109
  4. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 1,006
  5. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 843
  6. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 578
  7. 2021ARKitScenes: A Diverse Real-World Dataset For 3D Indoor Scene Understanding Using Mobile RGB-D Datacited by 454
  8. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Treescited by 370
  9. 2025EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Testcited by 331
  10. 2018Blockwise Parallel Decoding for Deep Autoregressive Modelscited by 323
  11. 2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decodingcited by 307
  12. 2024MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understandingcited by 182
38 more
  1. 2022Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generationcited by 171
  2. 2023Speculative Decoding with Big Little Decodercited by 164
  3. 2023REST: Retrieval-Based Speculative Decodingcited by 150
  4. 2024Hydra: Sequentially-Dependent Draft Heads for Medusa Decodingcited by 117
  5. 2023Online Speculative Decodingcited by 113
  6. 2024MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decodingcited by 91
  7. 2024PEARL: Parallel Speculative Decoding with Adaptive Draft Lengthcited by 85
  8. 2024SpecDec++: Boosting Speculative Decoding via Adaptive Candidate Lengthscited by 75
  9. 2024SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Accelerationcited by 72
  10. 2024OPT-Tree: Speculative Decoding with Adaptive Draft Tree Structurecited by 67
  11. 2023Inference with Reference: Lossless Acceleration of Large Language Modelscited by 65
  12. 2014Lipschitz Bandits: Regret Lower Bounds and Optimal Algorithmscited by 60
  13. 2025SpecPrune-VLA: Accelerating Vision-Language-Action Models via Action-Aware Self-Speculative Pruningcited by 43
  14. 2025DiffuSpec: Unlocking Diffusion Language Models for Speculative Decodingcited by 42
  15. 2014Unimodal Bandits: Regret Lower Bounds and Optimal Algorithmscited by 37
  16. 2024Dynamic Speculation Lookahead Accelerates Speculative Decoding of Large Language Modelscited by 31
  17. 2024A Theoretical Perspective for Speculative Decoding Algorithmcited by 29
  18. 2024Ouroboros: Generating Longer Drafts Phrase by Phrase for Faster Speculative Decodingcited by 25
  19. 2024DySpec: Faster Speculative Decoding with Dynamic Token Tree Structurecited by 25
  20. 2024AdaEDL: Early Draft Stopping for Speculative Decoding of Large Language Models via an Entropy-based Lower Bound on Token Acceptance Probabilitycited by 25
  21. 2024Lossless Acceleration of Large Language Model via Adaptive N-gram Parallel Decodingcited by 23
  22. 2024ParallelSpec: Parallel Drafter for Efficient Speculative Decodingcited by 22
  23. 2025BanditSpec: Adaptive Speculative Decoding via Bandit Algorithmscited by 20
  24. 2025SpecVLM: Enhancing Speculative Decoding of Video LLMs via Verifier-Guided Token Pruningcited by 20
  25. 2024Draft Model Knows When to Stop: Self-Verification Speculative Decoding for Long-Form Generationcited by 16
  26. 2025TETRIS: Optimal Draft Token Selection for Batch Speculative Decodingcited by 15
  27. 2025CLaSp: In-Context Layer Skip for Self-Speculative Decodingcited by 11
  28. 2025Alignment-Augmented Speculative Decoding with Alignment Sampling and Conditional Verificationcited by 10
  29. 2024Adaptive Draft-Verification for Efficient Large Language Model Decodingcited by 9
  30. 2025Small Drafts, Big Verdict: Information-Intensive Visual Reasoning via Speculationcited by 8
  31. 2025Not-a-Bandit: Provably No-Regret Drafter Selection in Speculative Decoding for LLMscited by 7
  32. 2025Token-Driven GammaTune: Adaptive Calibration for Enhanced Speculative Decodingcited by 6
  33. 2026HSD: Training-Free Acceleration for Document Parsing Vision-Language Models with Hierarchical Speculative Decodingcited by 6
  34. 2026KnapSpec: Self-Speculative Decoding via Adaptive Layer Selection as a Knapsack Problemcited by 5
  35. 2026Learning to Draft: Adaptive Speculative Decoding with Reinforcement Learningcited by 5
  36. 2025Semi-Clairvoyant Scheduling of Speculative Decoding Requests to Minimize LLM Inference Latencycited by 4
  37. 2025Sparse-to-Dense: A Free Lunch for Lossless Acceleration of Video Understanding in LLMscited by 4
  38. 2026Multi-Drafter Speculative Decoding with Alignment Feedbackcited by 4

50 most cited of 56 All 56 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026MMSpec: Benchmarking Speculative Decoding for Vision-Language Models
  2. 2024Draft on the Fly: Adaptive Self-Speculative Decoding using Cosine Similarity
  3. 2026Vision Is Not Overhead: One-Pass Block Drafting for Lossless Speculative Decoding in Vision-Language Models
  4. 2025FastVLM: Self-Speculative Decoding for Fast Vision-Language Model Inference
  5. 2025SpecVLM: Fast Speculative Decoding in Vision-Language Models
  6. 2026A Sparse Glimpse of the Whole: Train-Free Self-Speculative Decoding
  7. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding
  8. 2024Closer Look at Efficient Inference Methods: A Survey of Speculative Decoding