mlsanityData to 7 Oct 2026

On Speculative Decoding for Multimodal Large Language Models

arXiv 2404.08856 ↗First posted 13 Apr 2024cs.CL cs.AI cs.LGcited by 24

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

also in 1 more
Abstract (arXiv)

Inference with Multimodal Large Language Models (MLLMs) is slow due to their large-language-model backbone which suffers from memory bandwidth bottleneck and generates tokens auto-regressively. In this paper, we explore the application of speculative decoding to enhance the inference efficiency of MLLMs, specifically the LLaVA 7B model. We show that a language-only model can serve as a good draft model for speculative decoding with LLaVA 7B, bypassing the need for image tokens and their associated processing components from the draft model. Our experiments across three different tasks show that speculative decoding can achieve a memory-bound speedup of up to 2.37× using a 115M parameter language model that we trained from scratch. Additionally, we introduce a compact LLaVA draft model incorporating an image adapter, which shows marginal performance gains in image captioning while maintaining comparable results in other tasks.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 18 of its 22 references that are in the map. First posted on arXiv 13 Apr 2024.

Builds on

18 of 22 references.

References · 18

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  6. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  7. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  8. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  9. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 975
  10. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  11. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  12. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 591
6 more
  1. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decodingcited by 295
  2. 2023SpecTr: Fast Speculative Decoding via Optimal Transportcited by 129
  3. 2023Accelerating Transformer Inference for Translation via Parallel Decodingcited by 123
  4. 2023Weight subcloning: direct initialization of transformers using larger pretrained onescited by 30
  5. 2024Recursive Speculative Decoding: Accelerating LLM Inference via Sampling Without Replacementcited by 27
  6. 2024Direct Alignment of Draft Model for Speculative Decoding with Chat-Fine-Tuned LLMscited by 21

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 11; 2026: 10.

24 papers.

Sort:

By area:

  1. 2025DivPrune: Diversity-based Visual Token Pruning for Large Multimodal Modelscited by 177
  2. 2024Efficient Multimodal Large Language Models: A Surveycited by 95
  3. 2024LANTERN: Accelerating Visual Autoregressive Models with Relaxed Speculative Decodingcited by 46
  4. 2024A Theoretical Perspective for Speculative Decoding Algorithmcited by 28
  5. 2025Speculative Decoding and Beyond: An In-Depth Survey of Techniquescited by 22
  6. 2025ViSpec: Accelerating Vision-Language Models with Vision-Aware Speculative Decodingcited by 21
  7. 2025DREAM: Drafting with Refined Target Features and Entropy-Adaptive Cross-Attention Fusion for Multimodal Speculative Decodingcited by 13
  8. 2025VOCABTRIM: Vocabulary Pruning for Efficient Speculative Decoding in LLMscited by 11
  9. 2026HSD: Training-Free Acceleration for Document Parsing Vision-Language Models with Hierarchical Speculative Decodingcited by 5
  10. 2025MASSV: Multimodal Adaptation and Self-Data Distillation for Speculative Decoding of Vision-Language Modelscited by 4
  11. 2025SpecVLM: Fast Speculative Decoding in Vision-Language Modelscited by 4
  12. 2025HiViS: Hiding Visual Tokens from the Drafter for Speculative Decoding in Vision-Language Modelscited by 4
12 more
  1. 2025SpecFLASH: A Latent-Guided Semi-autoregressive Speculative Decoding Framework for Efficient Multimodal Generationcited by 2
  2. 2025GreedyPrune: Retenting Critical Visual Token Set for Large Vision Language Modelscited by 2
  3. 2025Fast Inference of Visual Autoregressive Model with Adjacency-Adaptive Dynamical Draft Treescited by 1
  4. 2026OTPrune: Distribution-Aligned Visual Token Pruning via Optimal Transportcited by 1
  5. 2026Parallel In-context Learning for Large Vision Language Modelscited by 0
  6. 2026Towards Efficient Large Vision-Language Models: A Comprehensive Survey on Inference Strategiescited by 0
  7. 2026DREAM-R: Multimodal Speculative Reasoning with RL-Based Refined Drafting, Precise Verification, and Fully Parallel Executioncited by 0
  8. 2026Beyond Tokens: A Survey on Decoding Methods for Large Language and Vision-Language Modelscited by 0
  9. 2026Algorithm-Architecture Co-Design for Efficient VLA Inference via Speculative Inference and Verificationcited by 0
  10. 2026Is Multimodal Speculative Decoding Ready for Diffusion-Based Parallel Drafting? A Survey and Empirical Diagnosiscited by 0
  11. 2026Vision Is Not Overhead: One-Pass Block Drafting for Lossless Speculative Decoding in Vision-Language Modelscited by 0
  12. 2026FlowTool: Controlling Tool Parameter in Image Retouching via Flow Matchingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Speculative Decoding Reimagined for Multimodal Large Language Models
  2. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding
  3. 2024Closer Look at Efficient Inference Methods: A Survey of Speculative Decoding
  4. 2025Spec-LLaVA: Accelerating Vision-Language Models with Dynamic Tree-Based Speculative Decoding
  5. 2024Towards Fast Multilingual LLM Inference: Speculative Decoding and Specialized Drafters
  6. 2026MMSpec: Benchmarking Speculative Decoding for Vision-Language Models
  7. 2024Decoding Speculative Decoding
  8. 2025Speculative Decoding: Performance or Illusion?