# On Speculative Decoding for Multimodal Large Language Models

arXiv: https://arxiv.org/abs/2404.08856 · HTML: https://mlsanity.com/paper/2404.08856/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 18 of its 22 references that are in the map.
- First posted on arXiv 13 Apr 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Visual token pruning and efficiency](https://mlsanity.com/mllm/visual-token-pruning-and-efficiency/)
- Efficient LLM inference and serving / [Fast autoregressive visual generation](https://mlsanity.com/inference/fast-autoregressive-visual-generation/)

## Builds on (most cited first, 18 of 18)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](https://mlsanity.com/paper/2304.10592/) (2023)
- [Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering](https://mlsanity.com/paper/2209.09513/) (2022)
- [Fast Inference from Transformers via Speculative Decoding](https://mlsanity.com/paper/2211.17192/) (2022)
- [Accelerating Large Language Model Decoding with Speculative Sampling](https://mlsanity.com/paper/2302.01318/) (2023)
- [Fast Transformer Decoding: One Write-Head is All You Need](https://mlsanity.com/paper/1911.02150/) (2019)
- [Multimodal Few-Shot Learning with Frozen Language Models](https://mlsanity.com/paper/2106.13884/) (2021)
- [Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models](https://mlsanity.com/paper/2311.07919/) (2023)
- [Break the Sequential Dependency of LLM Inference Using Lookahead Decoding](https://mlsanity.com/paper/2402.02057/) (2024)
- [SpecTr: Fast Speculative Decoding via Optimal Transport](https://mlsanity.com/paper/2310.15141/) (2023)
- [Accelerating Transformer Inference for Translation via Parallel Decoding](https://mlsanity.com/paper/2305.10427/) (2023)
- [Weight subcloning: direct initialization of transformers using larger pretrained ones](https://mlsanity.com/paper/2312.09299/) (2023)
- [Recursive Speculative Decoding: Accelerating LLM Inference via Sampling Without Replacement](https://mlsanity.com/paper/2402.14160/) (2024)
- [Direct Alignment of Draft Model for Speculative Decoding with Chat-Fine-Tuned LLMs](https://mlsanity.com/paper/2403.00858/) (2024)

## Built on by (most cited first, 25 of 25)

- [DivPrune: Diversity-based Visual Token Pruning for Large Multimodal Models](https://mlsanity.com/paper/2503.02175/) (2025)
- [Efficient Multimodal Large Language Models: A Survey](https://mlsanity.com/paper/2405.10739/) (2024)
- [LANTERN: Accelerating Visual Autoregressive Models with Relaxed Speculative Decoding](https://mlsanity.com/paper/2410.03355/) (2024)
- [A Theoretical Perspective for Speculative Decoding Algorithm](https://mlsanity.com/paper/2411.00841/) (2024)
- [Speculative Decoding and Beyond: An In-Depth Survey of Techniques](https://mlsanity.com/paper/2502.19732/) (2025)
- [ViSpec: Accelerating Vision-Language Models with Vision-Aware Speculative Decoding](https://mlsanity.com/paper/2509.15235/) (2025)
- [DREAM: Drafting with Refined Target Features and Entropy-Adaptive Cross-Attention Fusion for Multimodal Speculative Decoding](https://mlsanity.com/paper/2505.19201/) (2025)
- [VOCABTRIM: Vocabulary Pruning for Efficient Speculative Decoding in LLMs](https://mlsanity.com/paper/2506.22694/) (2025)
- [HSD: Training-Free Acceleration for Document Parsing Vision-Language Models with Hierarchical Speculative Decoding](https://mlsanity.com/paper/2602.12957/) (2026)
- [MASSV: Multimodal Adaptation and Self-Data Distillation for Speculative Decoding of Vision-Language Models](https://mlsanity.com/paper/2505.10526/) (2025)
- [HiViS: Hiding Visual Tokens from the Drafter for Speculative Decoding in Vision-Language Models](https://mlsanity.com/paper/2509.23928/) (2025)
- [SpecVLM: Fast Speculative Decoding in Vision-Language Models](https://mlsanity.com/paper/2509.11815/) (2025)
- [OTPrune: Distribution-Aligned Visual Token Pruning via Optimal Transport](https://mlsanity.com/paper/2602.20205/) (2026)
- [SpecFLASH: A Latent-Guided Semi-autoregressive Speculative Decoding Framework for Efficient Multimodal Generation](https://mlsanity.com/paper/2505.12728/) (2025)
- [GreedyPrune: Retenting Critical Visual Token Set for Large Vision Language Models](https://mlsanity.com/paper/2506.13166/) (2025)
- [Fast Inference of Visual Autoregressive Model with Adjacency-Adaptive Dynamical Draft Trees](https://mlsanity.com/paper/2512.21857/) (2025)
- [Parallel In-context Learning for Large Vision Language Models](https://mlsanity.com/paper/2603.16092/) (2026)
- [Towards Efficient Large Vision-Language Models: A Comprehensive Survey on Inference Strategies](https://mlsanity.com/paper/2603.27960/) (2026)
- [DREAM-R: Multimodal Speculative Reasoning with RL-Based Refined Drafting, Precise Verification, and Fully Parallel Execution](https://mlsanity.com/paper/2605.28678/) (2026)
- [Beyond Tokens: A Survey on Decoding Methods for Large Language and Vision-Language Models](https://mlsanity.com/paper/2608.14797/) (2026)
- [Algorithm-Architecture Co-Design for Efficient VLA Inference via Speculative Inference and Verification](https://mlsanity.com/paper/2608.15636/) (2026)
- [Is Multimodal Speculative Decoding Ready for Diffusion-Based Parallel Drafting? A Survey and Empirical Diagnosis](https://mlsanity.com/paper/2608.20743/) (2026)
- [Vision Is Not Overhead: One-Pass Block Drafting for Lossless Speculative Decoding in Vision-Language Models](https://mlsanity.com/paper/2609.00355/) (2026)
- [FlowTool: Controlling Tool Parameter in Image Retouching via Flow Matching](https://mlsanity.com/paper/2609.35673/) (2026)
- [SP2ec: Adaptive Self-Speculative Decoding for Vision-Language Models](https://mlsanity.com/paper/2610.18735/) (2026)

## Abstract (arXiv)

Inference with Multimodal Large Language Models (MLLMs) is slow due to their large-language-model backbone which suffers from memory bandwidth bottleneck and generates tokens auto-regressively. In this paper, we explore the application of speculative decoding to enhance the inference efficiency of MLLMs, specifically the LLaVA 7B model. We show that a language-only model can serve as a good draft model for speculative decoding with LLaVA 7B, bypassing the need for image tokens and their associated processing components from the draft model. Our experiments across three different tasks show that speculative decoding can achieve a memory-bound speedup of up to 2.37× using a 115M parameter language model that we trained from scratch. Additionally, we introduce a compact LLaVA draft model incorporating an image adapter, which shows marginal performance gains in image captioning while maintaining comparable results in other tasks.