Is Multimodal Speculative Decoding Ready for Diffusion-Based Parallel Drafting? A Survey and Empirical Diagnosis
On the map: Efficient LLM inference and serving → Fast autoregressive visual generation
Abstract (arXiv)
Speculative decoding accelerates autoregressive generation by allowing a lightweight drafter to propose future tokens while a target model verifies them in parallel. Its lossless guarantee has motivated a line of work that pushes the drafter itself toward parallel generation. The most recent paradigm is block-parallel generative drafting, including diffusion-based methods such as DFlash and DSpark, achieving up to 3.6x speedup on common daily chatting tasks. While this transition is well studied in text-only LLMs, its applicability to multimodal models remains an open question. Existing multimodal speculative decoding efforts focus on input compression, adapter alignment, candidate coverage, or modality-specific verification; however, block-parallel generative drafting remains largely unexplored. To bridge this gap, this paper combines a modality-centered survey with a cross-architecture empirical study to ask: Is multimodal speculative decoding ready for diffusion-based parallel drafting? In this survey, we systematically analyze a wide spectrum of multimodal models, spanning Vision-Language, Video-Language, Audio, and Vision-Language-Action (VLA) architectures, from the dual perspectives of drafting parallelism and cross-modal information interaction. We introduce a unified taxonomy that isolates drafter-side parallelism from orthogonal design choices such as tree construction and verification strategies. Furthermore, we provide a comprehensive empirical comparison of existing methods under varying degrees of parallelism across standardized multimodal benchmarks, including OCR, VQA, visual reasoning, and image captioning. Finally, we summarize the limitations of current approaches, discuss open challenges, and outline promising future directions for this rapidly evolving field.
Key facts. No citing papers in the map yet (data through 13 Oct 2026). Placed in Fast autoregressive visual generation, one of 12 directions in Efficient LLM inference and serving. Builds on 71 of its 75 references that are in the map. First posted on arXiv 21 Aug 2026.
Builds on
71 of 75 references.
Closest ancestors · 29
Same direction, newest first
- 2026TIGER: Text-Conditioned Visual Gated Routing with Acceptance Alignment for Multimodal Speculative Decoding13 Jul
- 2026SSD: Spatially Speculative Decoding Accelerates Autoregressive Image Generation18 Jun
- 2026Speculative Decoding for Autoregressive Video Generation19 Apr
- 2026See the Forest for the Trees: Loosely Speculative Decoding via Visual-Semantic Guidance for Efficient Inference of Video LLMs7 Apr
- 2026ParallelVLM: Lossless Video-LLM Acceleration with Visual Alignment Aware Parallel Speculative Decoding20 Mar
- 2026MMSpec: Benchmarking Speculative Decoding for Vision-Language Models16 Mar
- 2026Sparrow: Text-Anchored Window Attention with Visual-Semantic Glimpsing for Speculative Decoding in Video LLMs17 Feb
- 2026SAGE: Accelerating Vision-Language Models via Entropy-Guided Adaptive Speculative Decoding31 Jan
- 2026HIPPO: Accelerating Video Large Language Models Inference via Holistic-aware Parallel Speculative Decoding13 Jan
- 2026Multi-Scale Local Speculative Decoding for Image Generation8 Jan
- 2025SJD++: Improved Speculative Jacobi Decoding for Training-free Acceleration of Discrete Auto-regressive Text-to-Image Generation8 Dec
- 2025VVS: Accelerating Speculative Decoding for Visual Autoregressive Generation via Partial Verification Skipping17 Nov
17 more
- 2025HiViS: Hiding Visual Tokens from the Drafter for Speculative Decoding in Vision-Language Models28 Sep
- 2025ViSpec: Accelerating Vision-Language Models with Vision-Aware Speculative Decoding17 Sep
- 2025Spec-LLaVA: Accelerating Vision-Language Models with Dynamic Tree-Based Speculative Decoding15 Sep
- 2025SpecVLM: Fast Speculative Decoding in Vision-Language Models15 Sep
- 2025SpecVLM: Enhancing Speculative Decoding of Video LLMs via Verifier-Guided Token Pruning22 Aug
- 2025Grouped Speculative Decoding for Autoregressive Image Generation11 Aug
- 2025Spec-VLA: Speculative Decoding for Vision-Language-Action Models with Relaxed Acceptance30 Jul
- 2025DREAM: Drafting with Refined Target Features and Entropy-Adaptive Cross-Attention Fusion for Multimodal Speculative Decoding25 May
- 2025Speculative Decoding Reimagined for Multimodal Large Language Models20 May
- 2025SpecFLASH: A Latent-Guided Semi-autoregressive Speculative Decoding Framework for Efficient Multimodal Generation19 May
- 2025MASSV: Multimodal Adaptation and Self-Data Distillation for Speculative Decoding of Vision-Language Models15 May
- 2025LANTERN++: Enhancing Relaxed Speculative Decoding with Static Tree Drafting for Visual Auto-regressive Models10 Feb
- 2024ZipAR: Parallel Auto-regressive Image Generation through Spatial Locality5 Dec
- 2024Continuous Speculative Decoding for Autoregressive Image Generation18 Nov
- 2024LANTERN: Accelerating Visual Autoregressive Models with Relaxed Speculative Decoding4 Oct
- 2024Accelerating Auto-regressive Text-to-Image Generation with Training-free Speculative Jacobi Decoding2 Oct
- 2024On Speculative Decoding for Multimodal Large Language Models13 Apr
Other references · 42
- 2025Qwen3-VL Technical Reportcited by 2,663
- 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,057
- 2019Towards VQA Models That Can Readcited by 1,705
- 2022Fast Inference from Transformers via Speculative Decodingcited by 1,625
- 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,355
- 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,323
- 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 990
- 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 829
- 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 569
- 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Treescited by 362
- 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verificationcited by 361
- 2024Better & Faster Large Language Models via Multi-token Predictioncited by 350
30 more
- 2018Blockwise Parallel Decoding for Deep Autoregressive Modelscited by 319
- 2025EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Testcited by 319
- 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decodingcited by 266
- 2024CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMscited by 231
- 2026DFlash: Block Diffusion for Flash Speculative Decodingcited by 122
- 2026DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generationcited by 39
- 2026Accelerating Speculative Decoding with Block Diffusion Draft Treescited by 28
- 2024ConvBench: A Multi-Turn Conversation Evaluation Benchmark with Hierarchical Capability for Large Vision-Language Modelscited by 24
- 2026Nemotron-Labs-Diffusion: A Tri-Mode Language Model Unifying Autoregressive, Diffusion, and Self-Speculation Decodingcited by 21
- 2024Fast and High-Quality Auto-Regressive Speech Synthesis via Speculative Decodingcited by 17
- 2026SpecForge: A Flexible and Efficient Open-Source Training Framework for Speculative Decodingcited by 16
- 2024Accelerating Codec-based Speech Synthesis with Multi-Token Prediction and Speculative Decodingcited by 10
- 2026KERV: Kinematic-Rectified Speculative Decoding for Embodied VLA Modelscited by 9
- 2026Fast-dVLM: Efficient Block-Diffusion VLM via Direct Conversion from Autoregressive VLMcited by 8
- 2026Realtime-VLA FLASH: Speculative Inference Framework for Diffusion-based VLAscited by 8
- 2026TAPS: Target-Aware Prefix Tree Selection for Diffusion-Drafted Speculative Decodingcited by 8
- 2025Speculative Jacobi-Denoising Decoding for Accelerating Autoregressive Text-to-image Generationcited by 7
- 2025Accelerating Autoregressive Speech Synthesis Inference With Speech Speculative Decodingcited by 6
- 2026HeiSD: Hybrid Speculative Decoding for Embodied Vision-Language-Action Models with Kinematic Awarenesscited by 6
- 2026Open-Loop Planning, Closed-Loop Verification: Speculative Verification for VLAcited by 5
- 2026Fast-dDrive: Efficient Block-Diffusion VLM for Autonomous Drivingcited by 5
- 2025SpecASR: Accelerating LLM-based Automatic Speech Recognition via Speculative Decodingcited by 4
- 2025Speculative Coupled Decoding for Training-Free Lossless Acceleration of Autoregressive Visual Generationcited by 4
- 2025Sparse-to-Dense: A Free Lunch for Lossless Acceleration of Video Understanding in LLMscited by 3
- 2026Chatterbox-Flash: Prior-Calibrated Block Diffusion for Streaming Zero-Shot TTScited by 3
- 2025Principled Coarse-Grained Acceptance for Speculative Decoding in Speechcited by 2
- 2026Self-Speculative Decoding for LLM-based ASR with CTC Encoder Draftscited by 2
- 2026Edge-Cloud Collaborative Speech Emotion Captioning via Token-Level Speculative Decoding in Audio-Language Modelscited by 1
- 2026Reasoning-aware Speculative Decoding for Efficient Vision-Language-Action Models in Autonomous Drivingcited by 1
- 2026ParaASR: Multi-Token Prediction for Fast and Long-Context LLM-Based Speech Recognitioncited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026HAWK: Rethinking Multimodal Drafting for Speculative Decoding
- 2025DiffuSpec: Unlocking Diffusion Language Models for Speculative Decoding
- 2025SpecDiff-2: Scaling Diffusion Drafter Alignment For Faster Speculative Decoding
- 2026Vision Is Not Overhead: One-Pass Block Drafting for Lossless Speculative Decoding in Vision-Language Models
- 2025Fail Fast, Win Big: Rethinking the Drafting Strategy in Speculative Decoding via Diffusion LLMs
- 2026AdaFlash: Adaptive Speculative Decoding via On-Policy Distilled Diffusion Drafters
- 2024ParallelSpec: Parallel Drafter for Efficient Speculative Decoding
- 2024Speculative Diffusion Decoding: Accelerating Language Generation through Diffusion