mlsanityData to 7 Oct 2026

DREAM: Drafting with Refined Target Features and Entropy-Adaptive Cross-Attention Fusion for Multimodal Speculative Decoding

arXiv 2505.19201 ↗First posted 25 May 2025cs.CLcited by 13

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

also in 1 more
Abstract (arXiv)

Speculative decoding (SD) has emerged as a powerful method for accelerating autoregressive generation in large language models (LLMs), yet its integration into vision-language models (VLMs) remains underexplored. We introduce DREAM, a novel speculative decoding framework tailored for VLMs that combines three key innovations: (1) a cross-attention-based mechanism to inject intermediate features from the target model into the draft model for improved alignment, (2) adaptive intermediate feature selection based on attention entropy to guide efficient draft model training, and (3) visual token compression to reduce draft model latency. DREAM enables efficient, accurate, and parallel multimodal decoding with significant throughput improvement. Experiments across a diverse set of recent popular VLMs, including LLaVA, Pixtral, SmolVLM and Gemma3, demonstrate up to 3.6x speedup over conventional decoding and significantly outperform prior SD baselines in both inference throughput and speculative draft acceptance length across a broad range of multimodal benchmarks. The code is publicly available at: https://github.com/SAI-Lab-NYU/DREAM.git

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 57 of its 62 references that are in the map. First posted on arXiv 25 May 2025.

Builds on

57 of 62 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025SmolVLM: Redefining small and efficient multimodal models7 Apr
  2. 2024On Speculative Decoding for Multimodal Large Language Models13 Apr

Other references · 55

  1. 2023Visual Instruction Tuningcited by 7,792
  2. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  3. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  4. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  5. 2015Fast R-CNNcited by 2,930
  6. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  7. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  8. 2025Gemma 3 Technical Reportcited by 1,686
  9. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  10. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  11. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  12. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
38 more
  1. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,059
  2. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 975
  3. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 821
  4. 2024PaliGemma: A versatile 3B VLM for transfercited by 679
  5. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 564
  6. 2023OCRBench: On the Hidden Mystery of OCR in Large Multimodal Modelscited by 466
  7. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Treescited by 362
  8. 2018Blockwise Parallel Decoding for Deep Autoregressive Modelscited by 318
  9. 2025EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Testcited by 318
  10. 2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decodingcited by 302
  11. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decodingcited by 295
  12. 2025Layer by Layer: Uncovering Hidden Representations in Language Modelscited by 293
  13. 2024MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGIcited by 192
  14. 2022Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generationcited by 169
  15. 2024TinyLLaVA: A Framework of Small-scale Large Multimodal Modelscited by 164
  16. 2023DistillSpec: Improving Speculative Decoding via Knowledge Distillationcited by 160
  17. 2023Accelerating LLM Inference with Staged Speculative Decodingcited by 151
  18. 2023REST: Retrieval-Based Speculative Decodingcited by 148
  19. 2023SpecTr: Fast Speculative Decoding via Optimal Transportcited by 129
  20. 2023Accelerating Transformer Inference for Translation via Parallel Decodingcited by 123
  21. 2024Hydra: Sequentially-Dependent Draft Heads for Medusa Decodingcited by 113
  22. 2024Sequoia: Scalable, Robust, and Hardware-aware Speculative Decodingcited by 103
  23. 2024TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decodingcited by 98
  24. 2023SEED-Bench-2: Benchmarking Multimodal Large Language Modelscited by 96
  25. 2022Less is More: Task-aware Layer-wise Distillation for Language Model Compressioncited by 91
  26. 2024PEARL: Parallel Speculative Decoding with Adaptive Draft Lengthcited by 83
  27. 2023TinyGPT-V: Efficient Multimodal Large Language Model via Small Backbonescited by 77
  28. 2024Speculative RAG: Enhancing Retrieval Augmented Generation through Draftingcited by 76
  29. 2024SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Accelerationcited by 69
  30. 2023Inference with Reference: Lossless Acceleration of Large Language Modelscited by 63
  31. 2024Kangaroo: Lossless Self-Speculative Decoding via Double Early Exitingcited by 58
  32. 2020Contrastive Distillation on Intermediate Representations for Language Model Compressioncited by 57
  33. 2023PaSS: Parallel Speculative Samplingcited by 49
  34. 2023SPEED: Speculative Pipelined Execution for Efficient Decodingcited by 47
  35. 2024Speculative Decoding via Early-exiting for Faster LLM Inference with Thompson Sampling Control Mechanismcited by 31
  36. 2025Speculative Decoding and Beyond: An In-Depth Survey of Techniquescited by 22
  37. 2024Lossless Acceleration of Large Language Model via Adaptive N-gram Parallel Decodingcited by 21
  38. 2024Fast and High-Quality Auto-Regressive Speech Synthesis via Speculative Decodingcited by 17

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 11.

13 papers.

Sort:

By area:

  1. 2025Kelle: Co-design KV Caching and eDRAM for Efficient LLM Serving in Edge Computingcited by 5
  2. 2026HSD: Training-Free Acceleration for Document Parsing Vision-Language Models with Hierarchical Speculative Decodingcited by 5
  3. 2025HiViS: Hiding Visual Tokens from the Drafter for Speculative Decoding in Vision-Language Modelscited by 4
  4. 2026Sparrow: Text-Anchored Window Attention with Visual-Semantic Glimpsing for Speculative Decoding in Video LLMscited by 3
  5. 2026Variational Speculative Decoding: Rethinking Draft Training from Token Likelihood to Sequence Acceptancecited by 2
  6. 2026DREAM-S: Speculative Decoding with Searchable Drafting and Target-Aware Refinement for Multimodal Generationcited by 1
  7. 2026DREAM-R: Multimodal Speculative Reasoning with RL-Based Refined Drafting, Precise Verification, and Fully Parallel Executioncited by 0
  8. 2026Is Multimodal Speculative Decoding Ready for Diffusion-Based Parallel Drafting? A Survey and Empirical Diagnosiscited by 0
  9. 2026FOVEA: Focused On-Demand Visual Evidence Adaptation for Cache-Friendly Multimodal Speculative Decodingcited by 0
  10. 2026Vision Is Not Overhead: One-Pass Block Drafting for Lossless Speculative Decoding in Vision-Language Modelscited by 0
  11. 2026Acceptance-Aware Draft Model Training for Speculative Decodingcited by 0
  12. 2026Recovering Off-Policy Supervision for Speculative Decodingcited by 0
1 more
  1. 2026HAWK: Rethinking Multimodal Drafting for Speculative Decodingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025SpecFLASH: A Latent-Guided Semi-autoregressive Speculative Decoding Framework for Efficient Multimodal Generation
  2. 2025SpecVLM: Fast Speculative Decoding in Vision-Language Models
  3. 2026MMSpec: Benchmarking Speculative Decoding for Vision-Language Models
  4. 2025Speculative Decoding Reimagined for Multimodal Large Language Models
  5. 2025Spec-LLaVA: Accelerating Vision-Language Models with Dynamic Tree-Based Speculative Decoding
  6. 2025ViSpec: Accelerating Vision-Language Models with Vision-Aware Speculative Decoding
  7. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding
  8. 2025Cross-Attention Speculative Decoding