mlsanityData to 7 Oct 2026

DREAM-S: Speculative Decoding with Searchable Drafting and Target-Aware Refinement for Multimodal Generation

arXiv 2606.00535 ↗First posted 30 May 2026cs.LGcited by 1

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

also in 1 more
Abstract (arXiv)

Speculative decoding (SD) has proven to be an effective technique for accelerating autoregressive generation in large language models (LLMs) however, its application to vision-language models (VLMs) remains relatively unexplored. We propose~textitDREAM-S, a novel SD framework designed specifically for fast and efficient decoding in VLMs. DREAM-S leverages a neural architecture search (NAS) framework with target-aware supernet training to automatically identify both the optimal interaction strategy between the draft and target models, and the most suitable draft model architecture for the underlying hardware implementation platform. DREAM-S additionally incorporates adaptive intermediate feature distillation, guided by attention entropy, to enable efficient draft training. Experiments on a range of well-established VLMs show that DREAM-S achieves up to a 3.85× speedup compared to standard decoding approaches and significantly outperforms existing SD baselines. The code is publicly available at: https://github.com/SAI-Lab-NYU/DREAM-S .

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 40 of its 48 references that are in the map. First posted on arXiv 30 May 2026.

Builds on

40 of 48 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025ViSpec: Accelerating Vision-Language Models with Vision-Aware Speculative Decoding17 Sep
  2. 2025DREAM: Drafting with Refined Target Features and Entropy-Adaptive Cross-Attention Fusion for Multimodal Speculative Decoding25 May
  3. 2025SmolVLM: Redefining small and efficient multimodal models7 Apr

Other references · 37

  1. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  2. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  3. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  4. 2018DARTS: Differentiable Architecture Searchcited by 1,782
  5. 2025Gemma 3 Technical Reportcited by 1,686
  6. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  7. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  8. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 975
  9. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  10. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 821
  11. 2019Are Sixteen Heads Really Better than One?cited by 803
  12. 2019Once-for-All: Train One Network and Specialize it for Efficient Deploymentcited by 778
25 more
  1. 2018ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardwarecited by 666
  2. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 564
  3. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  4. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 408
  5. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Treescited by 362
  6. 2019Dynamic Convolution: Attention over Convolution Kernelscited by 355
  7. 2018Blockwise Parallel Decoding for Deep Autoregressive Modelscited by 318
  8. 2025EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Testcited by 318
  9. 2025Layer by Layer: Uncovering Hidden Representations in Language Modelscited by 293
  10. 2019Progressive Differentiable Architecture Search: Bridging the Depth Gap between Search and Evaluationcited by 230
  11. 2024MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGIcited by 192
  12. 2021VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioningcited by 190
  13. 2022Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generationcited by 169
  14. 2020FBNetV2: Differentiable Neural Architecture Search for Spatial and Channel Dimensionscited by 152
  15. 2024Sequoia: Scalable, Robust, and Hardware-aware Speculative Decodingcited by 103
  16. 2023SEED-Bench-2: Benchmarking Multimodal Large Language Modelscited by 96
  17. 2024Kangaroo: Lossless Self-Speculative Decoding via Double Early Exitingcited by 58
  18. 2020Contrastive Distillation on Intermediate Representations for Language Model Compressioncited by 57
  19. 2024Surgical-LVLM: Learning to Adapt Large Vision-Language Model for Grounded Visual Question Answering in Robotic Surgerycited by 39
  20. 2025Speculative Decoding and Beyond: An In-Depth Survey of Techniquescited by 22
  21. 2024SearchLVLMs: A Plug-and-Play Framework for Augmenting Large Vision-Language Models by Searching Up-to-Date Internet Knowledgecited by 20
  22. 2024VLRM: Vision-Language Models act as Reward Models for Image Captioningcited by 18
  23. 2024Fast and High-Quality Auto-Regressive Speech Synthesis via Speculative Decodingcited by 17
  24. 2024Elevating Visual Perception in Multimodal LLMs with Visual Embedding Distillationcited by 9
  25. 2024Faster Speech-LLaMA Inference with Multi-token Predictioncited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026FOVEA: Focused On-Demand Visual Evidence Adaptation for Cache-Friendly Multimodal Speculative Decodingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025SpecVLM: Fast Speculative Decoding in Vision-Language Models
  2. 2025Spec-LLaVA: Accelerating Vision-Language Models with Dynamic Tree-Based Speculative Decoding
  3. 2026MMSpec: Benchmarking Speculative Decoding for Vision-Language Models
  4. 2025Spec-VLA: Speculative Decoding for Vision-Language-Action Models with Relaxed Acceptance
  5. 2025SpecFLASH: A Latent-Guided Semi-autoregressive Speculative Decoding Framework for Efficient Multimodal Generation
  6. 2025MASSV: Multimodal Adaptation and Self-Data Distillation for Speculative Decoding of Vision-Language Models
  7. 2025Speculative Decoding Reimagined for Multimodal Large Language Models
  8. 2024On Speculative Decoding for Multimodal Large Language Models