mlsanityData to 7 Oct 2026

FOVEA: Focused On-Demand Visual Evidence Adaptation for Cache-Friendly Multimodal Speculative Decoding

arXiv 2608.22883 ↗First posted 24 Aug 2026cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

also in 1 more
Abstract (arXiv)

Multimodal speculative decoding accelerates vision-language models by allowing a lightweight draft model to propose candidate tokens for parallel verification by a larger target model. Existing methods typically condition the drafter on a fixed visual interface, such as a predefined visual-token budget or a static compressed representation. However, our controlled visual-budget analysis shows that visual demand varies substantially across tasks and decoding stages, which means more visual input is not always beneficial. Actually, insufficient evidence may weaken visual grounding, while excessive context adds overhead and may disrupt drafting. We propose FOVEA (Focused On-demand Visual Evidence Adaptation), a cache-friendly approach that builds a reusable visual memory and dynamically retrieves a bounded subset for a draft state. A cumulative-mass rule determines both how many and which entries are selected. The selected entries are aggregated into a visual readout and fused with the current draft hidden state through a lightweight gated residual correction. Rather than inserting visual tokens into the autoregressive context, the correction modifies only the representation passed to the language-model head. Experiments across multiple vision-language backbones and multimodal benchmarks show that FOVEA improves draft acceptance and end-to-end decoding speed, achieving up to 2.13× speedup over autoregressive decoding. These results demonstrate that state-conditioned evidence retrieval is an effective alternative to reusing a fixed visual representation throughout multimodal generation.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 28 of its 42 references that are in the map. First posted on arXiv 24 Aug 2026.

Builds on

28 of 42 references.

Closest ancestors · 10

Same direction, newest first

  1. 2026DREAM-S: Speculative Decoding with Searchable Drafting and Target-Aware Refinement for Multimodal Generation30 May
  2. 2026CoVSpec: Efficient Device-Edge Co-Inference for Vision-Language Models via Speculative Decoding4 May
  3. 2026MMSpec: Benchmarking Speculative Decoding for Vision-Language Models16 Mar
  4. 2025ViSpec: Accelerating Vision-Language Models with Vision-Aware Speculative Decoding17 Sep
  5. 2025DREAM: Drafting with Refined Target Features and Entropy-Adaptive Cross-Attention Fusion for Multimodal Speculative Decoding25 May
  6. 2025LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision Token7 Jan
  7. 2024Dynamic-LLaVA: Efficient Multimodal Large Language Models via Dynamic Vision-language Context Sparsification1 Dec
  8. 2024FocusLLaVA: A Coarse-to-Fine Approach for Efficient and Effective Visual Token Compression21 Nov
  9. 2024VL-Cache: Sparsity and Modality-Aware KV Cache Compression for Vision-Language Model Inference Acceleration29 Oct
  10. 2023MobileVLM : A Fast, Strong and Open Vision Language Assistant for Mobile Devices28 Dec

Other references · 18

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  3. 2019Towards VQA Models That Can Readcited by 1,696
  4. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  5. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  6. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  7. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 975
  8. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 821
  9. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 564
  10. 2023OCRBench: On the Hidden Mystery of OCR in Large Multimodal Modelscited by 466
  11. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Treescited by 362
  12. 2025EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Testcited by 318
6 more
  1. 2023Honeybee: Locality-enhanced Projector for Multimodal LLMcited by 195
  2. 2026DFlash: Block Diffusion for Flash Speculative Decodingcited by 120
  3. 2023Lookahead: An Inference Acceleration Framework for Large Language Model with Lossless Generation Accuracycited by 37
  4. 2025Steering Pretrained Drafters during Speculative Decodingcited by 4
  5. 2025Scaling LLM Speculative Decoding: Non-Autoregressive Forecasting in Large-Batch Scenarioscited by 3
  6. 2026TIGER: Text-Conditioned Visual Gated Routing with Acceptance Alignment for Multimodal Speculative Decodingcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025MASSV: Multimodal Adaptation and Self-Data Distillation for Speculative Decoding of Vision-Language Models
  2. 2025Spec-LLaVA: Accelerating Vision-Language Models with Dynamic Tree-Based Speculative Decoding
  3. 2026HAWK: Rethinking Multimodal Drafting for Speculative Decoding
  4. 2025HiViS: Hiding Visual Tokens from the Drafter for Speculative Decoding in Vision-Language Models
  5. 2025SpecVLM: Fast Speculative Decoding in Vision-Language Models
  6. 2025SpecFLASH: A Latent-Guided Semi-autoregressive Speculative Decoding Framework for Efficient Multimodal Generation
  7. 2026DIVE: Dynamic Iterative Visual Evidence Construction for Efficient Vision-Language Models
  8. 2026VisionPulse: Dynamic Visual Sparsity for Efficient Multimodal Reasoning