mlsanityData to 7 Oct 2026

FastVLM: Self-Speculative Decoding for Fast Vision-Language Model Inference

arXiv 2510.22641 ↗First posted 26 Oct 2025cs.LG cs.AIcited by 1

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

also in 1 more
Abstract (arXiv)

Vision-language Models (VLMs) have made significant strides in visual understanding and query response generation, but often face challenges of high computational cost and inference latency due to autoregressive decoding. In this work, we introduce an imitation-learning-based Self-Speculative Decoding (SSD) framework, named FastVLM, to address these limitations. Our approach employs a lightweight draft model for token generation in an autoregressive manner, while a full model verifies these tokens non-autoregressively. Accepted tokens proceed seamlessly, while rejected tokens are corrected by the full model and used to guide the draft model's refinement. Through an imitation network, FastVLM enhances the draft model by integrating deeper level insights from the full model's architecture. Also, it maintains the performance integrity of the full model while training the draft model, achieving a balance between efficiency and accuracy. Our method speeds up the inference process by 1.55-1.85x as compared to the final layer with minimal loss in performance.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 37 of its 58 references that are in the map. First posted on arXiv 26 Oct 2025.

Builds on

37 of 58 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022EfficientVLM: Fast and Accurate Vision-Language Models via Knowledge Distillation and Modal-adaptive Pruning14 Oct

Other references · 36

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  6. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  7. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  8. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  9. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  10. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  11. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  12. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
24 more
  1. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 975
  2. 2019Are Sixteen Heads Really Better than One?cited by 803
  3. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  4. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  5. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 564
  6. 2018nocaps: novel object captioning at scalecited by 444
  7. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  8. 2016Visual Dialogcited by 377
  9. 2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decodingcited by 302
  10. 2020BERT Loses Patience: Fast and Robust Inference with Early Exitcited by 267
  11. 2020BinaryBERT: Pushing the Limit of BERT Quantizationcited by 175
  12. 2018Auto-Keras: An Efficient Neural Architecture Search Systemcited by 169
  13. 2023Speculative Decoding with Big Little Decodercited by 163
  14. 2020TernaryBERT: Distillation-aware Ultra-low Bit BERTcited by 145
  15. 2020TurboTransformers: An Efficient GPU Serving System For Transformer Modelscited by 71
  16. 2021Zero Time Waste: Recycling Predictions in Early Exit Neural Networkscited by 39
  17. 2018Deep Active Learning with a Neural Architecture Searchcited by 17
  18. 2023SplitEE: Early Exit in Deep Neural Networks with Split Computingcited by 13
  19. 2024CEEBERT: Cross-Domain Inference in Early Exit BERTcited by 12
  20. 2025FREE: Fast and Robust Vision Language Models with Early Exitscited by 12
  21. 2025A Survey of Early Exit Deep Neural Networks in NLPcited by 10
  22. 2025BEEM: Boosting Performance of Early Exit DNNs using Multi-Exit Classifiers as Expertscited by 6
  23. 2024DAdEE: Unsupervised Domain Adaptation in Early Exit PLMscited by 5
  24. 2025Beyond Greedy Exits: Improved Early Exit Decisions for Risk Control and Reliabilitycited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026See the Forest for the Trees: Loosely Speculative Decoding via Visual-Semantic Guidance for Efficient Inference of Video LLMscited by 5

Similar, not cited

Close in wording, no citation either way

  1. 2025SpecVLM: Fast Speculative Decoding in Vision-Language Models
  2. 2025MASSV: Multimodal Adaptation and Self-Data Distillation for Speculative Decoding of Vision-Language Models
  3. 2025Spec-LLaVA: Accelerating Vision-Language Models with Dynamic Tree-Based Speculative Decoding
  4. 2025HiViS: Hiding Visual Tokens from the Drafter for Speculative Decoding in Vision-Language Models
  5. 2026MMSpec: Benchmarking Speculative Decoding for Vision-Language Models
  6. 2025ViSpec: Accelerating Vision-Language Models with Vision-Aware Speculative Decoding
  7. 2025Spec-VLA: Speculative Decoding for Vision-Language-Action Models with Relaxed Acceptance
  8. 2026CoVSpec: Efficient Device-Edge Co-Inference for Vision-Language Models via Speculative Decoding