# Spec-LLaVA: Accelerating Vision-Language Models with Dynamic Tree-Based Speculative Decoding

arXiv: https://arxiv.org/abs/2509.11961 · HTML: https://mlsanity.com/paper/2509.11961/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning.
- Builds on 28 of its 36 references that are in the map.
- First posted on arXiv 15 Sep 2025.

## Placed in

- LLM reasoning / [Test-time compute and efficient thinking](https://mlsanity.com/lr/test-time-compute-and-efficient-thinking/)
- Multimodal LLMs / vision-language models / [Visual token pruning and efficiency](https://mlsanity.com/mllm/visual-token-pruning-and-efficiency/)
- Efficient LLM inference and serving / [Fast autoregressive visual generation](https://mlsanity.com/inference/fast-autoregressive-visual-generation/)

## Builds on (most cited first, 28 of 28)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [Fast Inference from Transformers via Speculative Decoding](https://mlsanity.com/paper/2211.17192/) (2022)
- [Accelerating Large Language Model Decoding with Speculative Sampling](https://mlsanity.com/paper/2302.01318/) (2023)
- [Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads](https://mlsanity.com/paper/2401.10774/) (2024)
- [EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees](https://mlsanity.com/paper/2406.16858/) (2024)
- [Hydra: Sequentially-Dependent Draft Heads for Medusa Decoding](https://mlsanity.com/paper/2402.05109/) (2024)
- [On-Device Language Models: A Comprehensive Review](https://mlsanity.com/paper/2409.00088/) (2024)
- [Sequoia: Scalable, Robust, and Hardware-aware Speculative Decoding](https://mlsanity.com/paper/2402.12374/) (2024)
- [Consistent Accelerated Inference via Confident Adaptive Transformers](https://mlsanity.com/paper/2104.08803/) (2021)
- [OPT-Tree: Speculative Decoding with Adaptive Draft Tree Structure](https://mlsanity.com/paper/2406.17276/) (2024)
- [PaSS: Parallel Speculative Sampling](https://mlsanity.com/paper/2311.13581/) (2023)
- [MENTOR: Multi-level Self-supervised Learning for Multimodal Recommendation](https://mlsanity.com/paper/2402.19407/) (2024)
- [SETransformer: A Hybrid Attention-Based Architecture for Robust Human Activity Recognition](https://mlsanity.com/paper/2505.19369/) (2025)
- [MonoFusion: Sparse-View 4D Reconstruction via Monocular Fusion](https://mlsanity.com/paper/2507.23782/) (2025)
- [BiTA: Bi-Directional Tuning for Lossless Acceleration in Large Language Models](https://mlsanity.com/paper/2401.12522/) (2024)
- [Human-oriented Representation Learning for Robotic Manipulation](https://mlsanity.com/paper/2310.03023/) (2023)
- [Joint Pedestrian Trajectory Prediction through Posterior Sampling](https://mlsanity.com/paper/2404.00237/) (2024)
- [COHESION: Composite Graph Convolutional Network with Dual-Stage Fusion for Multimodal Recommendation](https://mlsanity.com/paper/2504.04452/) (2025)
- [An Integrated Transportation Distance Between Kernels and Approximate Dynamic Risk Evaluation in Markov Systems](https://mlsanity.com/paper/2311.06645/) (2023)
- [Impact of Noisy Supervision in Foundation Model Learning](https://mlsanity.com/paper/2403.06869/) (2024)
- [AbHE: All Attention-based Homography Estimation](https://mlsanity.com/paper/2212.03029/) (2022)
- [Fast Dual Subgradient Optimization of the Integrated Transportation Distance Between Stochastic Kernels](https://mlsanity.com/paper/2312.01432/) (2023)
- [Don't Lose Yourself: Boosting Multimodal Recommendation via Reducing Node-neighbor Discrepancy in Graph Convolutional Network](https://mlsanity.com/paper/2412.18962/) (2024)
- [Leveraging Optimal Transport for Distributed Two-Sample Testing: An Integrated Transportation Distance-based Framework](https://mlsanity.com/paper/2506.16047/) (2025)
- [Federated Calculation of the Free-Support Transportation Barycenter by Single-Loop Dual Decomposition](https://mlsanity.com/paper/2507.19627/) (2025)

## Built on by (most cited first, 9 of 9)

- [Efficient Inference for Large Vision-Language Models: Bottlenecks, Techniques, and Prospects](https://mlsanity.com/paper/2604.05546/) (2026)
- [HSD: Training-Free Acceleration for Document Parsing Vision-Language Models with Hierarchical Speculative Decoding](https://mlsanity.com/paper/2602.12957/) (2026)
- [See the Forest for the Trees: Loosely Speculative Decoding via Visual-Semantic Guidance for Efficient Inference of Video LLMs](https://mlsanity.com/paper/2604.05650/) (2026)
- [Sparrow: Text-Anchored Window Attention with Visual-Semantic Glimpsing for Speculative Decoding in Video LLMs](https://mlsanity.com/paper/2602.15318/) (2026)
- [AsymSpec: Context-Asymmetric Speculative Decoding for Agentic LLMs](https://mlsanity.com/paper/2608.26004/) (2026)
- [Deltoris: Enabling Real-time VLA Inference in Embodied AI via Bit-level Sparsity and Speculative Inference](https://mlsanity.com/paper/2608.04428/) (2026)
- [Is Multimodal Speculative Decoding Ready for Diffusion-Based Parallel Drafting? A Survey and Empirical Diagnosis](https://mlsanity.com/paper/2608.20743/) (2026)
- [Vision Is Not Overhead: One-Pass Block Drafting for Lossless Speculative Decoding in Vision-Language Models](https://mlsanity.com/paper/2609.00355/) (2026)
- [SP2ec: Adaptive Self-Speculative Decoding for Vision-Language Models](https://mlsanity.com/paper/2610.18735/) (2026)

## Abstract (arXiv)

Vision-Language Models (VLMs) enable powerful multimodal reasoning but suffer from slow autoregressive inference, limiting their deployment in real-time applications. We introduce Spec-LLaVA, a system that applies speculative decoding to accelerate VLMs without sacrificing output quality. Spec-LLaVA pairs a lightweight draft VLM with a large target model: the draft speculates future tokens, which the target verifies in parallel, allowing multiple tokens to be generated per step. To maximize efficiency, we design a dynamic tree-based verification algorithm that adaptively expands and prunes speculative branches using draft model confidence. On MS COCO out-of-domain images, Spec-LLaVA achieves up to 3.28× faster decoding on LLaVA-1.5 (7B, 13B) with no loss in generation quality. This work presents a lossless acceleration framework for VLMs using dynamic tree-structured speculative decoding, opening a path toward practical real-time multimodal assistants. Importantly, the lightweight draft model design makes the framework amenable to resource-constrained or on-device deployment settings.