mlsanityData to 7 Oct 2026

See the Forest for the Trees: Loosely Speculative Decoding via Visual-Semantic Guidance for Efficient Inference of Video LLMs

arXiv 2604.05650 ↗First posted 7 Apr 2026cs.CLcited by 5

On the map: Efficient LLM inference and serving → Fast autoregressive visual generation

Abstract (arXiv)

Video Large Language Models (Video-LLMs) excel in video understanding but suffer from high inference latency during autoregressive generation. Speculative Decoding (SD) mitigates this by applying a draft-and-verify paradigm, yet existing methods are constrained by rigid exact-match rules, severely limiting the acceleration potential. To bridge this gap, we propose LVSpec, the first training-free loosely SD framework tailored for Video-LLMs. Grounded in the insight that generation is governed by sparse visual-relevant anchors (mandating strictness) amidst abundant visual-irrelevant fillers (permitting loose verification), LVSpec employs a lightweight visual-relevant token identification scheme to accurately pinpoint the former. To further maximize acceptance, we augment this with a position-shift tolerant mechanism that effectively salvages positionally mismatched but semantically equivalent tokens. Experiments demonstrate that LVSpec achieves high fidelity and speed: it preserves >99.8 of target performance while accelerating Qwen2.5-VL-32B by 2.70x and LLaVA-OneVision-72B by 2.94x. Notably, it boosts the mean accepted length and speedup ratio by 136% and 35% compared to SOTA training-free SD methods for Video-LLMs.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Fast autoregressive visual generation, one of 12 directions in Efficient LLM inference and serving. Builds on 37 of its 52 references that are in the map. First posted on arXiv 7 Apr 2026.

Builds on

37 of 52 references.

Closest ancestors · 6

Same direction, newest first

  1. 2026ParallelVLM: Lossless Video-LLM Acceleration with Visual Alignment Aware Parallel Speculative Decoding20 Mar
  2. 2026Sparrow: Text-Anchored Window Attention with Visual-Semantic Glimpsing for Speculative Decoding in Video LLMs17 Feb
  3. 2025FastVLM: Self-Speculative Decoding for Fast Vision-Language Model Inference26 Oct
  4. 2025ViSpec: Accelerating Vision-Language Models with Vision-Aware Speculative Decoding17 Sep
  5. 2025Spec-LLaVA: Accelerating Vision-Language Models with Dynamic Tree-Based Speculative Decoding15 Sep
  6. 2025SpecVLM: Enhancing Speculative Decoding of Video LLMs via Verifier-Guided Token Pruning22 Aug

Other references · 31

  1. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  2. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  3. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  4. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  5. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 821
  6. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 536
  7. 2024SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inferencecited by 378
  8. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Treescited by 362
  9. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verificationcited by 359
  10. 2022Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generationcited by 169
  11. 2023Speculative Decoding with Big Little Decodercited by 163
  12. 2024Video-of-Thought: Step-by-Step Video Reasoning from Perception to Cognitioncited by 151
19 more
  1. 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 137
  2. 2024TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decodingcited by 98
  3. 2024MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decodingcited by 87
  4. 2024PEARL: Parallel Speculative Decoding with Adaptive Draft Lengthcited by 83
  5. 2024GliDe with a CaPE: A Low-Hassle Method to Accelerate Speculative Decodingcited by 74
  6. 2024Learning Harmonized Representations for Speculative Samplingcited by 69
  7. 2024SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Accelerationcited by 69
  8. 2024VideoEspresso: A Large-Scale Chain-of-Thought Dataset for Fine-Grained Video Reasoning via Core Frame Selectioncited by 64
  9. 2025Judge Decoding: Faster Speculative Sampling Requires Going Beyond Model Alignmentcited by 59
  10. 2024Turning Trash into Treasure: Accelerating Inference of Large Language Models with Token Recyclingcited by 48
  11. 2026Double: Breaking the Acceleration Limit via Double Retrieval Speculative Parallelismcited by 15
  12. 2025Fuzzy Speculative Decoding for a Tunable Accuracy-Runtime Tradeoffcited by 14
  13. 2025AutoJudge: Judge Decoding Without Manual Annotationcited by 13
  14. 2026Speculative Speculative Decodingcited by 13
  15. 2025Speeding up Speculative Decoding via Sequential Approximate Verificationcited by 11
  16. 2025Training-Free Loosely Speculative Decoding: Accepting Semantically Correct Drafts Beyond Exact Matchcited by 11
  17. 2025Alignment-Augmented Speculative Decoding with Alignment Sampling and Conditional Verificationcited by 10
  18. 2025Think Before You Accept: Semantic Reflective Verification for Faster Speculative Decodingcited by 6
  19. 2025MMESGBench: Pioneering Multimodal Understanding and Complex Reasoning Benchmark for ESG Taskscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026Draft Less, Retrieve More: Hybrid Tree Construction for Speculative Decodingcited by 5
  2. 2026TailSieve: Partial-Rollout-Guided Tail Routing for LLM Rolloutscited by 2
  3. 2026EchoPrune: Interpreting Redundancy as Temporal Echoes for Efficient VideoLLMscited by 1
  4. 2026MM-ShiftKV: Decode-Aware Prefill-Stage KV Selection for Multimodal Large Language Modelscited by 0
  5. 2026Is Multimodal Speculative Decoding Ready for Diffusion-Based Parallel Drafting? A Survey and Empirical Diagnosiscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026HIPPO: Accelerating Video Large Language Models Inference via Holistic-aware Parallel Speculative Decoding
  2. 2025SpecFLASH: A Latent-Guided Semi-autoregressive Speculative Decoding Framework for Efficient Multimodal Generation
  3. 2025SpecVLM: Fast Speculative Decoding in Vision-Language Models
  4. 2026TIGER: Text-Conditioned Visual Gated Routing with Acceptance Alignment for Multimodal Speculative Decoding
  5. 2025HiViS: Hiding Visual Tokens from the Drafter for Speculative Decoding in Vision-Language Models
  6. 2026SDFP: Speculative Decoding with FIT-Pruned Models for Training-Free and Plug-and-Play LLM Acceleration
  7. 2026Why Is Video Still So Expensive? A Survey of Inference-Efficiency Mechanisms in Video and Audiovisual LLMs
  8. 2025Video Compression Commander: Plug-and-Play Inference Acceleration for Video Large Language Models