mlsanityData to 7 Oct 2026

QuantSpec: Self-Speculative Decoding with Hierarchical Quantized KV Cache

arXiv 2502.10424 ↗First posted 5 Feb 2025cs.LG cs.AIcited by 26

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

Large Language Models (LLMs) are increasingly being deployed on edge devices for long-context settings, creating a growing need for fast and efficient long-context inference. In these scenarios, the Key-Value (KV) cache is the primary bottleneck in terms of both GPU memory and latency, as the full KV cache must be loaded for each decoding step. While speculative decoding is a widely accepted technique to accelerate autoregressive decoding, existing methods often struggle to achieve significant speedups due to inefficient KV cache optimization strategies and result in low acceptance rates. To address these challenges, we propose a novel self-speculative decoding framework, QuantSpec, where the draft model shares the architecture of the target model but employs a hierarchical 4-bit quantized KV cache and 4-bit quantized weights for acceleration. QuantSpec maintains high acceptance rates (>90%) and reliably provides consistent end-to-end speedups upto ~2.5×, outperforming other self-speculative decoding methods that use sparse KV cache for long-context LLM inference. QuantSpec also reduces the memory requirements by ~ 1.3× compared to these alternatives.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 49 of its 55 references that are in the map. First posted on arXiv 5 Feb 2025.

Builds on

49 of 55 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024QSpec: Speculative Decoding with Complementary Quantization Schemes15 Oct
  2. 2024MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decoding20 Aug
  3. 2024TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding18 Apr
  4. 2024Sequoia: Scalable, Robust, and Hardware-aware Speculative Decoding19 Feb
  5. 2024Speculative Streaming: Fast LLM Inference without Auxiliary Models16 Feb
  6. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty26 Jan
  7. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads19 Jan
  8. 2023Speculative Decoding with Big Little Decoder15 Feb
  9. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb
  10. 2022Fast Inference from Transformers via Speculative Decoding30 Nov

Other references · 39

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  3. 2016Pointer Sentinel Mixture Modelscited by 2,992
  4. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  5. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  6. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 848
  7. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  8. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  9. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  10. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  11. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 507
  12. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencecited by 487
27 more
  1. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 466
  2. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 428
  3. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 403
  4. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 397
  5. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  6. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 360
  7. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 285
  8. 2024SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Accelerationcited by 158
  9. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLMcited by 155
  10. 2024Reducing Transformer Key-Value Cache Size with Cross-Layer Attentioncited by 122
  11. 2024HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughlycited by 119
  12. 2024Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inferencecited by 115
  13. 2024RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrievalcited by 114
  14. 2023Full Stack Optimization of Transformer Inference: a Surveycited by 79
  15. 2023FP8-LM: Training FP8 Large Language Modelscited by 75
  16. 2024LazyLLM: Dynamic Token Pruning for Efficient Long Context LLM Inferencecited by 70
  17. 2024Scaling FP8 training to trillion-token LLMscited by 63
  18. 2024Any-Precision LLM: Low-Cost Deployment of Multiple, Different-Sized LLMscited by 50
  19. 2022Multi-LexSum: Real-World Summaries of Civil Rights Lawsuits at Multiple Granularitiescited by 47
  20. 2024Post-Training Sparse Attention with Double Sparsitycited by 45
  21. 2024COAT: Compressing Optimizer states and Activation for Memory-Efficient FP8 Trainingcited by 32
  22. 2024Jetfire: Efficient and Accurate Transformer Pretraining with INT8 Data Flow and Per-Block Quantizationcited by 30
  23. 2021Transformer Acceleration with Dynamic Sparse Attentioncited by 19
  24. 2024SirLLM: Streaming Infinite Retentive LLMcited by 16
  25. 2024INT-FlashAttention: Enabling Flash Attention for INT8 Quantizationcited by 9
  26. 2024KV Prediction for Improved Time to First Tokencited by 9
  27. 2024Sirius: Contextual Sparsity with Correction for Efficient LLMscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 11; 2026: 15.

26 papers.

Sort:

By area:

  1. 2025CDLM: Consistency Diffusion Language Models For Faster Samplingcited by 20
  2. 2025LongSpec: Long-Context Lossless Speculative Decoding with Efficient Drafting and Verificationcited by 19
  3. 2025Beyond Next-Token Prediction: A Performance Characterization of Diffusion versus Autoregressive Language Modelscited by 13
  4. 2025ML-SpecQD: Multi-Level Speculative Decoding with Quantized Draftscited by 9
  5. 2026Agent Memory Below the Prompt: Persistent Q4 KV Cache for Multi-Agent LLM Inference on Edge Devicescited by 8
  6. 2025Draft-based Approximate Inference for LLMscited by 6
  7. 2026Vegas: Self-Speculative Decoding with Verification-Guided Sparse Attentioncited by 5
  8. 2026VeriCache: Turning Lossy KV Cache into Lossless LLM Inferencecited by 5
  9. 2026EfficientRollout: System-Aware Self-Speculative Decoding for RL Rolloutscited by 5
  10. 2025XQuant: Breaking the Memory Wall for LLM Inference with KV Cache Rematerializationcited by 4
  11. 2025Speculative Decoding Meets Quantization: Compatibility Evaluation and Hierarchical Framework Designcited by 3
  12. 2025textttSPECS: Faster Test-Time Scaling through Speculative Draftscited by 3
14 more
  1. 2026MoSE: Mixture of Slimmable Experts for Efficient and Adaptive Language Modelscited by 3
  2. 2025Arbitrage: Efficient Reasoning via Advantage-Aware Speculationcited by 2
  3. 2025Energy-Efficient Wireless LLM Inference via Uncertainty and Importance-Aware Speculative Decodingcited by 1
  4. 2025Pipeline Parallelism is All You Need for Optimized Early-Exit Based Self-Speculative Decodingcited by 1
  5. 2026SpecKV: Adaptive Speculative Decoding with Compression-Aware Gamma Selectioncited by 0
  6. 2026Cassandra: Enabling Reasoning LLMs at Edge via Self-Speculative Decodingcited by 0
  7. 2026Dustin: Draft-Augmented Sparse Verification for Efficient Long-Context Generation with Speculative Decodingcited by 0
  8. 2026High-accuracy Low-Bit KV-Cache Quantization via Local Distribution Restorationcited by 0
  9. 2026KAP: Bridging the Knowledge Selection-Runtime Consumption Gap in LLM Systemscited by 0
  10. 2026BALANCE: Hybrid Autoregressive-Speculative LLM Inference at the Network Edgecited by 0
  11. 2026Strong Drafts Need Compact Memories: Long-Context Speculative Decoding with Compressed KV Cachecited by 0
  12. 2026A Calibrated Instrument for Measuring How Inference Optimizations Affect Output Qualitycited by 0
  13. 2026How Divergence Becomes Decision Flips in Compressed Language Modelscited by 0
  14. 2026BitNest: Bit-Nested Speculative Decoding for Memory-Efficient LLM Inference Accelerationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026A Sparse Glimpse of the Whole: Train-Free Self-Speculative Decoding
  2. 2026Lynx: Progressive Speculative Quantization for accelerating KV Transfer in Long-Context Inference
  3. 2025CARD: A Cache-Assisted Parallel Speculative Decoding Framework via Query-and-Correct Paradigm for Accelerating LLM Inference
  4. 2025SPIRe: Boosting LLM Inference Throughput with Speculative Decoding
  5. 2026CATS: Cascaded Adaptive Tree Speculation for Memory-Limited LLM Inference Acceleration
  6. 2025PipeSpec: Breaking Stage Dependencies in Hierarchical LLM Decoding
  7. 2026SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding
  8. 2025HiSpec: Hierarchical Speculative Decoding for LLMs