mlsanityData to 7 Oct 2026

QSpec: Speculative Decoding with Complementary Quantization Schemes

arXiv 2410.11305 ↗First posted 15 Oct 2024cs.LG cs.AIcited by 18

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

Quantization is widely adopted to accelerate inference and reduce memory consumption in large language models (LLMs). While activation-weight joint quantization enables efficient low-precision decoding, it suffers from substantial performance degradation on multi-step reasoning tasks. We propose QSpec, a novel quantization paradigm that decouples efficiency from quality by integrating two complementary schemes via speculative decoding: low-precision joint quantization for fast drafting and high-precision weight-only quantization for accurate verification. QSpec reuses both weights and KV cache across stages, enabling near-zero-cost switching without retraining or auxiliary models. Compared to high-precision baselines, QSpec achieves up to 1.64x speedup without quality degradation, and outperforms state-of-the-art speculative decoding methods by up to 1.55x in batched settings. Furthermore, QSpec supports plug-and-play deployment and generalizes well across model scales, quantization methods, and workloads. These properties make QSpec a practical and scalable solution for high-fidelity quantized LLM serving under memory-constrained scenarios. Our code is available at https://github.com/hku-netexplo-lab/QSpec.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 47 of its 66 references that are in the map. First posted on arXiv 15 Oct 2024.

Builds on

47 of 66 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024OPT-Tree: Speculative Decoding with Adaptive Draft Tree Structure25 Jun
  2. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees24 Jun
  3. 2024Sequoia: Scalable, Robust, and Hardware-aware Speculative Decoding19 Feb
  4. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty26 Jan
  5. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads19 Jan
  6. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification16 May
  7. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb
  8. 2022Fast Inference from Transformers via Speculative Decoding30 Nov

Other references · 39

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  8. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  9. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  10. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  11. 2021Program Synthesis with Large Language Modelscited by 3,509
  12. 2016Pointer Sentinel Mixture Modelscited by 2,992
27 more
  1. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  2. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  3. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  4. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,154
  5. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 598
  6. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  7. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 449
  8. 2024Understanding the planning of LLM agents: A surveycited by 414
  9. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 405
  10. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 403
  11. 2023How Language Model Hallucinations Can Snowballcited by 310
  12. 2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decodingcited by 302
  13. 2023VeRA: Vector-based Random Matrix Adaptationcited by 294
  14. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Servingcited by 255
  15. 2024QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Servingcited by 200
  16. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLMcited by 155
  17. 2018PipeDream: Fast and Efficient Pipeline Parallel DNN Trainingcited by 116
  18. 2023OWQ: Outlier-Aware Weight Quantization for Efficient Fine-Tuning and Inference of Large Language Modelscited by 113
  19. 2024QAQ: Quality Adaptive Quantization for LLM KV Cachecited by 66
  20. 2024Building Math Agents with Multi-Turn Iterative Preference Learningcited by 66
  21. 2021Subformer: Exploring Weight Sharing for Parameter Efficiency in Generative Transformerscited by 42
  22. 2024WildVis: Open Source Visualizer for Million-Scale Chat Logs in the Wildcited by 20
  23. 2024PRoLoRA: Partial Rotation Empowers More Parameter-Efficient LoRAcited by 16
  24. 2024LLM-PQ: Serving LLM on Heterogeneous Clusters with Phase-Aware Partition and Adaptive Quantizationcited by 14
  25. 2024Can Large Language Models Play Games? A Case Study of A Self-Play Approachcited by 12
  26. 2024MoS: Unleashing Parameter Efficiency of Low-Rank Adaptation with Mixture of Shardscited by 7
  27. 2024QSync: Quantization-Minimized Synchronous Distributed Training Across Hybrid Devicescited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 10; 2026: 7.

18 papers.

Sort:

By area:

  1. 2025MiniCPM4: Ultra-Efficient LLMs on End Devicescited by 51
  2. 2025QuantSpec: Self-Speculative Decoding with Hierarchical Quantized KV Cachecited by 26
  3. 2025Speculative Decoding and Beyond: An In-Depth Survey of Techniquescited by 22
  4. 2025ParoQuant: Pairwise Rotation Quantization for Efficient Reasoning LLM Inferencecited by 16
  5. 2024Draft Model Knows When to Stop: Self-Verification Speculative Decoding for Long-Form Generationcited by 15
  6. 2025Quamba2: A Robust and Scalable Post-training Quantization Framework for Selective State Space Modelscited by 14
  7. 2025Cognitive Edge Computing: A Comprehensive Survey on Optimizing Large Models and AI Agents for Pervasive Deploymentcited by 10
  8. 2025ML-SpecQD: Multi-Level Speculative Decoding with Quantized Draftscited by 9
  9. 2025CAS-Spec: Cascade Adaptive Self-Speculative Decoding for On-the-Fly Lossless Inference Acceleration of LLMscited by 4
  10. 2025Speculative Decoding Meets Quantization: Compatibility Evaluation and Hierarchical Framework Designcited by 3
  11. 2026Mix-Quant: Quantized Prefilling, Precise Decoding for Agentic LLMscited by 3
  12. 2026S2-MoE: Enabling Efficient Self-Speculative Decoding for Mixture-of-Experts on Edge Devicescited by 2
6 more
  1. 2025Pipeline Parallelism is All You Need for Optimized Early-Exit Based Self-Speculative Decodingcited by 1
  2. 2026HiFloat4 Format for End-To-End Reinforcement Learning Post-Training of Large Language Modelscited by 0
  3. 2026A Calibrated Instrument for Measuring How Inference Optimizations Affect Output Qualitycited by 0
  4. 2026The Undetected Damage of Quantization on Retrieval and How to Fix Itcited by 0
  5. 2026How Divergence Becomes Decision Flips in Compressed Language Modelscited by 0
  6. 2026BitNest: Bit-Nested Speculative Decoding for Memory-Efficient LLM Inference Accelerationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026TriSpec: Ternary Speculative Decoding via Lightweight Proxy Verification
  2. 2026SpecQuant: Speculative Decoding with Multi-Parent Quantization for Adaptive LLM Inference
  3. 2025From Quarter to All: Accelerating Speculative LLM Decoding via Floating-Point Exponent Remapping and Parameter Sharing
  4. 2025AdaSPEC: Selective Knowledge Distillation for Efficient Speculative Decoders
  5. 2026SpecForge: A Flexible and Efficient Open-Source Training Framework for Speculative Decoding
  6. 2024SSSD: Simply-Scalable Speculative Decoding
  7. 2026Quasar: Quantized Self-Speculative Acceleration for Rapid Inference via Memory-Efficient Verification
  8. 2025AdaSpec: Adaptive Speculative Decoding for Fast, SLO-Aware Large Language Model Serving