# QSpec: Speculative Decoding with Complementary Quantization Schemes

arXiv: https://arxiv.org/abs/2410.11305 · HTML: https://mlsanity.com/paper/2410.11305/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving.
- Builds on 47 of its 66 references that are in the map.
- First posted on arXiv 15 Oct 2024.

## Placed in

- Efficient LLM inference and serving / [Speculative decoding](https://mlsanity.com/inference/speculative-decoding/)

## Builds on (most cited first, 30 of 47)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [Fast Inference from Transformers via Speculative Decoding](https://mlsanity.com/paper/2211.17192/) (2022)
- [SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models](https://mlsanity.com/paper/2211.10438/) (2022)
- [GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints](https://mlsanity.com/paper/2305.13245/) (2023)
- [DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence](https://mlsanity.com/paper/2401.14196/) (2024)
- [Accelerating Large Language Model Decoding with Speculative Sampling](https://mlsanity.com/paper/2302.01318/) (2023)
- [Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads](https://mlsanity.com/paper/2401.10774/) (2024)
- [QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs](https://mlsanity.com/paper/2404.00456/) (2024)
- [KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization](https://mlsanity.com/paper/2401.18079/) (2024)
- [EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty](https://mlsanity.com/paper/2401.15077/) (2024)
- [LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Dataset](https://mlsanity.com/paper/2309.11998/) (2023)
- [OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models](https://mlsanity.com/paper/2308.13137/) (2023)
- [Understanding the planning of LLM agents: A survey](https://mlsanity.com/paper/2402.02716/) (2024)
- [How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources](https://mlsanity.com/paper/2306.04751/) (2023)
- [EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees](https://mlsanity.com/paper/2406.16858/) (2024)
- [SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification](https://mlsanity.com/paper/2305.09781/) (2023)
- [How Language Model Hallucinations Can Snowball](https://mlsanity.com/paper/2305.13534/) (2023)
- [LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding](https://mlsanity.com/paper/2404.16710/) (2024)

## Built on by (most cited first, 18 of 18)

- [MiniCPM4: Ultra-Efficient LLMs on End Devices](https://mlsanity.com/paper/2506.07900/) (2025)
- [QuantSpec: Self-Speculative Decoding with Hierarchical Quantized KV Cache](https://mlsanity.com/paper/2502.10424/) (2025)
- [Speculative Decoding and Beyond: An In-Depth Survey of Techniques](https://mlsanity.com/paper/2502.19732/) (2025)
- [ParoQuant: Pairwise Rotation Quantization for Efficient Reasoning LLM Inference](https://mlsanity.com/paper/2511.10645/) (2025)
- [Draft Model Knows When to Stop: Self-Verification Speculative Decoding for Long-Form Generation](https://mlsanity.com/paper/2411.18462/) (2024)
- [Quamba2: A Robust and Scalable Post-training Quantization Framework for Selective State Space Models](https://mlsanity.com/paper/2503.22879/) (2025)
- [Cognitive Edge Computing: A Comprehensive Survey on Optimizing Large Models and AI Agents for Pervasive Deployment](https://mlsanity.com/paper/2501.03265/) (2025)
- [ML-SpecQD: Multi-Level Speculative Decoding with Quantized Drafts](https://mlsanity.com/paper/2503.13565/) (2025)
- [CAS-Spec: Cascade Adaptive Self-Speculative Decoding for On-the-Fly Lossless Inference Acceleration of LLMs](https://mlsanity.com/paper/2510.26843/) (2025)
- [Mix-Quant: Quantized Prefilling, Precise Decoding for Agentic LLMs](https://mlsanity.com/paper/2605.20315/) (2026)
- [Speculative Decoding Meets Quantization: Compatibility Evaluation and Hierarchical Framework Design](https://mlsanity.com/paper/2505.22179/) (2025)
- [S2-MoE: Enabling Efficient Self-Speculative Decoding for Mixture-of-Experts on Edge Devices](https://mlsanity.com/paper/2608.15018/) (2026)
- [Pipeline Parallelism is All You Need for Optimized Early-Exit Based Self-Speculative Decoding](https://mlsanity.com/paper/2509.19368/) (2025)
- [HiFloat4 Format for End-To-End Reinforcement Learning Post-Training of Large Language Models](https://mlsanity.com/paper/2607.26515/) (2026)
- [A Calibrated Instrument for Measuring How Inference Optimizations Affect Output Quality](https://mlsanity.com/paper/2609.18005/) (2026)
- [The Undetected Damage of Quantization on Retrieval and How to Fix It](https://mlsanity.com/paper/2609.24322/) (2026)
- [How Divergence Becomes Decision Flips in Compressed Language Models](https://mlsanity.com/paper/2610.00694/) (2026)
- [BitNest: Bit-Nested Speculative Decoding for Memory-Efficient LLM Inference Acceleration](https://mlsanity.com/paper/2610.02800/) (2026)

## Abstract (arXiv)

Quantization is widely adopted to accelerate inference and reduce memory consumption in large language models (LLMs). While activation-weight joint quantization enables efficient low-precision decoding, it suffers from substantial performance degradation on multi-step reasoning tasks. We propose QSpec, a novel quantization paradigm that decouples efficiency from quality by integrating two complementary schemes via speculative decoding: low-precision joint quantization for fast drafting and high-precision weight-only quantization for accurate verification. QSpec reuses both weights and KV cache across stages, enabling near-zero-cost switching without retraining or auxiliary models. Compared to high-precision baselines, QSpec achieves up to 1.64x speedup without quality degradation, and outperforms state-of-the-art speculative decoding methods by up to 1.55x in batched settings. Furthermore, QSpec supports plug-and-play deployment and generalizes well across model scales, quantization methods, and workloads. These properties make QSpec a practical and scalable solution for high-fidelity quantized LLM serving under memory-constrained scenarios. Our code is available at https://github.com/hku-netexplo-lab/QSpec.