mlsanityData to 14 Oct 2026

ML-SpecQD: Multi-Level Speculative Decoding with Quantized Drafts

arXiv 2503.13565 ↗First posted 17 Mar 2025cs.CL cs.AI cs.LGcited by 9

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

Speculative decoding (SD) has emerged as a method to accelerate LLM inference without sacrificing any accuracy over the 16-bit model inference. In a typical SD setup, the idea is to use a full-precision, small, fast model as "draft" to generate the next few tokens and use the "target" large model to verify the draft-generated tokens. The efficacy of this method heavily relies on the acceptance ratio of the draft-generated tokens and the relative token throughput of the draft versus the target model. Nevertheless, an efficient SD pipeline requires pre-training and aligning the draft model to the target model, making it impractical for LLM inference in a plug-and-play fashion. In this work, we propose using MXFP4 models as drafts in a plug-and-play fashion since the MXFP4 Weight-Only-Quantization (WOQ) merely direct-casts the BF16 target model weights to MXFP4. In practice, our plug-and-play solution gives speedups up to 2x over the BF16 baseline. Then we pursue an opportunity for further acceleration: the MXFP4 draft token generation itself can be accelerated via speculative decoding by using yet another smaller draft. We call our method ML-SpecQD: Multi-Level Speculative Decoding with Quantized Drafts since it recursively applies speculation for accelerating the draft-token generation. Combining Multi-Level Speculative Decoding with MXFP4 Quantized Drafts we outperform state-of-the-art speculative decoding, yielding speedups up to 2.72x over the BF16 baseline.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 44 of its 52 references that are in the map. First posted on arXiv 17 Mar 2025.

Builds on

44 of 52 references.

Closest ancestors · 17

Same direction, newest first

  1. 2025QuantSpec: Self-Speculative Decoding with Hierarchical Quantized KV Cache5 Feb
  2. 2024FastDraft: How to Train Your Draft17 Nov
  3. 2024QSpec: Speculative Decoding with Complementary Quantization Schemes15 Oct
  4. 2024MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decoding20 Aug
  5. 2024TurboSpec: Closed-loop Speculation Control System for Optimizing LLM Serving Goodput20 Jun
  6. 2024TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding18 Apr
  7. 2024Speculative Streaming: Fast LLM Inference without Auxiliary Models16 Feb
  8. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty26 Jan
  9. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads19 Jan
  10. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding15 Jan
  11. 2023Cascade Speculative Drafting for Even Faster LLM Inference18 Dec
  12. 2023The Synergy of Speculative Decoding and Batching in Serving Large Language Models28 Oct
5 more
  1. 2023Accelerating LLM Inference with Staged Speculative Decoding8 Aug
  2. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification16 May
  3. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb
  4. 2022Fast Inference from Transformers via Speculative Decoding30 Nov
  5. 2022Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation30 Mar

Other references · 27

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  7. 2021Program Synthesis with Large Language Modelscited by 3,637
  8. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  9. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  10. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  11. 2024Qwen2.5-Coder Technical Reportcited by 1,203
  12. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
15 more
  1. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 880
  2. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 414
  3. 2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decodingcited by 307
  4. 2024QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Servingcited by 208
  5. 2024DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMscited by 167
  6. 2023Microscaling Data Formats for Deep Learningcited by 152
  7. 2022Efficient Methods for Natural Language Processing: A Surveycited by 78
  8. 2023HAGRID: A Human-LLM Collaborative Dataset for Generative Information-Seeking with Attributioncited by 56
  9. 2025ParetoQ: Improving Scaling Laws in Extremely Low-bit LLM Quantizationcited by 46
  10. 2023Efficient LLM Inference on CPUscited by 24
  11. 2024AMXFP4: Taming Activation Outliers with Asymmetric Microscaling Floating-Point for 4-bit LLM Inferencecited by 19
  12. 2021Tensor Processing Primitives: A Programming Abstraction for Efficiency and Portability in Deep Learning & HPC Workloadscited by 9
  13. 2024Nanoscaling Floating-Point (NxFP): NanoMantissa, Adaptive Microexponents, and Code Recycling for Direct-Cast Compression of Large Language Modelscited by 7
  14. 2023Harnessing Deep Learning and HPC Kernels via High-Level Loop and Tensor Abstractions on CPU Architecturescited by 5
  15. 2024Instruction-Guided Editing Controls for Images and Multimedia: A Survey in LLM eracited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 5.

9 papers.

  1. 2025Training-Free Loosely Speculative Decoding: Accepting Semantically Correct Drafts Beyond Exact Matchcited by 10
  2. 2025CAS-Spec: Cascade Adaptive Self-Speculative Decoding for On-the-Fly Lossless Inference Acceleration of LLMscited by 4
  3. 2025Speculative Decoding Meets Quantization: Compatibility Evaluation and Hierarchical Framework Designcited by 3
  4. 2025Pushing the Envelope of LLM Inference with Ultra-Low-Bit Quantized Modelscited by 1
  5. 2026Extreme Low-Bit Inference in Reasoning Models: Failure Modes and Targeted Recoverycited by 1
  6. 2026ELMoE-3D: Leveraging Intrinsic Elasticity of MoE for Hybrid-Bonding-Enabled Self-Speculative Decoding in On-Premises Servingcited by 0
  7. 2026EdgeXpert: An Edge Device for Memory-Efficient LLM Inference with Mixture-of-Experts and Speculative Decodingcited by 0
  8. 2026A Calibrated Instrument for Measuring How Inference Optimizations Affect Output Qualitycited by 0
  9. 2026The Undetected Damage of Quantization on Retrieval and How to Fix Itcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Tutorial Proposal: Speculative Decoding for Efficient LLM Inference
  2. 2024Decoding Speculative Decoding
  3. 2024Speculative Decoding with CTC-based Draft Model for LLM Inference Acceleration
  4. 2025Speculative Decoding: Performance or Illusion?
  5. 2026Faster LLM Inference via Sequential Monte Carlo
  6. 2026Make Every Draft Count: Hidden State based Speculative Decoding
  7. 2025HiSpec: Hierarchical Speculative Decoding for LLMs
  8. 2026Test-Time Speculation