mlsanityData to 14 Oct 2026

Disaggregated Quantization: Specializing LLM Prefill and Decode

arXiv 2609.26333 ↗First posted 22 Sep 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Prefill and decode reward different approaches to quantization: low-precision arithmetic accelerates prompt processing, while compact weights reduce memory traffic during generation. We propose "disaggregated quantization" (DQ), which specializes computation formats, weights and storage placement to both of these phases. On Qwen 3 and Gemma 3, removing activation quantization specifically on decode improves accuracy on decode-heavy tasks without increasing inference cost. Training separate compute-native prefill weights accelerates prompt processing relative to weight-only inference while matching or exceeding its accuracy at 2-3-bit decode on both decode-heavy and prefill-heavy tasks. With released Qwen3.8-27B GGUF decoders, training an NVFP4 prefiller improves 1-bit accuracy by 32.5 points on MMLU-Pro and 35.3 on MMMU-Pro without modifying the decode checkpoint. To accommodate the additional checkpoint on a single device, offloaded disaggregated prefill (ODP) streams its weights from SSD, amortizing loading over prompt length. On the same 27B model, ODP delivers a 1.78x time-to-first-token speedup over the weight-only baseline at 8K prompt length in llama.cpp. We evaluate accuracy under disaggregated serving in vLLM and further validate shared-weight format disaggregation through post-training quantization on models up to 2.8T parameters.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 35 of its 44 references that are in the map. First posted on arXiv 22 Sep 2026.

Builds on

35 of 44 references.

References · 35

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  6. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  7. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  8. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  9. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  10. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 634
  11. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  12. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 512
23 more
  1. 2024SpinQuant: LLM quantization with learned rotationscited by 471
  2. 2024QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebookscited by 378
  3. 2018Model compression via distillation and quantizationcited by 289
  4. 2024Extreme Compression of Large Language Models via Additive Quantizationcited by 232
  5. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 196
  6. 2024Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloadscited by 121
  7. 2024MARLIN: Mixed-Precision Auto-Regressive Parallel Inference on Large Language Modelscited by 52
  8. 2025ParetoQ: Improving Scaling Laws in Extremely Low-bit LLM Quantizationcited by 46
  9. 2025Bridging the Gap Between Promise and Performance for Microscaling FP4 Quantizationcited by 32
  10. 2025QuEST: Stable Training of LLMs with 1-Bit Weights and Activationscited by 22
  11. 2026Quantization-Aware Distillation for NVFP4 Inference Accuracy Recoverycited by 22
  12. 2024Progressive Mixed-Precision Decoding for Efficient LLM Inferencecited by 18
  13. 2025INT v.s. FP: A Comprehensive Study of Fine-Grained Low-bit Quantization Formatscited by 18
  14. 2026Cross-Model KV Cache Transfer in LLM Families: A Closed-Form Linear Mapping for Prefill Reusecited by 15
  15. 2025Unifying Block-wise PTQ and Distillation-based QAT for Progressive Quantization toward 2-bit Instruction-Tuned LLMscited by 8
  16. 2026When LLMs get significantly worse: A statistical approach to detect model degradationscited by 7
  17. 2026Leech Lattice Vector Quantization for Efficient LLM Compressioncited by 7
  18. 2025Improving Block-Wise LLM Quantization by 4-bit Block-Wise Optimal Float (BOF4): Analysis and Variationscited by 5
  19. 2026Mix-Quant: Quantized Prefilling, Precise Decoding for Agentic LLMscited by 4
  20. 2026Grid Games: The Power of Multiple Grids for Quantizing Large Language Modelscited by 2
  21. 2026When Does Disaggregation Pay? Simulating Prefill--Decode--Attention--FFN Specialization for Agentic LLM Inferencecited by 2
  22. 2026SUN: Shared Use of Next-token Prediction for Efficient Multi-LLM Disaggregated Servingcited by 1
  23. 2026HBM Is Not All You Need: Efficient Disaggregated LLM Serving across Memory-heterogeneous Acceleratorscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026GRINQH: Graded Input-based Quantization Hierarchy for Efficient LLM Generation
  2. 2025SPAD: Specialized Prefill and Decode Hardware for Disaggregated LLM Inference
  3. 2026D2Quant: Accurate Low-bit Post-Training Weight Quantization for LLMs
  4. 2025Prefill-Decode Aggregation or Disaggregation? Unifying Both for Goodput-Optimized LLM Serving
  5. 2026Quantize the Target, Quantize the Drafter: Efficient Inference with Qwen3.5-4B
  6. 2025FGMP: Fine-Grained Mixed-Precision Weight and Activation Quantization for Hardware-Accelerated LLM Inference
  7. 2025HACK: Homomorphic Acceleration via Compression of the Key-Value Cache for Disaggregated LLM Inference
  8. 2025Q-Palette: Fractional-Bit Quantizers Toward Optimal Bit Allocation for Efficient LLM Deployment