mlsanityData to 14 Oct 2026

RDQ: Residual Distribution Quantization for Large Language Models

arXiv 2607.10137 ↗First posted 11 Jul 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Post-training quantization (PTQ) of large language models degrades sharply below 4-bit precision. We identify the root cause as residual stream distributional drift: quantization noise injected at each transformer layer accumulates in the shared residual representation, causing KL divergence from the FP16 baseline to grow super-linearly with depth (Pearson r=0.999 with log-perplexity, p<0.001, confirmed across all tested methods and bit-widths). We discover that 84% of LLaMA-3-8B layers exhibit non-Gaussian residual distributions (KS test, p<=0.05), and that per-layer residual stream variance grows 6,548x across depth. We propose RDQ (Residual Distribution Quantization), a PTQ framework whose central contribution is Cascaded Error Compensation (CEC): a sequential calibration procedure that captures the actual drifted activations each layer receives (computed by running calibration data through already-quantized upstream layers) and fits per-channel AWQ-style scales against those drifted inputs, with scales folded into preceding RMSNorm weights for exact mathematical equivalence at zero inference overhead. RDQ achieves state-of-the-art results on all three tested architectures: LLaMA-3-8B: 7.55 / 5.62 PPL (W3/W4); Qwen-2.5-7B: 7.46 / 6.38 PPL; Mistral-7B: 6.88 / 5.73 PPL. RDQ beats the best published baseline (LeanQuant/SpinQuant) at every model and bit-width combination, with gains up to -46.4% vs. RTN at W3A16 on LLaMA-3-8B. All output is standard group-128 asymmetric quantization, deployable on Qualcomm AIMET, GGUF, and any standard inference stack at zero runtime overhead.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 8 of its 14 references that are in the map. First posted on arXiv 11 Jul 2026.

Builds on

8 of 14 references.

References · 8

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2024Qwen2.5 Technical Reportcited by 4,942
  3. 2023Mistral 7Bcited by 3,164
  4. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  5. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  6. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 634
  7. 2024SpinQuant: LLM quantization with learned rotationscited by 471
  8. 2024LeanQuant: Accurate and Scalable Large Language Model Quantization with Loss-error-aware Gridcited by 22

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Recurrent Residual Quantization: A Progressive Multi-Precision Representation for LLMs
  2. 2025Layer-Wise High-Impact Parameter Ratio Optimization in Post-Training Quantization for Large Language Models
  3. 2024ResQ: Mixed-Precision Quantization of Large Language Models with Low-Rank Residuals
  4. 2025A Comprehensive Evaluation on Quantization Techniques for Large Language Models
  5. 2025DLLMQuant: Quantizing Diffusion-based Large Language Models
  6. 2023RPTQ: Reorder-based Post-training Quantization for Large Language Models
  7. 2025Quantization Meets dLLMs: A Systematic Study of Post-training Quantization for Diffusion LLMs
  8. 2024Post Training Quantization of Large Language Models with Microscaling Formats