mlsanityData to 7 Oct 2026

DecDEC: A Systems Approach to Advancing Low-Bit LLM Quantization

arXiv 2412.20185 ↗First posted 28 Dec 2024cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

Quantization of Large Language Models (LLMs) has recently gained popularity, particularly for on-device settings with limited hardware resources. While efficient, quantization inevitably degrades model quality, especially in aggressive low-bit settings such as 3-bit and 4-bit precision. In this paper, we propose DecDEC, an inference scheme that improves the quality of low-bit LLMs while preserving the key benefits of quantization: GPU memory savings and latency reduction. DecDEC stores the residual matrix -- the difference between full-precision and quantized weights -- in CPU, and dynamically fetches the residuals for only a small portion of the weights. This portion corresponds to the salient channels, marked by activation outliers, with the fetched residuals helping to correct quantization errors in these channels. Salient channels are identified dynamically at each decoding step by analyzing the input activations -- this enables adaptation to the dynamic nature of activation distribution, thus maximizing the effectiveness of error compensation. We demonstrate the effectiveness of DecDEC by augmenting state-of-the-art quantization methods. For example, DecDEC reduces the perplexity of a 3-bit Llama-3-8B-Instruct model from 10.15 to 9.12 -- outperforming its 3.5-bit counterpart -- while adding less than 0.0003% to GPU memory usage and incurring only a 1.7% inference slowdown on NVIDIA RTX 4050 Mobile.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 50 of its 58 references that are in the map. First posted on arXiv 28 Dec 2024.

Builds on

50 of 58 references.

References · 50

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  6. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  7. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  8. 2016Pointer Sentinel Mixture Modelscited by 2,992
  9. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  10. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  11. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  12. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
38 more
  1. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  2. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  3. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  4. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  5. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  6. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 710
  7. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 643
  8. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  9. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
  10. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  11. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 403
  12. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 397
  13. 2019HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precisioncited by 387
  14. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 359
  15. 2024QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebookscited by 354
  16. 2023LLM-QAT: Data-Free Quantization Aware Training for Large Language Modelscited by 320
  17. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 285
  18. 2020ZeroQ: A Novel Zero Shot Quantization Frameworkcited by 265
  19. 2023A Survey on Model Compression for Large Language Modelscited by 263
  20. 2019HAWQ-V2: Hessian Aware trace-Weighted Quantization of Neural Networkscited by 250
  21. 2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 247
  22. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Managementcited by 242
  23. 2024Extreme Compression of Large Language Models via Additive Quantizationcited by 223
  24. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 210
  25. 2023LLM in a flash: Efficient Large Language Model Inference with Limited Memorycited by 170
  26. 2023ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Modelscited by 156
  27. 2023Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothingcited by 147
  28. 2023OWQ: Outlier-Aware Weight Quantization for Efficient Fine-Tuning and Inference of Large Language Modelscited by 113
  29. 2023Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inferencecited by 101
  30. 2022LUT-GEMM: Quantized Matrix Multiplication based on LUTs for Efficient Inference in Large-Scale Generative Language Modelscited by 98
  31. 2023Memory-Efficient Fine-Tuning of Compressed Large Language Models via sub-4-bit Integer Quantizationcited by 97
  32. 2024An empirical study of LLaMA3 quantization: from LLMs to MLLMscited by 63
  33. 2024SliM-LLM: Salience-Driven Mixed-Precision Quantization for Large Language Modelscited by 61
  34. 2024Any-Precision LLM: Low-Cost Deployment of Multiple, Different-Sized LLMscited by 50
  35. 2024Fast Matrix Multiplications for Lookup Table-Quantized LLMscited by 27
  36. 2021Large Graph Convolutional Network Training with GPU-Oriented Data Communication Architecturecited by 25
  37. 2023Matrix Compression via Randomized Low Rank and Low Precision Factorizationcited by 25
  38. 2023Enhancing Computation Efficiency in Large Language Models through Weight and Activation Quantizationcited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 5.

7 papers.

  1. 2025GS-Scale: Unlocking Large-Scale 3D Gaussian Splatting Training via Host Offloadingcited by 5
  2. 2025Accelerating Mobile Language Model via Speculative Decoding and NPU-Coordinated Executioncited by 4
  3. 2026Adversarial Contrastive Learning for LLM Quantization Attackscited by 4
  4. 2026Agent-X: Full Pipeline Acceleration of On-device AI Agentscited by 4
  5. 2026Dissecting Outlier Dynamics in LLM NVFP4 Pretrainingcited by 2
  6. 2026AgenticCache: Cache-Driven Asynchronous Planning for Embodied AI Agentscited by 2
  7. 2026HCInfer: An Efficient Inference System via Error Compensation for Resource-Constrained Devicescited by 0