mlsanityData to 14 Oct 2026

Atom: Low-bit Quantization for Efficient and Accurate LLM Serving

arXiv 2310.19102 ↗First posted 29 Oct 2023cs.LGcited by 272

On the map: Efficient LLM inference and serving → KV cache quantization

Abstract (arXiv)

The growing demand for Large Language Models (LLMs) in applications such as content generation, intelligent chatbots, and sentiment analysis poses considerable challenges for LLM service providers. To efficiently use GPU resources and boost throughput, batching multiple requests has emerged as a popular paradigm; to further speed up batching, LLM quantization techniques reduce memory consumption and increase computing capacity. However, prevalent quantization schemes (e.g., 8-bit weight-activation quantization) cannot fully leverage the capabilities of modern GPUs, such as 4-bit integer operators, resulting in sub-optimal performance. To maximize LLMs' serving throughput, we introduce Atom, a low-bit quantization method that achieves high throughput improvements with negligible accuracy loss. Atom significantly boosts serving throughput by using low-bit operators and considerably reduces memory consumption via low-bit quantization. It attains high accuracy by applying a novel mixed-precision and fine-grained quantization process. We evaluate Atom on 4-bit weight-activation quantization in the serving context. Atom improves end-to-end throughput (token/s) by up to 7.7× compared to the FP16 and by 2.5× compared to INT8 quantization, while maintaining the same latency target.

Key facts. Cited by 272 papers in the map (data through 14 Oct 2026). Placed in KV cache quantization, one of 12 directions in Efficient LLM inference and serving. Builds on 18 of its 46 references that are in the map. First posted on arXiv 29 Oct 2023.

Builds on

18 of 46 references.

References · 18

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  5. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  6. 2016Pointer Sentinel Mixture Modelscited by 3,096
  7. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  8. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  9. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  10. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  11. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  12. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,593
6 more
  1. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  2. 2022Efficiently Scaling Transformer Inferencecited by 539
  3. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
  4. 2023OliVe: Accelerating Large Language Models via Hardware-friendly Outlier-Victim Pair Quantizationcited by 92
  5. 2023LLM-FP4: 4-Bit Floating-Point Quantized Transformerscited by 69
  6. 2023ZeroQuant-FP: A Leap Forward in LLMs Post-Training W4A8 Quantization Using Floating-Point Formatscited by 44

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 82; 2025: 95; 2026: 91.

272 papers.

Sort:

By area:

  1. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  2. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  3. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 634
  4. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
  5. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencecited by 501
  6. 2024SpinQuant: LLM quantization with learned rotationscited by 471
  7. 2025Sparse VideoGen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsitycited by 217
  8. 2024QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Servingcited by 208
  9. 2024A Survey on Efficient Inference for Large Language Modelscited by 205
  10. 2023LLM in a flash: Efficient Large Language Model Inference with Limited Memorycited by 177
  11. 2024SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Modelscited by 170
  12. 2024DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMscited by 167
38 more
  1. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLMcited by 158
  2. 2024FlatQuant: Flatness Matters for LLM Quantizationcited by 131
  3. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 131
  4. 2024EfficientQAT: Efficient Quantization-Aware Training for Large Language Modelscited by 127
  5. 2024ViDiT-Q: Efficient and Accurate Quantization of Diffusion Transformers for Image and Video Generationcited by 115
  6. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 102
  7. 2024Q-DiT: Accurate Post-Training Quantization for Diffusion Transformerscited by 94
  8. 2024NanoFlow: Towards Optimal Large Language Model Serving Throughputcited by 90
  9. 2024BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systemscited by 87
  10. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 86
  11. 2024Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Modelscited by 71
  12. 2025KVLink: Accelerating Large Language Models via Efficient KV Cache Reusecited by 66
  13. 2024Palu: Compressing KV-Cache with Low-Rank Projectioncited by 64
  14. 2025ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inferencecited by 60
  15. 2024MesonGS: Post-training Compression of 3D Gaussians via Efficient Attribute Transformationcited by 56
  16. 2025Twilight: Adaptive Attention Sparsity with Hierarchical Top-p Pruningcited by 49
  17. 2025Massive Values in Self-Attention Modules are the Key to Contextual Knowledge Understandingcited by 47
  18. 2025ParetoQ: Improving Scaling Laws in Extremely Low-bit LLM Quantizationcited by 46
  19. 2024MBQ: Modality-Balanced Quantization for Large Vision-Language Modelscited by 45
  20. 2024SKVQ: Sliding-window Key and Value Cache Quantization for Large Language Modelscited by 44
  21. 2024KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approachescited by 42
  22. 2024SampleAttention: Near-Lossless Acceleration of Long Context LLM Inference with Adaptive Structured Sparse Attentioncited by 39
  23. 2024Large Language Model Inference Acceleration: A Comprehensive Hardware Perspectivecited by 38
  24. 2024NEO: Saving GPU Memory Crisis with CPU Offloading for Online LLM Inferencecited by 37
  25. 2024Basis Sharing: Cross-Layer Parameter Sharing for Large Language Model Compressioncited by 35
  26. 2023Understanding the Potential of FPGA-Based Spatial Acceleration for Large Language Model Inferencecited by 33
  27. 2024Fast State Restoration in LLM Serving with HCachecited by 33
  28. 2025Bridging the Gap Between Promise and Performance for Microscaling FP4 Quantizationcited by 32
  29. 2024A Survey of Low-bit Large Language Models: Basics, Systems, and Algorithmscited by 31
  30. 2025MxMoE: Mixed-precision Quantization for MoE with Accuracy and Performance Co-Designcited by 30
  31. 2024Fast Matrix Multiplications for Lookup Table-Quantized LLMscited by 29
  32. 2024Quamba: A Post-Training Quantization Recipe for Selective State Space Modelscited by 28
  33. 2024MicroScopiQ: Accelerating Foundational Models through Outlier-Aware Microscaling Quantizationcited by 28
  34. 2025AI Agentic Programming: A Survey of Techniques, Challenges, and Opportunitiescited by 28
  35. 2026The Spike, the Sparse and the Sink: Anatomy of Massive Activations and Attention Sinkscited by 27
  36. 2024DiffKV: Differentiated Memory Management for Large Language Models with Parallel KV Compactioncited by 26
  37. 2025Exploring the Roles of Large Language Models in Reshaping Transportation Systems: A Survey, Framework, and Roadmapcited by 26
  38. 2024Mitigating the Impact of Outlier Channels for Language Model Quantization with Activation Regularizationcited by 23

50 most cited of 272 All 272 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Systematic Characterization of LLM Quantization: A Performance, Energy, and Quality Perspective
  2. 2023FineQuant: Unlocking Efficiency with Fine-Grained Weight-Only Quantization for LLMs
  3. 2024MobileQuant: Mobile-friendly Quantization for On-device Language Models
  4. 2024Evaluating the Generalization Ability of Quantized LLMs: Benchmark, Analysis, and Toolbox
  5. 2025Gradual Binary Search and Dimension Expansion : A general method for activation quantization in LLMs
  6. 2024LLMEasyQuant: Scalable Quantization for Parallel and Distributed LLM Inference
  7. 2025SplitQuantV2: Enhancing Low-Bit Quantization of LLMs Without GPUs
  8. 2024A Comprehensive Study on Quantization Techniques for Large Language Models