mlsanityData to 14 Oct 2026

M-ANT: Efficient Low-bit Group Quantization for LLMs via Mathematically Adaptive Numerical Type

arXiv 2502.18755 ↗First posted 26 Feb 2025cs.ARcited by 16

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) are one of the most important killer computer applications. The recent algorithmic advancement proposes a fine-grained group-wise quantization for LLMs, which treats a small set (e.g., 64) of values in a tensor as a compression unit. It effectively preserves the model accuracy without retraining, and has become the standard approach to efficiently deploy LLMs. On the other hand, there are works that propose various adaptive data types to better adapt to different distributions and further reduce the required bit length for LLMs. In this work, our detailed analysis unveils a key finding that while different tensors exhibit similar distributions, small groups can have markedly different distributions. As such, the group-level diversity requires a new level of adaptivity for which existing adaptive data types fail to provide. In this paper, we propose MANT, a mathematically adaptive numeric type, featuring a more flexible encoding paradigm with a wider range of data distribution and more efficient decodingcomputation fusion mechanism to address these challenges. Based on MANT, we develop a supporting framework to assign the appropriate data type for each group adaptively. Meanwhile, the dynamically generated Key-Value (KV) caches in LLMs introduce further complexity for real-time quantization. To tackle this, we propose an efficient real-time quantization mechanism. Besides, we implement a specific processing element (PE) to efficiently support MANT and incorporate a real-time quantization unit. By integrating these components into a systolic array, MANT unifies the group-wise weight and KV cache quantization and addresses the associated challenges. Our evaluation shows achieving, on average, 2.99x (up to 4.46x) speedup and 2.81x (up to 4.10x) energy reduction to the state-of-the-art LLM accelerator.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Builds on 70 of its 91 references that are in the map. First posted on arXiv 26 Feb 2025.

Builds on

70 of 91 references.

References · 70

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  8. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  9. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  10. 2016Pointer Sentinel Mixture Modelscited by 3,096
  11. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  12. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
38 more
  1. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  2. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  3. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  4. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  5. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  6. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  7. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  8. 2017Mixed Precision Trainingcited by 1,156
  9. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  10. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  11. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 880
  12. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  13. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 634
  14. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
  15. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  16. 2016DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradientscited by 569
  17. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
  18. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 424
  19. 2024QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebookscited by 378
  20. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 367
  21. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 359
  22. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 294
  23. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Servingcited by 272
  24. 2020ZeroQ: A Novel Zero Shot Quantization Frameworkcited by 269
  25. 2017Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weightscited by 217
  26. 2023TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddingscited by 202
  27. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLMcited by 158
  28. 2023Microscaling Data Formats for Deep Learningcited by 152
  29. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 152
  30. 2024Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inferencecited by 126
  31. 2017Bit Fusion: Bit-Level Dynamically Composable Architecture for Accelerating Deep Neural Networkscited by 109
  32. 2020GOBO: Quantizing Attention-Based NLP Models for Low Latency and Energy Efficient Inferencecited by 106
  33. 2024TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decodingcited by 102
  34. 2016Convolutional Neural Networks using Logarithmic Data Representationcited by 99
  35. 2023OliVe: Accelerating Large Language Models via Hardware-friendly Outlier-Victim Pair Quantizationcited by 92
  36. 2020Rethinking Differentiable Search for Mixed-Precision Neural Networkscited by 82
  37. 2021VS-Quant: Per-vector Scaled Quantization for Accurate Low-Precision Neural Network Inferencecited by 64
  38. 2021DNNFusion: Accelerating Deep Neural Networks Execution with Advanced Operator Fusioncited by 57

50 most cited of 70 All 70 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 10.

16 papers.

Sort:

By area:

  1. 2025Ecco: Improving Memory Bandwidth and Capacity for LLMs via Entropy-aware Cache Compressioncited by 14
  2. 2026M2XFP: A Metadata-Augmented Microscaling Data Format for Efficient Low-bit Quantizationcited by 8
  3. 2025Yggdrasil: Bridging Dynamic Speculation and Static Runtime for Latency-Optimal Tree-Based LLM Decodingcited by 4
  4. 2025P3-LLM: An Integrated NPU-PIM Accelerator for Edge LLM Inference Using Hybrid Numerical Formatscited by 3
  5. 2025Focus: A Streaming Concentration Architecture for Efficient Vision-Language Modelscited by 3
  6. 2026AQPIM: Breaking the PIM Capacity Wall for LLMs with In-Memory Activation Quantizationcited by 2
  7. 2026EVA: Accelerating LLM Decoding via an Efficient Vector Quantization Architecturecited by 2
  8. 2025SeVeDo: A Heterogeneous Transformer Accelerator for Low-Bit Inference via Hierarchical Group Quantization and SVD-Guided Mixed Precisioncited by 1
  9. 2026TOM: A Ternary Read-only Memory Accelerator for LLM-powered Edge Intelligencecited by 1
  10. 2025Reliable Evaluation Protocol for Low-Precision Retrievalcited by 0
  11. 2026TokenStack: A Heterogeneous HBM-PIM Architecture and Runtime for Efficient LLM Inferencecited by 0
  12. 2026GyRot: Leveraging Hidden Synergy between Rotation and Fine-grained Group Quantization for Low-bit LLM Inferencecited by 0
4 more
  1. 2026Heterogeneity-Aware Microscaling for Efficient Low-Bit LLM Inferencecited by 0
  2. 2026HBQ: Hierarchical Scaling Block Quantization with Hardware-Efficiency-Aware Design for Accurate LLM Inferencecited by 0
  3. 2026Vortex: Bridging Extreme Compression and Efficient LLM Inferencecited by 0
  4. 2026MiX: Micro-Inverted-Scaling for End-to-End Low-Bit Vision-Language Model Accelerationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Two-Stage Grid Optimization for Group-wise Quantization of LLMs
  2. 2025Calibration and Transformation-Free Weight-Only LLMs Quantization via Dynamic Grouping
  3. 2026GlowQ: Group-Shared LOw-Rank Approximation for Quantized LLMs
  4. 2025LLM Compression: How Far Can We Go in Balancing Size and Performance?
  5. 2025Learning Grouped Lattice Vector Quantizers for Low-Bit LLM Compression
  6. 2023A Speed Odyssey for Deployable Quantization of LLMs
  7. 2025FineQ: Software-Hardware Co-Design for Low-Bit Fine-Grained Mixed-Precision Quantization of LLMs
  8. 2024EasyQuant: An Efficient Data-free Quantization Algorithm for LLMs