mlsanityData to 14 Oct 2026

Anda: Unlocking Efficient LLM Inference with a Variable-Length Grouped Activation Data Format

arXiv 2411.15982 ↗First posted 24 Nov 2024cs.AR cs.AI cs.LGcited by 9

On the map: Efficient LLM inference and serving → KV cache quantization

Abstract (arXiv)

The widely-used, weight-only quantized large language models (LLMs), which leverage low-bit integer (INT) weights and retain floating-point (FP) activations, reduce storage requirements while maintaining accuracy. However, this shifts the energy and latency bottlenecks towards the FP activations that are associated with costly memory accesses and computations. Existing LLM accelerators focus primarily on computation optimizations, overlooking the potential of jointly optimizing FP computations and data movement, particularly for the dominant FP-INT GeMM operations in LLM inference. To address these challenges, we investigate the sensitivity of activation precision across various LLM modules and its impact on overall model accuracy. Based on our findings, we first propose the Anda data type: an adaptive data format with group-shared exponent bits and dynamic mantissa bit allocation. Secondly, we develop an iterative post-training adaptive precision search algorithm that optimizes the bit-width for different LLM modules to balance model accuracy, energy efficiency, and inference speed. Lastly, a suite of hardware optimization techniques is proposed to maximally exploit the benefits of the Anda format. These include a bit-plane-based data organization scheme, Anda-enhanced processing units with bit-serial computation, and a runtime bit-plane Anda compressor to simultaneously optimize storage, computation, and memory footprints. Our evaluations on FPINT GeMM operations show that Anda achieves a 2.4x speedup, 4.0x area efficiency, and 3.1x energy efficiency improvement on average for popular LLMs including OPT, LLaMA, and LLaMA-2 series over the GPU-like FP-FP baseline. Anda demonstrates strong adaptability across various application scenarios, accuracy requirements, and system performance, enabling efficient LLM inference across a wide range of deployment scenarios.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Placed in KV cache quantization, one of 12 directions in Efficient LLM inference and serving. Builds on 61 of its 90 references that are in the map. First posted on arXiv 24 Nov 2024.

Builds on

61 of 90 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Efficient Arbitrary Precision Acceleration for Large Language Models on GPU Tensor Cores26 Sep
  2. 2024SKVQ: Sliding-window Key and Value Cache Quantization for Large Language Models10 May
  3. 2024QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving7 May
  4. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache5 Feb
  5. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization31 Jan
  6. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Serving29 Oct

Other references · 55

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  6. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  7. 2016Pointer Sentinel Mixture Modelscited by 3,096
  8. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  9. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  10. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  11. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  12. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 916
38 more
  1. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  2. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 832
  3. 2018HAQ: Hardware-Aware Automated Quantization with Mixed Precisioncited by 443
  4. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 424
  5. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 414
  6. 2019HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precisioncited by 402
  7. 2024QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebookscited by 378
  8. 2021Explaining Neural Scaling Lawscited by 363
  9. 2023LLM-QAT: Data-Free Quantization Aware Training for Large Language Modelscited by 324
  10. 2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 251
  11. 2024Extreme Compression of Large Language Models via Additive Quantizationcited by 232
  12. 2022Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Modelscited by 181
  13. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 152
  14. 2024EfficientQAT: Efficient Quantization-Aware Training for Large Language Modelscited by 127
  15. 2017Bit Fusion: Bit-Level Dynamically Composable Architecture for Accelerating Deep Neural Networkscited by 109
  16. 2022LUT-GEMM: Quantized Matrix Multiplication based on LUTs for Efficient Inference in Large-Scale Generative Language Modelscited by 105
  17. 2024Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumptioncited by 97
  18. 2023OliVe: Accelerating Large Language Models via Hardware-friendly Outlier-Victim Pair Quantizationcited by 92
  19. 2023A Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4cited by 81
  20. 2024OneBit: Towards Extremely Low-bit Large Language Modelscited by 79
  21. 2021VS-Quant: Per-vector Scaled Quantization for Accurate Low-Precision Neural Network Inferencecited by 64
  22. 2016ESE: Efficient Speech Recognition Engine with Sparse LSTM on FPGAcited by 52
  23. 2023With Shared Microexponents, A Little Shifting Goes a Long Waycited by 50
  24. 2017Flexpoint: An Adaptive Numerical Format for Efficient Training of Deep Neural Networkscited by 44
  25. 2023Compressed Context Memory For Online Language Model Interactioncited by 43
  26. 2024NACL: A General and Effective KV Cache Eviction Framework for LLMs at Inference Timecited by 43
  27. 2024KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approachescited by 42
  28. 2018Training DNNs with Hybrid Block Floating Pointcited by 40
  29. 2023Norm Tweaking: High-performance Low-bit Quantization of Large Language Modelscited by 33
  30. 2024LLMC: Benchmarking Large Language Model Quantization with a Versatile Compression Toolkitcited by 33
  31. 2016Bit-pragmatic Deep Neural Network Computingcited by 32
  32. 2021FAST: DNN Training Under Variable Precision Block Floating Point with Stochastic Roundingcited by 31
  33. 2024Tender: Accelerating Large Language Models via Tensor Decomposition and Runtime Requantizationcited by 28
  34. 2020BiQGEMM: Matrix Multiplication with Lookup Table For Binary-Coding-based Quantized DNNscited by 21
  35. 2022Mokey: Enabling Narrow Fixed-Point Inference for Out-of-the-Box Floating-Point Transformer Modelscited by 19
  36. 2022BEBERT: Efficient and Robust Binary Ensemble BERTcited by 14
  37. 2023A Precision-Scalable RISC-V DNN Processor with On-Device Learning Capability at the Extreme Edgecited by 11
  38. 2022FlexBlock: A Flexible DNN Training Accelerator with Multi-Mode Block Floating Point Supportcited by 10

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 5.

9 papers.

  1. 2026M2XFP: A Metadata-Augmented Microscaling Data Format for Efficient Low-bit Quantizationcited by 8
  2. 2025CoDec: Prefix-Shared Decoding Kernel for LLMscited by 4
  3. 2025P3-LLM: An Integrated NPU-PIM Accelerator for Edge LLM Inference Using Hybrid Numerical Formatscited by 3
  4. 2025APT-LLM: Exploiting Arbitrary-Precision Tensor Core Computing for LLM Accelerationcited by 1
  5. 2025Precision-Scalable Microscaling Datapaths with Optimized Reduction Tree for Efficient NPU Integrationcited by 0
  6. 2026A Scheduling Framework for Efficient MoE Inference on Edge GPU-NDP Systemscited by 0
  7. 2026TriGen: NPU Architecture for End-to-End Acceleration of Large Language Models based on SW-HW Co-Designcited by 0
  8. 2026StreamDQ: Near-Memory Weight DeQuantization in Custom HBM for Scalable AI Inference Accelerationcited by 0
  9. 2026HiKV: Hierarchical Importance-Aware KV Cache with Hardware Acceleration for LLM Decodingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Efficient LLM Inference on CPUs
  2. 2024Highly Optimized Kernels and Fine-Grained Codebooks for LLM Inference on Arm CPUs
  3. 2024I-LLM: Efficient Integer-Only Inference for Fully-Quantized Low-Bit Large Language Models
  4. 2025FGMP: Fine-Grained Mixed-Precision Weight and Activation Quantization for Hardware-Accelerated LLM Inference
  5. 2025LUT-LLM: Efficient Large Language Model Inference with Memory-based Computations on FPGAs
  6. 2024MixPE: Quantization and Hardware Co-design for Efficient LLM Inference
  7. 2025ITERA-LLM: Boosting Sub-8-Bit Large Language Model Inference via Iterative Tensor Decomposition
  8. 2024"Give Me BF16 or Give Me Death"? Accuracy-Performance Trade-Offs in LLM Quantization