mlsanityData to 7 Oct 2026

BlockDialect: Block-wise Fine-grained Mixed Format Quantization for Energy-Efficient LLM Inference

arXiv 2501.01144 ↗First posted 2 Jan 2025cs.CL cs.LGcited by 8

On the map: Efficient LLM inference and serving → KV cache quantization

Abstract (arXiv)

The rapidly increasing size of large language models (LLMs) presents significant challenges in memory usage and computational costs. Quantizing both weights and activations can address these issues, with hardware-supported fine-grained scaling emerging as a promising solution to mitigate outliers. However, existing methods struggle to capture nuanced block data distributions. We propose BlockDialect, a block-wise fine-grained mixed format technique that assigns a per-block optimal number format from a formatbook for better data representation. Additionally, we introduce DialectFP4, a formatbook of FP4 variants (akin to dialects) that adapt to diverse data distributions. To leverage this efficiently, we propose a two-stage approach for online DialectFP4 activation quantization. Importantly, DialectFP4 ensures energy efficiency by selecting representable values as scaled integers compatible with low-precision integer arithmetic. BlockDialect achieves 10.78% (7.48%) accuracy gain on the LLaMA3-8B (LLaMA2-7B) model compared to MXFP4 format with lower bit usage per data, while being only 5.45% (2.69%) below full precision even when quantizing full-path matrix multiplication. Focusing on how to represent over how to scale, our work presents a promising path for energy-efficient LLM inference.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Placed in KV cache quantization, one of 12 directions in Efficient LLM inference and serving. Builds on 43 of its 53 references that are in the map. First posted on arXiv 2 Jan 2025.

Builds on

43 of 53 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024SpinQuant: LLM quantization with learned rotations26 May
  2. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs30 Mar
  3. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache5 Feb
  4. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization31 Jan
  5. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Serving29 Oct

Other references · 38

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  8. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  9. 2023Mistral 7Bcited by 3,065
  10. 2016Pointer Sentinel Mixture Modelscited by 2,992
  11. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  12. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
26 more
  1. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  2. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  3. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  4. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  5. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 798
  6. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  7. 2019Release Strategies and the Social Impacts of Language Modelscited by 505
  8. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 397
  9. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 359
  10. 2024QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebookscited by 354
  11. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 285
  12. 2024Extreme Compression of Large Language Models via Additive Quantizationcited by 223
  13. 2024MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Casescited by 216
  14. 2023LLM in a flash: Efficient Large Language Model Inference with Limited Memorycited by 170
  15. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 147
  16. 2023Microscaling Data Formats for Deep Learningcited by 132
  17. 2024AI and Memory Wallcited by 89
  18. 2022FP8 Quantization: The Power of the Exponentcited by 87
  19. 2023LLM-FP4: 4-Bit Floating-Point Quantized Transformerscited by 65
  20. 2021VS-Quant: Per-vector Scaled Quantization for Accurate Low-Precision Neural Network Inferencecited by 59
  21. 2023ZeroQuant-FP: A Leap Forward in LLMs Post-Training W4A8 Quantization Using Floating-Point Formatscited by 44
  22. 2023Integer or Floating Point? New Outlooks for Low-Bit Quantization on Large Language Modelscited by 18
  23. 2024Learning from Students: Applying t-Distributions to Explore Accurate and Efficient Formats for LLMscited by 18
  24. 2023FPTQ: Fine-grained Post-Training Quantization for Large Language Modelscited by 16
  25. 2024Tree Attention: Topology-aware Decoding for Long-Context Attention on GPU clusterscited by 13
  26. 2024Nanoscaling Floating-Point (NxFP): NanoMantissa, Adaptive Microexponents, and Code Recycling for Direct-Cast Compression of Large Language Modelscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 6.

8 papers.

  1. 2026Is Finer Better? The Limits of Microscaling Formats in Large Language Modelscited by 7
  2. 2025RaZeR: Pushing the Limits of NVFP4 Quantization with Redundant Zero Remappingcited by 6
  3. 2025P3-LLM: An Integrated NPU-PIM Accelerator for Edge LLM Inference Using Hybrid Numerical Formatscited by 3
  4. 2026ScaleBITS: Scalable Bitwidth Search for Hardware-Aligned Mixed-Precision LLMscited by 3
  5. 2026SFMP: Fine-Grained, Hardware-Friendly and Search-Free Mixed-Precision Quantization for Large Language Modelscited by 1
  6. 2026SemanticDialect: Semantic-Aware Mixed-Format Quantization for Video Diffusion Transformerscited by 0
  7. 2026HBQ: Hierarchical Scaling Block Quantization with Hardware-Efficiency-Aware Design for Accurate LLM Inferencecited by 0
  8. 2026AlignQuant: Tile-Aligned Mixed-Precision Quantization for Efficient LLM Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024"Give Me BF16 or Give Me Death"? Accuracy-Performance Trade-Offs in LLM Quantization
  2. 2024Anda: Unlocking Efficient LLM Inference with a Variable-Length Grouped Activation Data Format
  3. 2026MXSens: Sensitivity-Aware Mixed-Precision Quantization for Efficient LLM Inference
  4. 2023Efficient LLM Inference on CPUs
  5. 2025Cocktail: Chunk-Adaptive Mixed-Precision Quantization for Long-Context LLM Inference
  6. 2023Agile-Quant: Activation-Guided Quantization for Faster Inference of LLMs on the Edge
  7. 2026SPARQLe: Sub-Precision Activation Representation for Quantized LLM Inference ↗
  8. 2024MixPE: Quantization and Hardware Co-design for Efficient LLM Inference