mlsanityData to 7 Oct 2026

To Compress or Not? Pushing the Frontier of Lossless GenAI Model Weights Compression with Exponent Concentration

arXiv 2510.02676 ↗First posted 3 Oct 2025cs.LG cs.AIcited by 4

Not on the map. Why?

Abstract (arXiv)

The scaling of Generative AI (GenAI) models into the hundreds of billions of parameters makes low-precision computation indispensable for efficient deployment. We argue that the fundamental solution lies in developing low-precision floating-point formats, which inherently provide numerical stability, memory savings, and hardware efficiency without dequantization overhead. In this paper, we present a theoretical and empirical study of an exponent concentration phenomenon in GenAI weights: exponents consistently exhibit low entropy across architectures and modalities. We show that this arises naturally from α-stable distributions induced by stochastic gradient descent, and we prove tight bounds on the entropy of exponents. Our analysis establishes a theoretical compression limit near FP4.67, which motivates the design of a practical FP8 format. Building on these insights, we propose Exponent-Concentrated FP8 (ECF8), a lossless compression framework with entropy-aware encoding and GPU-optimized decoding. Experiments on LLMs and DiTs up to 671B parameters demonstrate up to 26.9% memory savings and 177.1% throughput acceleration, with perfectly lossless computations, i.e., no deviation in model outputs. Our results establish exponent concentration as a statistical law of trained models and open a principled path for lossless low-precision floating-point design in the FP8 era.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 24 of its 32 references that are in the map. First posted on arXiv 3 Oct 2025.

Builds on

24 of 32 references.

References · 24

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2025Qwen3 Technical Reportcited by 8,195
  3. 2024GPT-4o System Cardcited by 4,467
  4. 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,536
  5. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  6. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  7. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  8. 2025Qwen-Image Technical Reportcited by 973
  9. 2025FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Spacecited by 960
  10. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  11. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
  12. 2024SpinQuant: LLM quantization with learned rotationscited by 446
12 more
  1. 2016Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activationscited by 413
  2. 2018Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learningcited by 211
  3. 2024QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Servingcited by 200
  4. 2022FP8 Formats for Deep Learningcited by 187
  5. 2020The Heavy-Tail Phenomenon in SGDcited by 122
  6. 2019Traditional and Heavy-Tailed Self Regularization in Neural Network Modelscited by 104
  7. 2024KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantizationcited by 74
  8. 2024A Comprehensive Evaluation of Quantization Strategies for Large Language Modelscited by 47
  9. 202570% Size, 100% Accuracy: Lossless LLM Compression for Efficient GPU Inference via Dynamic-Length Float (DFloat11)cited by 23
  10. 2024LeanQuant: Accurate and Scalable Large Language Model Quantization with Loss-error-aware Gridcited by 20
  11. 2022Deep neural networks with dependent weights: Gaussian Process mixture limit, heavy tails, sparsity and compressibilitycited by 13
  12. 2021α-Stable convergence of heavy-tailed infinitely-wide neural networkscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Statistically-Lossless Quantization of Large Language Modelscited by 4
  2. 2026Embedding Compression via Spherical Coordinatescited by 0
  3. 2026Predicting LLM Compression Degradation from Spectral Statisticscited by 0
  4. 2026Lossless Tensor Compression as Program Synthesiscited by 0