mlsanityData to 7 Oct 2026

Irrational Complex Rotations Empower Low-bit Optimizers

arXiv 2501.12896 ↗First posted 22 Jan 2025cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

In this paper, we propose a novel optimizer state compression algorithm, namely π-Quant, which leverages the properties of irrational numbers (e.g., π) for memory-efficient training. The core idea is based on our mathematical findings, which show that a pair of parameters can be represented by a single rotation angle using the complex rotation scheme. Building on this insight, we map the parameters into a complex space and perform quantization using the corresponding rotation angles. To efficiently integrate it into optimization process, we develop an efficient system of geometric equations that computes the precise rotation angles with linear complexity. We evaluate π-Quant on a wide range of tasks. Our experiments show that it can reduce the bit-width of parameters to 3.32-bit, achieving a 75% reduction in parameter scale and a 40% decrease in GPU memory usage, all while maintaining full accuracy.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 30 of its 42 references that are in the map. First posted on arXiv 22 Jan 2025.

Builds on

30 of 42 references.

References · 30

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  3. 2023A Survey of Large Language Modelscited by 2,537
  4. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  5. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  6. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  7. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 798
  8. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  9. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  10. 2024TinyLlama: An Open-Source Small Language Modelcited by 604
  11. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 584
  12. 2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 530
18 more
  1. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
  2. 20218-bit Optimizers via Block-wise Quantizationcited by 395
  3. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 359
  4. 2023LLM-QAT: Data-Free Quantization Aware Training for Large Language Modelscited by 320
  5. 2023BitNet: Scaling 1-bit Transformers for Large Language Modelscited by 175
  6. 2018ListOps: A Diagnostic Dataset for Latent Tree Learningcited by 128
  7. 2022A Length-Extrapolatable Transformercited by 123
  8. 2023OWQ: Outlier-Aware Weight Quantization for Efficient Fine-Tuning and Inference of Large Language Modelscited by 113
  9. 2018Learning long-range spatial dependencies with horizontal gated-recurrent unitscited by 98
  10. 2023AnglE-optimized Text Embeddingscited by 98
  11. 2023PB-LLM: Partially Binarized Large Language Modelscited by 80
  12. 2021Skyformer: Remodel Self-Attention with Gaussian Kernel and Nystr"om Methodcited by 61
  13. 2021An Image Patch is a Wave: Phase-Aware Vision MLPcited by 61
  14. 20158-Bit Approximations for Parallelism in Deep Learningcited by 56
  15. 2023Memory Efficient Optimizers with 4-bit Statescited by 54
  16. 2024APTQ: Attention-aware Post-Training Mixed-Precision Quantization for Large Language Modelscited by 33
  17. 2023EulerNet: Adaptive Feature Interaction Learning via Euler's Formula for CTR Predictioncited by 26
  18. 2024EulerFormer: Sequential User Behavior Modeling with Complex Vector Attentioncited by 3

Built on by

Not cited yet.